# Notebook from thomasfermi/Algorithms-for-Automated-Driving
Path: book/CameraCalibration/VanishingPointCameraCalibration.ipynb

# Extrinsic Camera Calibration

Sensor calibration is one of the most important aspects of building a robot. Different sensors have different calibration procedures. Typically one can distinguish between intrinsic and extrinsic calibration. Intrinsic camera calibration tries to estimate the camera matrix $K$ (and potentially other camera-specific parameters like distortion coefficients). You can learn more about it in this [lecture by Cyril Stachniss](https://www.youtube.com/watch?v=Ou9Uj75DJX0), and this [OpenCV python tutorial](https://docs.opencv.org/4.5.2/dc/dbb/tutorial_py_calibration.html). For this chapter, we will be focusing on extrinsic camera calibration.

Refer back to the [From Pixels to meters chapter](../LaneDetection/InversePerspectiveMapping.ipynb). If you did the exercises of that chapter, you may recall that to convert the lane line pixels detected by the neural network to meters, you needed the camera height $h$ and the rotation matrix $\mathbf{R_{cr}}$, which describes how the camera is oriented with respect to the road reference frame.

```{admonition} Reminder: Road reference frame
:class: tip
The "road reference frame" was defined in [the chapter on camera basics](../LaneDetection/CameraBasics.ipynb), and it is attached to the vehicle.  The $Z$ direction in the road reference frame is always identical to the vehicle's forwards axis and the $Y$ axis is identical to the vehicle's "downwards axis". The name "road reference frame" could be a bit misleading. The only reason we used the word "road" for this reference frame, is because the $Z-X$-plane in this reference frame is identical to the road plane, which we approximate as flat.
```

$\mathbf{R_{cr}}$ was very easy to obtain through the CARLA simulator. But this is not that easy in real life. When you mount the camera to your vehicle, you can try to make it perfectly straight, which would correspond to $\mathbf{R_{cr}} = \mathbf{1}$, but you will always make a small mistake, which makes calibration necessary. The orientation with respect to the road could also change dynamically, which makes calibration challenging:
The road is not always flat. It can be banked, elevated or just uneven and therefore, the orientation of the car and the camera keeps changing with respect to the flat road frame. The orientation of the car and the camera may also change with respect to the road if the car breaks are hit hard or due to the car suspension. _____no_output_____Proper extrinsic calibration is essential for good vehicle control. The following plot shows two simulations in which the camera was mounted with a yaw of 4 degrees and a pitch of -4 degrees. In the first simulation, these yaw and pitch values were passed correctly to the `CameraGeometry` object, which was implemented in the chapter on Lane Detection. Hence, in this first simulation the calibraton was *perfect*. In the second simulation, default values of zero degrees were incorrectly assumed for both pitch and yaw. Hence, there was no calibration at all. _____no_output_____

import numpy as np
cte_no_calib = np.load("../../data/cte_yaw_4_no_calib.npy")
cte_with_calib = np.load("../../data/cte_yaw_4_with_calib.npy")
cte_perfect_calib = np.load("../../data/cte_yaw_4_perfect_calib.npy")

import matplotlib.pyplot as plt
from IPython import display
display.set_matplotlib_formats('svg')_____no_output_____get_time = lambda x: np.array(range(len(x)))/30
plt.plot(get_time(cte_no_calib), cte_no_calib, label="no calib.")
plt.plot(get_time(cte_perfect_calib), cte_perfect_calib, label="perfect calib.")
plt.legend()
plt.xlabel("Time (seconds)")
plt.ylabel("Cross Track Error (meters)")
plt.xlim(0,10.5)
plt.tight_layout()_____no_output_____

As you can see in the plot, having no calibration at all makes the vehicle leave the center of the lane until it has a constant offset of around 40 cm from the middle of the lane. Why does this happen? The wrong yaw and pitch values lead to the computation of a wrong reference path, which is then passed to the controller.  In case of *perfect calibration* the results looks different. Here, the vehicle stays in the middle of the lane within an error of around 5 cm (except for one small disturbance at around 4 seconds which is due to a bad detection of the neural network).

But as already mentioned, in the real world we do not know the exact mounting of the camera. So what we will need to do is to estimate the mounting. In this chapter you will learn how to estimate the pitch and yaw angle. The method you will learn leads to simulation results denoted as "VP calib." in the plot below:_____no_output_____

get_time = lambda x: np.array(range(len(x)))/30
plt.plot(get_time(cte_no_calib), cte_no_calib, label="no calib.")
plt.plot(get_time(cte_perfect_calib), cte_perfect_calib, label="perfect calib.")
plt.plot(get_time(cte_with_calib), cte_with_calib, label="VP calib.")
plt.legend()
plt.xlabel("Time (seconds)")
plt.ylabel("Cross Track Error (meters)")
plt.xlim(0,10.5)
plt.tight_layout()_____no_output_____

First, the vehicle leaves the middle of the lane, but then it has gathered enough data to estimate yaw and pitch angles. Consequently, it can recover and stay close to the middle of the lane. Let's learn how this "VP calib." method works!_____no_output_____## Vanishing Point Method

Our aim is to find the rotation matrix $\mathbf{R_{cr}}$, which describes how the camera is rotated

We know that the rail tracks or the lane lines are mostly parallel, however, if we take an image of the track or road from a camera, we would observe that the track lines or the lanes are not parallel in the image. The point where these lines intersect in an image is known as the vanishing point.

```{figure} images/vanishing_point.png
---
width: 50%
name: vanishing_point_carla
---
```

```{figure} images/Vanishing_point.svg
---
width: 50%
name: vanishing_point
---
Image Source [wikipedia](https://en.wikipedia.org/wiki/Vanishing_point)
```_____no_output_____We know that in the world coordinate system, these parallel lines won't ever intersect. So we say that the vanishing point is at infinity. Assuming that the car's forwards direction is aligned with the lane lines, we can say that the lane lines intersect at $Z=\infty$, where $(X,Y,Z)$ are coordinates of a point in the road reference frame (reminder: the road reference frame is attached to the vehicle, and $Z$ is the vehicle's forwards axis).
```{admonition} Important assumption!
:class: warning
The assumption we just made is very important: The vehicle is aligned with the lane, and the lane is straight. Then, the intersection of the lane lines in the image (vanishing point), will give us information about the camera mounting, i.e., the orientation of the camera with respect to vehicle. Otherwise it would only tell us something about the orientation of the vehicle with respect to the lane lines.
```

Now recall the camera projection equation

$$
    \lambda \begin{pmatrix} u \\ v\\ 1 \end{pmatrix} = \mathbf{K} \begin{pmatrix} \mathbf{R_{cr}} | \mathbf{t} \end{pmatrix}  \begin{pmatrix} X \\ Y \\ Z \\ 1 \end{pmatrix}
$$

Since this is an equation in homogeneous coordinates, we can multiply both sides by $1/Z$ and absorb this number in $\lambda$ on the left hand side:

$$
    \lambda \begin{pmatrix} u \\ v\\ 1 \end{pmatrix} = \mathbf{K} \begin{pmatrix} \mathbf{R_{cr}} | \mathbf{t} \end{pmatrix}  \begin{pmatrix} X/Z \\ Y/Z \\ 1 \\ 1/Z \end{pmatrix}
$$

If we let $Z$ go to infinity, we obtain the coordinates $(u,v)$ of the vanishing point in image space:

$$
    \lambda \begin{pmatrix} u \\ v\\ 1 \end{pmatrix} = \mathbf{K} \begin{pmatrix} \mathbf{R_{cr}} | \mathbf{t} \end{pmatrix} \begin{pmatrix} 0 \\ 0 \\ 1 \\ 0 \end{pmatrix}
$$

We define $\mathbf{p_{\infty}} = (u,v,1)^T$ for the vanishing point and we write down the transformation matrix components to obtain

$$
    \lambda \mathbf{p_{\infty}} = \mathbf{K} \begin{pmatrix} R_{xx}  R_{xy}  R_{xz}  t_x \\ R_{yx}  R_{yy}  R_{yz}  t_y \\ R_{zx}  R_{zy}  R_{zz}  t_z \end{pmatrix} \begin{pmatrix} 0 \\ 0 \\ 1 \\ 0 \end{pmatrix}
$$

_____no_output_____Now, multiplying the transformation matrix with $(0,0,1,0)^T$ will eliminate the 1st, 2nd and 4th columnn, leaving us with only the 3rd column whose values are $R_{xz}$, $R_{yz}$  and $R_{zz}$. Let us denote this column by $\mathbf{r_{3}}$. Then

$$
    \lambda \mathbf{p_{\infty}} = \mathbf{K} \mathbf{r_{3}}
$$

Columns of a rotation matrix are always vectors with length 1 (unit vectors), so $\|\mathbf{r_{3}}\|=1$. Thus, $\lambda$ equals $\frac{1} {\| \mathbf{K}^{-1} \mathbf{p_{\infty}}\| }$ and we obtain

$$
    \mathbf{r_{3}} = \frac{\mathbf{K}^{-1} \mathbf{p_{\infty}}} {\| \mathbf{K}^{-1} \mathbf{p_{\infty}}\| }
$$ (label_r3_equals_invKpinf)

_____no_output_____The idea of the vanishing point method for extrinsic calibration is the following: We determine the vanishing point $(u,v)$ in the image as the point were the lanes lines intersect. This yields $\mathbf{p_{\infty}} = (u,v,1)^T$. Since we know the intrinsic matrix $\mathbf{K}$, we can compute $\mathbf{r_3}$ using the formula above (Eq. {eq}`label_r3_equals_invKpinf`). Of course $\mathbf{r_3}$ is only one of the three columns of the rotation matrix, but as we will show later, $\mathbf{r_3}$ contains enough information to determine the yaw and pitch angle of the rotation. If we then assume that the roll angle is zero (of course an approximation), we can compute the whole rotation matrix. _____no_output_____With this vanishing point method, we are able to only recover **yaw** and **pitch** of the camera. It makes intuitive sense that the vanishing point method is not able to recover **roll** and **translation** because the vanishing point is not affected by these two! Imagine looking at a star. The star's position is only going to change if you change the yaw and pitch of your head and is not affected by roll and translation.

```{figure} images/car_rpy.svg
---
width: 60%
name: coordinate_system
---
```_____no_output_____Let's understand this by an example from CARLA. First, let us inspect of changing the roll angle:

````{tabbed} roll = -20°
```{figure} images/images_vp/p-0-y-0-r-20.png
---
width: 90%
name: roll_minus_20
---
```
````

````{tabbed} roll = 0°
```{figure} images/images_vp/p-0-y-0.png
---
width: 90%
name: roll_0
---
```
````

````{tabbed} roll = 20°
```{figure} images/images_vp/p-0-y-0-r--20.png
---
width: 90%
name: roll_20
---
```
````

You can see that the vanishing point didn't change. Similarly, let's see if this also holds true for translation:_____no_output_____

````{tabbed} translation = -2 m
```{figure} images/images_vp/trans--2.png
---
width: 90%
name: translation_minus_2
---
```
````

````{tabbed} translation = 0 m
```{figure} images/images_vp/p-0-y-0.png
---
width: 90%
name: translation_0
---
```
````

````{tabbed} translation = 2 m
```{figure} images/images_vp/trans-2.png
---
width: 90%
name: translation_2
---
```
````

You can see that the vanishing point lies at the same position for all the above images. This shows that roll and translation won't affect the vanishing point._____no_output_____
Now let's see the effect of changing pitch

````{tabbed} pitch = -5°
```{figure} images/images_vp/p--5-y-0.png
---
width: 90%
name: pitch_minus_5
---
```
````

````{tabbed} pitch = 0°
```{figure} images/images_vp/p-0-y-0.png
---
width: 90%
name: pitch_0
---
```
````

````{tabbed} pitch = 5°
```{figure} images/images_vp/p-5-y-0.png
---
width: 90%
name: pitch_5
---
```
_____no_output_____and yaw:_____no_output_____

````{tabbed} yaw = -5°
```{figure} images/images_vp/p-0-y--10.png
---
width: 90%
name: yaw_minus_5
---
```
````

````{tabbed} yaw = 0°
```{figure} images/images_vp/p-0-y-0.png
---
width: 90%
name: yaw_0
---
```
````

````{tabbed} yaw = 5°
```{figure} images/images_vp/p-0-y-10.png
---
width: 90%
name: yaw_5
---
```
_____no_output_____As you can see, the vanishing point shifted in this case. Next let us explore how we can determine yaw and pitch from $\mathbf{r_3}$_____no_output_____## Deriving yaw and pitch from the vanishing point_____no_output_____We discussed the rotation matrix in detail in the section [Basis of Image Formation](../LaneDetection/CameraBasics.ipynb) within the Lane Detection chapter. The rotation matrix parametrized by yaw, pitch and roll is given by

$$
    {R} = R_{yaw}R_{pitch}R_{roll}
$$

$$
   {R} = \begin{pmatrix} \cos(\gamma)\cos(\beta) + \sin(\alpha)\sin(\gamma)\sin(\beta)  & \cos(\gamma)\sin(\alpha)\sin(\beta) -\cos(\beta)\sin(\gamma) & -\cos(\alpha)\sin(\beta)  \\
                          \cos(\alpha)\sin(\gamma) & \cos(\alpha)\cos(\gamma) & \sin(\alpha)  \\
                          \cos(\gamma)\sin(\beta) -\cos(\beta)\sin(\alpha)\sin(\gamma)  & -\cos(\gamma)\cos(\beta)\sin(\alpha) -\sin(\gamma)\sin(\beta)  & \cos(\alpha)\cos(\beta) \\
                            \end{pmatrix}
$$

$\alpha, \beta, \gamma$ are the pitch, yaw, and roll angles respectively._____no_output_____The third column of this rotation matrix is

$$
    \begin{pmatrix}  -\cos(\alpha)\sin(\beta) \\
                            \sin(\alpha) \\
                            \cos(\alpha)\cos(\beta) \\
                            \end{pmatrix}  = \mathbf{r_{3}} = \begin{pmatrix} R_{xz} \\ R_{yz} \\ R_{zz} \end{pmatrix}
$$ (label_r3_RRR)
_____no_output_____If we determine the vanishing point $(u,v)$ in the image, we know $\mathbf{p_\infty} = (u,v,1)^T$ and hence we can compute the value of $\mathbf{r_3}= (R_{xz}, R_{yz}, R_{zz})^T$ from Eq. {eq}`label_r3_equals_invKpinf`:  $\mathbf{r_{3}} = \mathbf{K}^{-1} \mathbf{p_{\infty}} / \| \mathbf{K}^{-1} \mathbf{p_{\infty}}\| $._____no_output_____Solving Eq. {eq}`label_r3_RRR` for $\alpha$ and $\beta$ we get:

$$
    \alpha = \sin^{-1}{R_{yz}}
$$ (label_eq_alpha)

$$
    \beta = -\tan^{-1}\frac{R_{xz}}{{R_{zz}}}
$$ (label_eq_beta)

We have thus derived pitch and yaw from the vanishing point! Next, let us see how we can determine the vanishing point from an image using our lane detector._____no_output_____## Vanishing point from lane lines_____no_output_____

import cv2
import numpy as np
import copy
%matplotlib inline
import matplotlib.pyplot as plt_____no_output_____img = cv2.imread('images/test.png')[:, :, (2, 1, 0)]
plt.imshow(img);_____no_output_____

This image was collected from CARLA having a pitch of -5 degrees and yaw of -2 degrees. We aim to obtain these values of pitch and yaw just from the image. For that we will be using our lane detector network trained previously to find the left and right lane boundaries. The intersection of them will lead to the vanishing point. _____no_output_____

import sys, copy
sys.path.append('../../code')
from solutions.lane_detection.lane_detector import LaneDetector

model_path = "../../code/solutions/lane_detection/fastai_model.pth"
ld = LaneDetector(model_path=model_path)_____no_output_____

First let us run our neural network to compute for each pixel the probability whether it is background, left boundary or right boundary:_____no_output_____

background_prob, left_prob, right_prob = ld.detect(img)_____no_output_____

All pixels where `left_prob` is large (let's say larger than `0.5`), can be considered as pixels of the left boundary. Let us make all those pixels blue to get a nice visualization. Similarly we make the right boundary pixels red:_____no_output_____

img_with_detection = copy.copy(img)
img_with_detection[left_prob > 0.5, :] = [0,0,255] # blue
img_with_detection[right_prob > 0.5, :] = [255,0,0] # red
plt.imshow(img_with_detection)
plt.xlabel('$u$'); plt.ylabel('$v$');_____no_output_____

To determine the intersection of the blue and red line, we describe each of them by an equation $v(u)=m u+c$, where $u$ and $v$ are the pixel coordinates (see the axes labels in the image above). Then, it is a high school math problem to find the intersection point.

Getting the line equation $v(u)=m u+c$ for the left lane line from our `prob_left` matrix is actually just two lines of python code:_____no_output_____

# get a list of all u and v values for which left_prob[v,u] > 0.5
v_list, u_list = np.nonzero(left_prob >0.5)
# Fit a polynomial of degree 1 (which is a line) to the list of pixels
poly_left = np.poly1d(np.polyfit(u_list, v_list, deg=1))_____no_output_____

Let's add two more lines of code for the right boundary_____no_output_____

v_list, u_list = np.nonzero(right_prob >0.5)
poly_right = np.poly1d(np.polyfit(u_list, v_list, deg=1))_____no_output_____

And finally some code to visualize our lines_____no_output_____

def show_img_and_lines():
    plt.imshow(img)
    u = np.arange(0, 1024)
    v_left = poly_left(u)
    v_right = poly_right(u)
    plt.plot(u, v_left, color="b")
    plt.plot(u, v_right, color="r")
    plt.xlim(0, 1024); plt.ylim(512, 0)
    plt.xlabel('$u$'); plt.ylabel('$v$');
show_img_and_lines()_____no_output_____

In the exercises you will write code that determines the coordinates of the intersection points of two lines. We can import it here to determine the vanishing point:_____no_output_____

from solutions.camera_calibration.calibrated_lane_detector import get_intersection
u_i, v_i = get_intersection(poly_left, poly_right)
show_img_and_lines()
plt.scatter([u_i], [v_i], marker="o", s=100, color="y", zorder=10);_____no_output_____

Finally, we can find pitch and yaw of the camera form the vanishing point using Eqs. {eq}`label_r3_equals_invKpinf`, {eq}`label_eq_alpha`, {eq}`label_eq_beta`. Again you will write some code for this in the exercises._____no_output_____

from solutions.camera_calibration.calibrated_lane_detector import get_py_from_vp
pitch, yaw = get_py_from_vp(u_i, v_i, ld.cg.intrinsic_matrix)

print(f'yaw degrees:   %.2f' % np.rad2deg(yaw))
print(f'pitch degrees: %.2f' % np.rad2deg(pitch))_____no_output_____

This is very close to the correct values of -2 and -5! You can expect small errors in yaw and pitch because small errors in lane detection can lead to large errors for the vanishing point. Also the car will not always be perfectly aligned with the lane lines as it was in our example image. Therefore, the readings are mostly taken for a period of time from a video stream and then averaged.

This is why several automated driving systems like Tesla's autopilot or comma ai's openpilot require you to drive straight for a while to calibrate the cameras before you can engage the software to drive the car._____no_output_____For the exercise, you will write a `CalibratedLaneDetector` class which inherits from the `LaneDetector` class. In each cycle it determines the vanishing points and uses that to estimate yaw and pitch. It stores this information to perform an averaging over time. Additionally, the `CalibratedLaneDetector` makes sure not to use images for averaging when the lane lines are curved, since the vanishing point should only be computed from straight lines. _____no_output_____## Exercise

We provide you with a video. Your task is to use the lane detection network to calibrate the camera and output the yaw and pitch values. The video can be found in the `data` folder and is named `calibration_video.mp4`.

To start working on the exercises, open `code/tests/camera_calibration/calibrated_lane_detector.ipynb` and follow the instructions in that notebook._____no_output_____If you completed the tasks from that notebook, you can optionally check your code with a CARLA simulation. Go to the parent folder of the `code` folder and run

```bash
python -m code.tests.camera_calibration.carla_sim --help
```

to see instructions on how to run the simulation. _____no_output_____


# Notebook from dumpmemory/Transformer-MM-Explainability
Path: Transformer_MM_explainability_ViT.ipynb

_____no_output_____# **Transformer MM explainability- ViT (single modality)**_____no_output_____

!git clone https://github.com/hila-chefer/Transformer-Explainability.git

import os
os.chdir(f'./Transformer-Explainability')

!pip install -r requirements.txtCloning into 'Transformer-Explainability'...
remote: Enumerating objects: 338, done.
remote: Counting objects: 100% (338/338), done.
remote: Compressing objects: 100% (255/255), done.
remote: Total 338 (delta 166), reused 180 (delta 69), pack-reused 0
Receiving objects: 100% (338/338), 2.87 MiB | 14.69 MiB/s, done.
Resolving deltas: 100% (166/166), done.
Requirement already satisfied: Pillow>=8.1.1 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 1)) (8.2.0)
Requirement already satisfied: einops==0.3.0 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 2)) (0.3.0)
Requirement already satisfied: h5py==2.8.0 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 3)) (2.8.0)
Requirement already satisfied: imageio==2.9.0 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 4)) (2.9.0)
Requirement already satisfied: matplotlib==3.3.2 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 5)) (3.3.2)
Requirement already satisfied: numpy==1.19.2 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 6)) (1.19.2)
Requirement already satisfied: opencv_python==3.4.2.17 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 7)) (3.4.2.17)
Requirement already satisfied: scikit_image==0.17.2 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 8)) (0.17.2)
Requirement already satisfied: scipy==1.5.2 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 9)) (1.5.2)
Requirement already satisfied: sklearn in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 10)) (0.0)
Requirement already satisfied: torch==1.7.0 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 11)) (1.7.0)
Requirement already satisfied: torchvision==0.8.1 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 12)) (0.8.1)
Requirement already satisfied: tqdm==4.51.0 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 13)) (4.51.0)
Requirement already satisfied: transformers==3.5.1 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 14)) (3.5.1)
Requirement already satisfied: utils==1.0.1 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 15)) (1.0.1)
Requirement already satisfied: Pygments>=2.7.4 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 16)) (2.9.0)
Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from h5py==2.8.0->-r requirements.txt (line 3)) (1.15.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /usr/local/lib/python3.7/dist-packages (from matplotlib==3.3.2->-r requirements.txt (line 5)) (2.4.7)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib==3.3.2->-r requirements.txt (line 5)) (1.3.1)
Requirement already satisfied: certifi>=2020.06.20 in /usr/local/lib/python3.7/dist-packages (from matplotlib==3.3.2->-r requirements.txt (line 5)) (2020.12.5)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib==3.3.2->-r requirements.txt (line 5)) (0.10.0)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib==3.3.2->-r requirements.txt (line 5)) (2.8.1)
Requirement already satisfied: tifffile>=2019.7.26 in /usr/local/lib/python3.7/dist-packages (from scikit_image==0.17.2->-r requirements.txt (line 8)) (2021.4.8)
Requirement already satisfied: networkx>=2.0 in /usr/local/lib/python3.7/dist-packages (from scikit_image==0.17.2->-r requirements.txt (line 8)) (2.5.1)
Requirement already satisfied: PyWavelets>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from scikit_image==0.17.2->-r requirements.txt (line 8)) (1.1.1)
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from sklearn->-r requirements.txt (line 10)) (0.22.2.post1)
Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from torch==1.7.0->-r requirements.txt (line 11)) (0.16.0)
Requirement already satisfied: dataclasses in /usr/local/lib/python3.7/dist-packages (from torch==1.7.0->-r requirements.txt (line 11)) (0.6)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch==1.7.0->-r requirements.txt (line 11)) (3.7.4.3)
Requirement already satisfied: sacremoses in /usr/local/lib/python3.7/dist-packages (from transformers==3.5.1->-r requirements.txt (line 14)) (0.0.45)
Requirement already satisfied: tokenizers==0.9.3 in /usr/local/lib/python3.7/dist-packages (from transformers==3.5.1->-r requirements.txt (line 14)) (0.9.3)
Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers==3.5.1->-r requirements.txt (line 14)) (3.0.12)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers==3.5.1->-r requirements.txt (line 14)) (2019.12.20)
Requirement already satisfied: protobuf in /usr/local/lib/python3.7/dist-packages (from transformers==3.5.1->-r requirements.txt (line 14)) (3.12.4)
Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from transformers==3.5.1->-r requirements.txt (line 14)) (20.9)
Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers==3.5.1->-r requirements.txt (line 14)) (2.23.0)
Requirement already satisfied: sentencepiece==0.1.91 in /usr/local/lib/python3.7/dist-packages (from transformers==3.5.1->-r requirements.txt (line 14)) (0.1.91)
Requirement already satisfied: decorator<5,>=4.3 in /usr/local/lib/python3.7/dist-packages (from networkx>=2.0->scikit_image==0.17.2->-r requirements.txt (line 8)) (4.4.2)
Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->sklearn->-r requirements.txt (line 10)) (1.0.1)
Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==3.5.1->-r requirements.txt (line 14)) (7.1.2)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from protobuf->transformers==3.5.1->-r requirements.txt (line 14)) (57.0.0)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==3.5.1->-r requirements.txt (line 14)) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==3.5.1->-r requirements.txt (line 14)) (2.10)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==3.5.1->-r requirements.txt (line 14)) (1.24.3)
from PIL import Image
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
import torch
import numpy as np
import cv2_____no_output_____#@title Imagenet class indices to names
%%capture
CLS2IDX = {0: 'tench, Tinca tinca',
 1: 'goldfish, Carassius auratus',
 2: 'great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias',
 3: 'tiger shark, Galeocerdo cuvieri',
 4: 'hammerhead, hammerhead shark',
 5: 'electric ray, crampfish, numbfish, torpedo',
 6: 'stingray',
 7: 'cock',
 8: 'hen',
 9: 'ostrich, Struthio camelus',
 10: 'brambling, Fringilla montifringilla',
 11: 'goldfinch, Carduelis carduelis',
 12: 'house finch, linnet, Carpodacus mexicanus',
 13: 'junco, snowbird',
 14: 'indigo bunting, indigo finch, indigo bird, Passerina cyanea',
 15: 'robin, American robin, Turdus migratorius',
 16: 'bulbul',
 17: 'jay',
 18: 'magpie',
 19: 'chickadee',
 20: 'water ouzel, dipper',
 21: 'kite',
 22: 'bald eagle, American eagle, Haliaeetus leucocephalus',
 23: 'vulture',
 24: 'great grey owl, great gray owl, Strix nebulosa',
 25: 'European fire salamander, Salamandra salamandra',
 26: 'common newt, Triturus vulgaris',
 27: 'eft',
 28: 'spotted salamander, Ambystoma maculatum',
 29: 'axolotl, mud puppy, Ambystoma mexicanum',
 30: 'bullfrog, Rana catesbeiana',
 31: 'tree frog, tree-frog',
 32: 'tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui',
 33: 'loggerhead, loggerhead turtle, Caretta caretta',
 34: 'leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea',
 35: 'mud turtle',
 36: 'terrapin',
 37: 'box turtle, box tortoise',
 38: 'banded gecko',
 39: 'common iguana, iguana, Iguana iguana',
 40: 'American chameleon, anole, Anolis carolinensis',
 41: 'whiptail, whiptail lizard',
 42: 'agama',
 43: 'frilled lizard, Chlamydosaurus kingi',
 44: 'alligator lizard',
 45: 'Gila monster, Heloderma suspectum',
 46: 'green lizard, Lacerta viridis',
 47: 'African chameleon, Chamaeleo chamaeleon',
 48: 'Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis',
 49: 'African crocodile, Nile crocodile, Crocodylus niloticus',
 50: 'American alligator, Alligator mississipiensis',
 51: 'triceratops',
 52: 'thunder snake, worm snake, Carphophis amoenus',
 53: 'ringneck snake, ring-necked snake, ring snake',
 54: 'hognose snake, puff adder, sand viper',
 55: 'green snake, grass snake',
 56: 'king snake, kingsnake',
 57: 'garter snake, grass snake',
 58: 'water snake',
 59: 'vine snake',
 60: 'night snake, Hypsiglena torquata',
 61: 'boa constrictor, Constrictor constrictor',
 62: 'rock python, rock snake, Python sebae',
 63: 'Indian cobra, Naja naja',
 64: 'green mamba',
 65: 'sea snake',
 66: 'horned viper, cerastes, sand viper, horned asp, Cerastes cornutus',
 67: 'diamondback, diamondback rattlesnake, Crotalus adamanteus',
 68: 'sidewinder, horned rattlesnake, Crotalus cerastes',
 69: 'trilobite',
 70: 'harvestman, daddy longlegs, Phalangium opilio',
 71: 'scorpion',
 72: 'black and gold garden spider, Argiope aurantia',
 73: 'barn spider, Araneus cavaticus',
 74: 'garden spider, Aranea diademata',
 75: 'black widow, Latrodectus mactans',
 76: 'tarantula',
 77: 'wolf spider, hunting spider',
 78: 'tick',
 79: 'centipede',
 80: 'black grouse',
 81: 'ptarmigan',
 82: 'ruffed grouse, partridge, Bonasa umbellus',
 83: 'prairie chicken, prairie grouse, prairie fowl',
 84: 'peacock',
 85: 'quail',
 86: 'partridge',
 87: 'African grey, African gray, Psittacus erithacus',
 88: 'macaw',
 89: 'sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita',
 90: 'lorikeet',
 91: 'coucal',
 92: 'bee eater',
 93: 'hornbill',
 94: 'hummingbird',
 95: 'jacamar',
 96: 'toucan',
 97: 'drake',
 98: 'red-breasted merganser, Mergus serrator',
 99: 'goose',
 100: 'black swan, Cygnus atratus',
 101: 'tusker',
 102: 'echidna, spiny anteater, anteater',
 103: 'platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus',
 104: 'wallaby, brush kangaroo',
 105: 'koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus',
 106: 'wombat',
 107: 'jellyfish',
 108: 'sea anemone, anemone',
 109: 'brain coral',
 110: 'flatworm, platyhelminth',
 111: 'nematode, nematode worm, roundworm',
 112: 'conch',
 113: 'snail',
 114: 'slug',
 115: 'sea slug, nudibranch',
 116: 'chiton, coat-of-mail shell, sea cradle, polyplacophore',
 117: 'chambered nautilus, pearly nautilus, nautilus',
 118: 'Dungeness crab, Cancer magister',
 119: 'rock crab, Cancer irroratus',
 120: 'fiddler crab',
 121: 'king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica',
 122: 'American lobster, Northern lobster, Maine lobster, Homarus americanus',
 123: 'spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish',
 124: 'crayfish, crawfish, crawdad, crawdaddy',
 125: 'hermit crab',
 126: 'isopod',
 127: 'white stork, Ciconia ciconia',
 128: 'black stork, Ciconia nigra',
 129: 'spoonbill',
 130: 'flamingo',
 131: 'little blue heron, Egretta caerulea',
 132: 'American egret, great white heron, Egretta albus',
 133: 'bittern',
 134: 'crane',
 135: 'limpkin, Aramus pictus',
 136: 'European gallinule, Porphyrio porphyrio',
 137: 'American coot, marsh hen, mud hen, water hen, Fulica americana',
 138: 'bustard',
 139: 'ruddy turnstone, Arenaria interpres',
 140: 'red-backed sandpiper, dunlin, Erolia alpina',
 141: 'redshank, Tringa totanus',
 142: 'dowitcher',
 143: 'oystercatcher, oyster catcher',
 144: 'pelican',
 145: 'king penguin, Aptenodytes patagonica',
 146: 'albatross, mollymawk',
 147: 'grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus',
 148: 'killer whale, killer, orca, grampus, sea wolf, Orcinus orca',
 149: 'dugong, Dugong dugon',
 150: 'sea lion',
 151: 'Chihuahua',
 152: 'Japanese spaniel',
 153: 'Maltese dog, Maltese terrier, Maltese',
 154: 'Pekinese, Pekingese, Peke',
 155: 'Shih-Tzu',
 156: 'Blenheim spaniel',
 157: 'papillon',
 158: 'toy terrier',
 159: 'Rhodesian ridgeback',
 160: 'Afghan hound, Afghan',
 161: 'basset, basset hound',
 162: 'beagle',
 163: 'bloodhound, sleuthhound',
 164: 'bluetick',
 165: 'black-and-tan coonhound',
 166: 'Walker hound, Walker foxhound',
 167: 'English foxhound',
 168: 'redbone',
 169: 'borzoi, Russian wolfhound',
 170: 'Irish wolfhound',
 171: 'Italian greyhound',
 172: 'whippet',
 173: 'Ibizan hound, Ibizan Podenco',
 174: 'Norwegian elkhound, elkhound',
 175: 'otterhound, otter hound',
 176: 'Saluki, gazelle hound',
 177: 'Scottish deerhound, deerhound',
 178: 'Weimaraner',
 179: 'Staffordshire bullterrier, Staffordshire bull terrier',
 180: 'American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier',
 181: 'Bedlington terrier',
 182: 'Border terrier',
 183: 'Kerry blue terrier',
 184: 'Irish terrier',
 185: 'Norfolk terrier',
 186: 'Norwich terrier',
 187: 'Yorkshire terrier',
 188: 'wire-haired fox terrier',
 189: 'Lakeland terrier',
 190: 'Sealyham terrier, Sealyham',
 191: 'Airedale, Airedale terrier',
 192: 'cairn, cairn terrier',
 193: 'Australian terrier',
 194: 'Dandie Dinmont, Dandie Dinmont terrier',
 195: 'Boston bull, Boston terrier',
 196: 'miniature schnauzer',
 197: 'giant schnauzer',
 198: 'standard schnauzer',
 199: 'Scotch terrier, Scottish terrier, Scottie',
 200: 'Tibetan terrier, chrysanthemum dog',
 201: 'silky terrier, Sydney silky',
 202: 'soft-coated wheaten terrier',
 203: 'West Highland white terrier',
 204: 'Lhasa, Lhasa apso',
 205: 'flat-coated retriever',
 206: 'curly-coated retriever',
 207: 'golden retriever',
 208: 'Labrador retriever',
 209: 'Chesapeake Bay retriever',
 210: 'German short-haired pointer',
 211: 'vizsla, Hungarian pointer',
 212: 'English setter',
 213: 'Irish setter, red setter',
 214: 'Gordon setter',
 215: 'Brittany spaniel',
 216: 'clumber, clumber spaniel',
 217: 'English springer, English springer spaniel',
 218: 'Welsh springer spaniel',
 219: 'cocker spaniel, English cocker spaniel, cocker',
 220: 'Sussex spaniel',
 221: 'Irish water spaniel',
 222: 'kuvasz',
 223: 'schipperke',
 224: 'groenendael',
 225: 'malinois',
 226: 'briard',
 227: 'kelpie',
 228: 'komondor',
 229: 'Old English sheepdog, bobtail',
 230: 'Shetland sheepdog, Shetland sheep dog, Shetland',
 231: 'collie',
 232: 'Border collie',
 233: 'Bouvier des Flandres, Bouviers des Flandres',
 234: 'Rottweiler',
 235: 'German shepherd, German shepherd dog, German police dog, alsatian',
 236: 'Doberman, Doberman pinscher',
 237: 'miniature pinscher',
 238: 'Greater Swiss Mountain dog',
 239: 'Bernese mountain dog',
 240: 'Appenzeller',
 241: 'EntleBucher',
 242: 'boxer',
 243: 'bull mastiff',
 244: 'Tibetan mastiff',
 245: 'French bulldog',
 246: 'Great Dane',
 247: 'Saint Bernard, St Bernard',
 248: 'Eskimo dog, husky',
 249: 'malamute, malemute, Alaskan malamute',
 250: 'Siberian husky',
 251: 'dalmatian, coach dog, carriage dog',
 252: 'affenpinscher, monkey pinscher, monkey dog',
 253: 'basenji',
 254: 'pug, pug-dog',
 255: 'Leonberg',
 256: 'Newfoundland, Newfoundland dog',
 257: 'Great Pyrenees',
 258: 'Samoyed, Samoyede',
 259: 'Pomeranian',
 260: 'chow, chow chow',
 261: 'keeshond',
 262: 'Brabancon griffon',
 263: 'Pembroke, Pembroke Welsh corgi',
 264: 'Cardigan, Cardigan Welsh corgi',
 265: 'toy poodle',
 266: 'miniature poodle',
 267: 'standard poodle',
 268: 'Mexican hairless',
 269: 'timber wolf, grey wolf, gray wolf, Canis lupus',
 270: 'white wolf, Arctic wolf, Canis lupus tundrarum',
 271: 'red wolf, maned wolf, Canis rufus, Canis niger',
 272: 'coyote, prairie wolf, brush wolf, Canis latrans',
 273: 'dingo, warrigal, warragal, Canis dingo',
 274: 'dhole, Cuon alpinus',
 275: 'African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus',
 276: 'hyena, hyaena',
 277: 'red fox, Vulpes vulpes',
 278: 'kit fox, Vulpes macrotis',
 279: 'Arctic fox, white fox, Alopex lagopus',
 280: 'grey fox, gray fox, Urocyon cinereoargenteus',
 281: 'tabby, tabby cat',
 282: 'tiger cat',
 283: 'Persian cat',
 284: 'Siamese cat, Siamese',
 285: 'Egyptian cat',
 286: 'cougar, puma, catamount, mountain lion, painter, panther, Felis concolor',
 287: 'lynx, catamount',
 288: 'leopard, Panthera pardus',
 289: 'snow leopard, ounce, Panthera uncia',
 290: 'jaguar, panther, Panthera onca, Felis onca',
 291: 'lion, king of beasts, Panthera leo',
 292: 'tiger, Panthera tigris',
 293: 'cheetah, chetah, Acinonyx jubatus',
 294: 'brown bear, bruin, Ursus arctos',
 295: 'American black bear, black bear, Ursus americanus, Euarctos americanus',
 296: 'ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus',
 297: 'sloth bear, Melursus ursinus, Ursus ursinus',
 298: 'mongoose',
 299: 'meerkat, mierkat',
 300: 'tiger beetle',
 301: 'ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle',
 302: 'ground beetle, carabid beetle',
 303: 'long-horned beetle, longicorn, longicorn beetle',
 304: 'leaf beetle, chrysomelid',
 305: 'dung beetle',
 306: 'rhinoceros beetle',
 307: 'weevil',
 308: 'fly',
 309: 'bee',
 310: 'ant, emmet, pismire',
 311: 'grasshopper, hopper',
 312: 'cricket',
 313: 'walking stick, walkingstick, stick insect',
 314: 'cockroach, roach',
 315: 'mantis, mantid',
 316: 'cicada, cicala',
 317: 'leafhopper',
 318: 'lacewing, lacewing fly',
 319: "dragonfly, darning needle, devil's darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk",
 320: 'damselfly',
 321: 'admiral',
 322: 'ringlet, ringlet butterfly',
 323: 'monarch, monarch butterfly, milkweed butterfly, Danaus plexippus',
 324: 'cabbage butterfly',
 325: 'sulphur butterfly, sulfur butterfly',
 326: 'lycaenid, lycaenid butterfly',
 327: 'starfish, sea star',
 328: 'sea urchin',
 329: 'sea cucumber, holothurian',
 330: 'wood rabbit, cottontail, cottontail rabbit',
 331: 'hare',
 332: 'Angora, Angora rabbit',
 333: 'hamster',
 334: 'porcupine, hedgehog',
 335: 'fox squirrel, eastern fox squirrel, Sciurus niger',
 336: 'marmot',
 337: 'beaver',
 338: 'guinea pig, Cavia cobaya',
 339: 'sorrel',
 340: 'zebra',
 341: 'hog, pig, grunter, squealer, Sus scrofa',
 342: 'wild boar, boar, Sus scrofa',
 343: 'warthog',
 344: 'hippopotamus, hippo, river horse, Hippopotamus amphibius',
 345: 'ox',
 346: 'water buffalo, water ox, Asiatic buffalo, Bubalus bubalis',
 347: 'bison',
 348: 'ram, tup',
 349: 'bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis',
 350: 'ibex, Capra ibex',
 351: 'hartebeest',
 352: 'impala, Aepyceros melampus',
 353: 'gazelle',
 354: 'Arabian camel, dromedary, Camelus dromedarius',
 355: 'llama',
 356: 'weasel',
 357: 'mink',
 358: 'polecat, fitch, foulmart, foumart, Mustela putorius',
 359: 'black-footed ferret, ferret, Mustela nigripes',
 360: 'otter',
 361: 'skunk, polecat, wood pussy',
 362: 'badger',
 363: 'armadillo',
 364: 'three-toed sloth, ai, Bradypus tridactylus',
 365: 'orangutan, orang, orangutang, Pongo pygmaeus',
 366: 'gorilla, Gorilla gorilla',
 367: 'chimpanzee, chimp, Pan troglodytes',
 368: 'gibbon, Hylobates lar',
 369: 'siamang, Hylobates syndactylus, Symphalangus syndactylus',
 370: 'guenon, guenon monkey',
 371: 'patas, hussar monkey, Erythrocebus patas',
 372: 'baboon',
 373: 'macaque',
 374: 'langur',
 375: 'colobus, colobus monkey',
 376: 'proboscis monkey, Nasalis larvatus',
 377: 'marmoset',
 378: 'capuchin, ringtail, Cebus capucinus',
 379: 'howler monkey, howler',
 380: 'titi, titi monkey',
 381: 'spider monkey, Ateles geoffroyi',
 382: 'squirrel monkey, Saimiri sciureus',
 383: 'Madagascar cat, ring-tailed lemur, Lemur catta',
 384: 'indri, indris, Indri indri, Indri brevicaudatus',
 385: 'Indian elephant, Elephas maximus',
 386: 'African elephant, Loxodonta africana',
 387: 'lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens',
 388: 'giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca',
 389: 'barracouta, snoek',
 390: 'eel',
 391: 'coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch',
 392: 'rock beauty, Holocanthus tricolor',
 393: 'anemone fish',
 394: 'sturgeon',
 395: 'gar, garfish, garpike, billfish, Lepisosteus osseus',
 396: 'lionfish',
 397: 'puffer, pufferfish, blowfish, globefish',
 398: 'abacus',
 399: 'abaya',
 400: "academic gown, academic robe, judge's robe",
 401: 'accordion, piano accordion, squeeze box',
 402: 'acoustic guitar',
 403: 'aircraft carrier, carrier, flattop, attack aircraft carrier',
 404: 'airliner',
 405: 'airship, dirigible',
 406: 'altar',
 407: 'ambulance',
 408: 'amphibian, amphibious vehicle',
 409: 'analog clock',
 410: 'apiary, bee house',
 411: 'apron',
 412: 'ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin',
 413: 'assault rifle, assault gun',
 414: 'backpack, back pack, knapsack, packsack, rucksack, haversack',
 415: 'bakery, bakeshop, bakehouse',
 416: 'balance beam, beam',
 417: 'balloon',
 418: 'ballpoint, ballpoint pen, ballpen, Biro',
 419: 'Band Aid',
 420: 'banjo',
 421: 'bannister, banister, balustrade, balusters, handrail',
 422: 'barbell',
 423: 'barber chair',
 424: 'barbershop',
 425: 'barn',
 426: 'barometer',
 427: 'barrel, cask',
 428: 'barrow, garden cart, lawn cart, wheelbarrow',
 429: 'baseball',
 430: 'basketball',
 431: 'bassinet',
 432: 'bassoon',
 433: 'bathing cap, swimming cap',
 434: 'bath towel',
 435: 'bathtub, bathing tub, bath, tub',
 436: 'beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon',
 437: 'beacon, lighthouse, beacon light, pharos',
 438: 'beaker',
 439: 'bearskin, busby, shako',
 440: 'beer bottle',
 441: 'beer glass',
 442: 'bell cote, bell cot',
 443: 'bib',
 444: 'bicycle-built-for-two, tandem bicycle, tandem',
 445: 'bikini, two-piece',
 446: 'binder, ring-binder',
 447: 'binoculars, field glasses, opera glasses',
 448: 'birdhouse',
 449: 'boathouse',
 450: 'bobsled, bobsleigh, bob',
 451: 'bolo tie, bolo, bola tie, bola',
 452: 'bonnet, poke bonnet',
 453: 'bookcase',
 454: 'bookshop, bookstore, bookstall',
 455: 'bottlecap',
 456: 'bow',
 457: 'bow tie, bow-tie, bowtie',
 458: 'brass, memorial tablet, plaque',
 459: 'brassiere, bra, bandeau',
 460: 'breakwater, groin, groyne, mole, bulwark, seawall, jetty',
 461: 'breastplate, aegis, egis',
 462: 'broom',
 463: 'bucket, pail',
 464: 'buckle',
 465: 'bulletproof vest',
 466: 'bullet train, bullet',
 467: 'butcher shop, meat market',
 468: 'cab, hack, taxi, taxicab',
 469: 'caldron, cauldron',
 470: 'candle, taper, wax light',
 471: 'cannon',
 472: 'canoe',
 473: 'can opener, tin opener',
 474: 'cardigan',
 475: 'car mirror',
 476: 'carousel, carrousel, merry-go-round, roundabout, whirligig',
 477: "carpenter's kit, tool kit",
 478: 'carton',
 479: 'car wheel',
 480: 'cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM',
 481: 'cassette',
 482: 'cassette player',
 483: 'castle',
 484: 'catamaran',
 485: 'CD player',
 486: 'cello, violoncello',
 487: 'cellular telephone, cellular phone, cellphone, cell, mobile phone',
 488: 'chain',
 489: 'chainlink fence',
 490: 'chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour',
 491: 'chain saw, chainsaw',
 492: 'chest',
 493: 'chiffonier, commode',
 494: 'chime, bell, gong',
 495: 'china cabinet, china closet',
 496: 'Christmas stocking',
 497: 'church, church building',
 498: 'cinema, movie theater, movie theatre, movie house, picture palace',
 499: 'cleaver, meat cleaver, chopper',
 500: 'cliff dwelling',
 501: 'cloak',
 502: 'clog, geta, patten, sabot',
 503: 'cocktail shaker',
 504: 'coffee mug',
 505: 'coffeepot',
 506: 'coil, spiral, volute, whorl, helix',
 507: 'combination lock',
 508: 'computer keyboard, keypad',
 509: 'confectionery, confectionary, candy store',
 510: 'container ship, containership, container vessel',
 511: 'convertible',
 512: 'corkscrew, bottle screw',
 513: 'cornet, horn, trumpet, trump',
 514: 'cowboy boot',
 515: 'cowboy hat, ten-gallon hat',
 516: 'cradle',
 517: 'crane',
 518: 'crash helmet',
 519: 'crate',
 520: 'crib, cot',
 521: 'Crock Pot',
 522: 'croquet ball',
 523: 'crutch',
 524: 'cuirass',
 525: 'dam, dike, dyke',
 526: 'desk',
 527: 'desktop computer',
 528: 'dial telephone, dial phone',
 529: 'diaper, nappy, napkin',
 530: 'digital clock',
 531: 'digital watch',
 532: 'dining table, board',
 533: 'dishrag, dishcloth',
 534: 'dishwasher, dish washer, dishwashing machine',
 535: 'disk brake, disc brake',
 536: 'dock, dockage, docking facility',
 537: 'dogsled, dog sled, dog sleigh',
 538: 'dome',
 539: 'doormat, welcome mat',
 540: 'drilling platform, offshore rig',
 541: 'drum, membranophone, tympan',
 542: 'drumstick',
 543: 'dumbbell',
 544: 'Dutch oven',
 545: 'electric fan, blower',
 546: 'electric guitar',
 547: 'electric locomotive',
 548: 'entertainment center',
 549: 'envelope',
 550: 'espresso maker',
 551: 'face powder',
 552: 'feather boa, boa',
 553: 'file, file cabinet, filing cabinet',
 554: 'fireboat',
 555: 'fire engine, fire truck',
 556: 'fire screen, fireguard',
 557: 'flagpole, flagstaff',
 558: 'flute, transverse flute',
 559: 'folding chair',
 560: 'football helmet',
 561: 'forklift',
 562: 'fountain',
 563: 'fountain pen',
 564: 'four-poster',
 565: 'freight car',
 566: 'French horn, horn',
 567: 'frying pan, frypan, skillet',
 568: 'fur coat',
 569: 'garbage truck, dustcart',
 570: 'gasmask, respirator, gas helmet',
 571: 'gas pump, gasoline pump, petrol pump, island dispenser',
 572: 'goblet',
 573: 'go-kart',
 574: 'golf ball',
 575: 'golfcart, golf cart',
 576: 'gondola',
 577: 'gong, tam-tam',
 578: 'gown',
 579: 'grand piano, grand',
 580: 'greenhouse, nursery, glasshouse',
 581: 'grille, radiator grille',
 582: 'grocery store, grocery, food market, market',
 583: 'guillotine',
 584: 'hair slide',
 585: 'hair spray',
 586: 'half track',
 587: 'hammer',
 588: 'hamper',
 589: 'hand blower, blow dryer, blow drier, hair dryer, hair drier',
 590: 'hand-held computer, hand-held microcomputer',
 591: 'handkerchief, hankie, hanky, hankey',
 592: 'hard disc, hard disk, fixed disk',
 593: 'harmonica, mouth organ, harp, mouth harp',
 594: 'harp',
 595: 'harvester, reaper',
 596: 'hatchet',
 597: 'holster',
 598: 'home theater, home theatre',
 599: 'honeycomb',
 600: 'hook, claw',
 601: 'hoopskirt, crinoline',
 602: 'horizontal bar, high bar',
 603: 'horse cart, horse-cart',
 604: 'hourglass',
 605: 'iPod',
 606: 'iron, smoothing iron',
 607: "jack-o'-lantern",
 608: 'jean, blue jean, denim',
 609: 'jeep, landrover',
 610: 'jersey, T-shirt, tee shirt',
 611: 'jigsaw puzzle',
 612: 'jinrikisha, ricksha, rickshaw',
 613: 'joystick',
 614: 'kimono',
 615: 'knee pad',
 616: 'knot',
 617: 'lab coat, laboratory coat',
 618: 'ladle',
 619: 'lampshade, lamp shade',
 620: 'laptop, laptop computer',
 621: 'lawn mower, mower',
 622: 'lens cap, lens cover',
 623: 'letter opener, paper knife, paperknife',
 624: 'library',
 625: 'lifeboat',
 626: 'lighter, light, igniter, ignitor',
 627: 'limousine, limo',
 628: 'liner, ocean liner',
 629: 'lipstick, lip rouge',
 630: 'Loafer',
 631: 'lotion',
 632: 'loudspeaker, speaker, speaker unit, loudspeaker system, speaker system',
 633: "loupe, jeweler's loupe",
 634: 'lumbermill, sawmill',
 635: 'magnetic compass',
 636: 'mailbag, postbag',
 637: 'mailbox, letter box',
 638: 'maillot',
 639: 'maillot, tank suit',
 640: 'manhole cover',
 641: 'maraca',
 642: 'marimba, xylophone',
 643: 'mask',
 644: 'matchstick',
 645: 'maypole',
 646: 'maze, labyrinth',
 647: 'measuring cup',
 648: 'medicine chest, medicine cabinet',
 649: 'megalith, megalithic structure',
 650: 'microphone, mike',
 651: 'microwave, microwave oven',
 652: 'military uniform',
 653: 'milk can',
 654: 'minibus',
 655: 'miniskirt, mini',
 656: 'minivan',
 657: 'missile',
 658: 'mitten',
 659: 'mixing bowl',
 660: 'mobile home, manufactured home',
 661: 'Model T',
 662: 'modem',
 663: 'monastery',
 664: 'monitor',
 665: 'moped',
 666: 'mortar',
 667: 'mortarboard',
 668: 'mosque',
 669: 'mosquito net',
 670: 'motor scooter, scooter',
 671: 'mountain bike, all-terrain bike, off-roader',
 672: 'mountain tent',
 673: 'mouse, computer mouse',
 674: 'mousetrap',
 675: 'moving van',
 676: 'muzzle',
 677: 'nail',
 678: 'neck brace',
 679: 'necklace',
 680: 'nipple',
 681: 'notebook, notebook computer',
 682: 'obelisk',
 683: 'oboe, hautboy, hautbois',
 684: 'ocarina, sweet potato',
 685: 'odometer, hodometer, mileometer, milometer',
 686: 'oil filter',
 687: 'organ, pipe organ',
 688: 'oscilloscope, scope, cathode-ray oscilloscope, CRO',
 689: 'overskirt',
 690: 'oxcart',
 691: 'oxygen mask',
 692: 'packet',
 693: 'paddle, boat paddle',
 694: 'paddlewheel, paddle wheel',
 695: 'padlock',
 696: 'paintbrush',
 697: "pajama, pyjama, pj's, jammies",
 698: 'palace',
 699: 'panpipe, pandean pipe, syrinx',
 700: 'paper towel',
 701: 'parachute, chute',
 702: 'parallel bars, bars',
 703: 'park bench',
 704: 'parking meter',
 705: 'passenger car, coach, carriage',
 706: 'patio, terrace',
 707: 'pay-phone, pay-station',
 708: 'pedestal, plinth, footstall',
 709: 'pencil box, pencil case',
 710: 'pencil sharpener',
 711: 'perfume, essence',
 712: 'Petri dish',
 713: 'photocopier',
 714: 'pick, plectrum, plectron',
 715: 'pickelhaube',
 716: 'picket fence, paling',
 717: 'pickup, pickup truck',
 718: 'pier',
 719: 'piggy bank, penny bank',
 720: 'pill bottle',
 721: 'pillow',
 722: 'ping-pong ball',
 723: 'pinwheel',
 724: 'pirate, pirate ship',
 725: 'pitcher, ewer',
 726: "plane, carpenter's plane, woodworking plane",
 727: 'planetarium',
 728: 'plastic bag',
 729: 'plate rack',
 730: 'plow, plough',
 731: "plunger, plumber's helper",
 732: 'Polaroid camera, Polaroid Land camera',
 733: 'pole',
 734: 'police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria',
 735: 'poncho',
 736: 'pool table, billiard table, snooker table',
 737: 'pop bottle, soda bottle',
 738: 'pot, flowerpot',
 739: "potter's wheel",
 740: 'power drill',
 741: 'prayer rug, prayer mat',
 742: 'printer',
 743: 'prison, prison house',
 744: 'projectile, missile',
 745: 'projector',
 746: 'puck, hockey puck',
 747: 'punching bag, punch bag, punching ball, punchball',
 748: 'purse',
 749: 'quill, quill pen',
 750: 'quilt, comforter, comfort, puff',
 751: 'racer, race car, racing car',
 752: 'racket, racquet',
 753: 'radiator',
 754: 'radio, wireless',
 755: 'radio telescope, radio reflector',
 756: 'rain barrel',
 757: 'recreational vehicle, RV, R.V.',
 758: 'reel',
 759: 'reflex camera',
 760: 'refrigerator, icebox',
 761: 'remote control, remote',
 762: 'restaurant, eating house, eating place, eatery',
 763: 'revolver, six-gun, six-shooter',
 764: 'rifle',
 765: 'rocking chair, rocker',
 766: 'rotisserie',
 767: 'rubber eraser, rubber, pencil eraser',
 768: 'rugby ball',
 769: 'rule, ruler',
 770: 'running shoe',
 771: 'safe',
 772: 'safety pin',
 773: 'saltshaker, salt shaker',
 774: 'sandal',
 775: 'sarong',
 776: 'sax, saxophone',
 777: 'scabbard',
 778: 'scale, weighing machine',
 779: 'school bus',
 780: 'schooner',
 781: 'scoreboard',
 782: 'screen, CRT screen',
 783: 'screw',
 784: 'screwdriver',
 785: 'seat belt, seatbelt',
 786: 'sewing machine',
 787: 'shield, buckler',
 788: 'shoe shop, shoe-shop, shoe store',
 789: 'shoji',
 790: 'shopping basket',
 791: 'shopping cart',
 792: 'shovel',
 793: 'shower cap',
 794: 'shower curtain',
 795: 'ski',
 796: 'ski mask',
 797: 'sleeping bag',
 798: 'slide rule, slipstick',
 799: 'sliding door',
 800: 'slot, one-armed bandit',
 801: 'snorkel',
 802: 'snowmobile',
 803: 'snowplow, snowplough',
 804: 'soap dispenser',
 805: 'soccer ball',
 806: 'sock',
 807: 'solar dish, solar collector, solar furnace',
 808: 'sombrero',
 809: 'soup bowl',
 810: 'space bar',
 811: 'space heater',
 812: 'space shuttle',
 813: 'spatula',
 814: 'speedboat',
 815: "spider web, spider's web",
 816: 'spindle',
 817: 'sports car, sport car',
 818: 'spotlight, spot',
 819: 'stage',
 820: 'steam locomotive',
 821: 'steel arch bridge',
 822: 'steel drum',
 823: 'stethoscope',
 824: 'stole',
 825: 'stone wall',
 826: 'stopwatch, stop watch',
 827: 'stove',
 828: 'strainer',
 829: 'streetcar, tram, tramcar, trolley, trolley car',
 830: 'stretcher',
 831: 'studio couch, day bed',
 832: 'stupa, tope',
 833: 'submarine, pigboat, sub, U-boat',
 834: 'suit, suit of clothes',
 835: 'sundial',
 836: 'sunglass',
 837: 'sunglasses, dark glasses, shades',
 838: 'sunscreen, sunblock, sun blocker',
 839: 'suspension bridge',
 840: 'swab, swob, mop',
 841: 'sweatshirt',
 842: 'swimming trunks, bathing trunks',
 843: 'swing',
 844: 'switch, electric switch, electrical switch',
 845: 'syringe',
 846: 'table lamp',
 847: 'tank, army tank, armored combat vehicle, armoured combat vehicle',
 848: 'tape player',
 849: 'teapot',
 850: 'teddy, teddy bear',
 851: 'television, television system',
 852: 'tennis ball',
 853: 'thatch, thatched roof',
 854: 'theater curtain, theatre curtain',
 855: 'thimble',
 856: 'thresher, thrasher, threshing machine',
 857: 'throne',
 858: 'tile roof',
 859: 'toaster',
 860: 'tobacco shop, tobacconist shop, tobacconist',
 861: 'toilet seat',
 862: 'torch',
 863: 'totem pole',
 864: 'tow truck, tow car, wrecker',
 865: 'toyshop',
 866: 'tractor',
 867: 'trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi',
 868: 'tray',
 869: 'trench coat',
 870: 'tricycle, trike, velocipede',
 871: 'trimaran',
 872: 'tripod',
 873: 'triumphal arch',
 874: 'trolleybus, trolley coach, trackless trolley',
 875: 'trombone',
 876: 'tub, vat',
 877: 'turnstile',
 878: 'typewriter keyboard',
 879: 'umbrella',
 880: 'unicycle, monocycle',
 881: 'upright, upright piano',
 882: 'vacuum, vacuum cleaner',
 883: 'vase',
 884: 'vault',
 885: 'velvet',
 886: 'vending machine',
 887: 'vestment',
 888: 'viaduct',
 889: 'violin, fiddle',
 890: 'volleyball',
 891: 'waffle iron',
 892: 'wall clock',
 893: 'wallet, billfold, notecase, pocketbook',
 894: 'wardrobe, closet, press',
 895: 'warplane, military plane',
 896: 'washbasin, handbasin, washbowl, lavabo, wash-hand basin',
 897: 'washer, automatic washer, washing machine',
 898: 'water bottle',
 899: 'water jug',
 900: 'water tower',
 901: 'whiskey jug',
 902: 'whistle',
 903: 'wig',
 904: 'window screen',
 905: 'window shade',
 906: 'Windsor tie',
 907: 'wine bottle',
 908: 'wing',
 909: 'wok',
 910: 'wooden spoon',
 911: 'wool, woolen, woollen',
 912: 'worm fence, snake fence, snake-rail fence, Virginia fence',
 913: 'wreck',
 914: 'yawl',
 915: 'yurt',
 916: 'web site, website, internet site, site',
 917: 'comic book',
 918: 'crossword puzzle, crossword',
 919: 'street sign',
 920: 'traffic light, traffic signal, stoplight',
 921: 'book jacket, dust cover, dust jacket, dust wrapper',
 922: 'menu',
 923: 'plate',
 924: 'guacamole',
 925: 'consomme',
 926: 'hot pot, hotpot',
 927: 'trifle',
 928: 'ice cream, icecream',
 929: 'ice lolly, lolly, lollipop, popsicle',
 930: 'French loaf',
 931: 'bagel, beigel',
 932: 'pretzel',
 933: 'cheeseburger',
 934: 'hotdog, hot dog, red hot',
 935: 'mashed potato',
 936: 'head cabbage',
 937: 'broccoli',
 938: 'cauliflower',
 939: 'zucchini, courgette',
 940: 'spaghetti squash',
 941: 'acorn squash',
 942: 'butternut squash',
 943: 'cucumber, cuke',
 944: 'artichoke, globe artichoke',
 945: 'bell pepper',
 946: 'cardoon',
 947: 'mushroom',
 948: 'Granny Smith',
 949: 'strawberry',
 950: 'orange',
 951: 'lemon',
 952: 'fig',
 953: 'pineapple, ananas',
 954: 'banana',
 955: 'jackfruit, jak, jack',
 956: 'custard apple',
 957: 'pomegranate',
 958: 'hay',
 959: 'carbonara',
 960: 'chocolate sauce, chocolate syrup',
 961: 'dough',
 962: 'meat loaf, meatloaf',
 963: 'pizza, pizza pie',
 964: 'potpie',
 965: 'burrito',
 966: 'red wine',
 967: 'espresso',
 968: 'cup',
 969: 'eggnog',
 970: 'alp',
 971: 'bubble',
 972: 'cliff, drop, drop-off',
 973: 'coral reef',
 974: 'geyser',
 975: 'lakeside, lakeshore',
 976: 'promontory, headland, head, foreland',
 977: 'sandbar, sand bar',
 978: 'seashore, coast, seacoast, sea-coast',
 979: 'valley, vale',
 980: 'volcano',
 981: 'ballplayer, baseball player',
 982: 'groom, bridegroom',
 983: 'scuba diver',
 984: 'rapeseed',
 985: 'daisy',
 986: "yellow lady's slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum",
 987: 'corn',
 988: 'acorn',
 989: 'hip, rose hip, rosehip',
 990: 'buckeye, horse chestnut, conker',
 991: 'coral fungus',
 992: 'agaric',
 993: 'gyromitra',
 994: 'stinkhorn, carrion fungus',
 995: 'earthstar',
 996: 'hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa',
 997: 'bolete',
 998: 'ear, spike, capitulum',
 999: 'toilet tissue, toilet paper, bathroom tissue'}_____no_output_____

# **ViT examples**_____no_output_____### ***Implementation of rules***_____no_output_____

# rule 5 from paper
def avg_heads(cam, grad):
    cam = cam.reshape(-1, cam.shape[-2], cam.shape[-1])
    grad = grad.reshape(-1, grad.shape[-2], grad.shape[-1])
    cam = grad * cam
    cam = cam.clamp(min=0).mean(dim=0)
    return cam

# rule 6 from paper
def apply_self_attention_rules(R_ss, cam_ss):
    R_ss_addition = torch.matmul(cam_ss, R_ss)
    return R_ss_addition

def generate_relevance(model, input, index=None):
    output = model(input, register_hook=True)
    if index == None:
        index = np.argmax(output.cpu().data.numpy(), axis=-1)

    one_hot = np.zeros((1, output.size()[-1]), dtype=np.float32)
    one_hot[0, index] = 1
    one_hot_vector = one_hot
    one_hot = torch.from_numpy(one_hot).requires_grad_(True)
    one_hot = torch.sum(one_hot.cuda() * output)
    model.zero_grad()
    one_hot.backward(retain_graph=True)

    num_tokens = model.blocks[0].attn.get_attention_map().shape[-1]
    R = torch.eye(num_tokens, num_tokens).cuda()
    for blk in model.blocks:
        grad = blk.attn.get_attn_gradients()
        cam = blk.attn.get_attention_map()
        cam = avg_heads(cam, grad)
        R += apply_self_attention_rules(R.cuda(), cam.cuda())
    return R[0, 1:]_____no_output_____from baselines.ViT.ViT_new import vit_base_patch16_224 as vit

normalize = transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5])
transform = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    normalize,
])

# create heatmap from mask on image
def show_cam_on_image(img, mask):
    heatmap = cv2.applyColorMap(np.uint8(255 * mask), cv2.COLORMAP_JET)
    heatmap = np.float32(heatmap) / 255
    cam = heatmap + np.float32(img)
    cam = cam / np.max(cam)
    return cam

# initialize ViT pretrained
model = vit(pretrained=True).cuda()
model.eval()

def generate_visualization(original_image, class_index=None):
    transformer_attribution = generate_relevance(model, original_image.unsqueeze(0).cuda(), index=class_index).detach()
    transformer_attribution = transformer_attribution.reshape(1, 1, 14, 14)
    transformer_attribution = torch.nn.functional.interpolate(transformer_attribution, scale_factor=16, mode='bilinear')
    transformer_attribution = transformer_attribution.reshape(224, 224).cuda().data.cpu().numpy()
    transformer_attribution = (transformer_attribution - transformer_attribution.min()) / (transformer_attribution.max() - transformer_attribution.min())
    image_transformer_attribution = original_image.permute(1, 2, 0).data.cpu().numpy()
    image_transformer_attribution = (image_transformer_attribution - image_transformer_attribution.min()) / (image_transformer_attribution.max() - image_transformer_attribution.min())
    vis = show_cam_on_image(image_transformer_attribution, transformer_attribution)
    vis =  np.uint8(255 * vis)
    vis = cv2.cvtColor(np.array(vis), cv2.COLOR_RGB2BGR)
    return vis

def print_top_classes(predictions, **kwargs):
    # Print Top-5 predictions
    prob = torch.softmax(predictions, dim=1)
    class_indices = predictions.data.topk(5, dim=1)[1][0].tolist()
    max_str_len = 0
    class_names = []
    for cls_idx in class_indices:
        class_names.append(CLS2IDX[cls_idx])
        if len(CLS2IDX[cls_idx]) > max_str_len:
            max_str_len = len(CLS2IDX[cls_idx])

    print('Top 5 classes:')
    for cls_idx in class_indices:
        output_string = '\t{} : {}'.format(cls_idx, CLS2IDX[cls_idx])
        output_string += ' ' * (max_str_len - len(CLS2IDX[cls_idx])) + '\t\t'
        output_string += 'value = {:.3f}\t prob = {:.1f}%'.format(predictions[0, cls_idx], 100 * prob[0, cls_idx])
        print(output_string)_____no_output_____image = Image.open('samples/catdog.png')
dog_cat_image = transform(image)

fig, axs = plt.subplots(1, 3)
axs[0].imshow(image);
axs[0].axis('off');

output = model(dog_cat_image.unsqueeze(0).cuda())
print_top_classes(output)

# cat - the predicted class
cat = generate_visualization(dog_cat_image)

# dog
# generate visualization for class 243: 'bull mastiff'
dog = generate_visualization(dog_cat_image, class_index=243)

axs[1].imshow(cat);
axs[1].axis('off');
axs[2].imshow(dog);
axs[2].axis('off');Top 5 classes:
	282 : tiger cat       		value = 10.559	 prob = 68.6%
	281 : tabby, tabby cat		value = 9.059	 prob = 15.3%
	285 : Egyptian cat    		value = 8.414	 prob = 8.0%
	243 : bull mastiff    		value = 7.425	 prob = 3.0%
	811 : space heater    		value = 5.152	 prob = 0.3%
image = Image.open('samples/el2.png')
tusker_zebra_image = transform(image)

fig, axs = plt.subplots(1, 3)
axs[0].imshow(image);
axs[0].axis('off');

output = model(tusker_zebra_image.unsqueeze(0).cuda())
print_top_classes(output)

# tusker - the predicted class
tusker = generate_visualization(tusker_zebra_image)

# zebra
# generate visualization for class 340: 'zebra'
zebra = generate_visualization(tusker_zebra_image, class_index=340)

axs[1].imshow(tusker);
axs[1].axis('off');
axs[2].imshow(zebra);
axs[2].axis('off');Top 5 classes:
	101 : tusker                              		value = 11.216	 prob = 37.9%
	340 : zebra                               		value = 10.973	 prob = 29.7%
	386 : African elephant, Loxodonta africana		value = 10.747	 prob = 23.7%
	385 : Indian elephant, Elephas maximus    		value = 9.547	 prob = 7.2%
	343 : warthog                             		value = 5.566	 prob = 0.1%
image = Image.open('samples/dogbird.png')
dog_bird_image = transform(image)

fig, axs = plt.subplots(1, 3)
axs[0].imshow(image);
axs[0].axis('off');

output = model(dog_bird_image.unsqueeze(0).cuda())
print_top_classes(output)

# basset - the predicted class
basset = generate_visualization(dog_bird_image, class_index=161)

# generate visualization for class 87: 'African grey, African gray, Psittacus erithacus (grey parrot)'
parrot = generate_visualization(dog_bird_image, class_index=87)

axs[1].imshow(basset);
axs[1].axis('off');
axs[2].imshow(parrot);
axs[2].axis('off');Top 5 classes:
	161 : basset, basset hound         		value = 10.514	 prob = 78.8%
	163 : bloodhound, sleuthhound      		value = 8.604	 prob = 11.7%
	166 : Walker hound, Walker foxhound		value = 7.446	 prob = 3.7%
	162 : beagle                       		value = 5.561	 prob = 0.6%
	168 : redbone                      		value = 5.249	 prob = 0.4%


# Notebook from digirak/codeastro
Path: Day1/numpy_pandas.ipynb

# Numpy & Pandas_____no_output_____## Numpy and Arrays

Numpy is a powerful numeric library that is essential for anyone analyzing data with Python. Numpy is a huge package that can support a multitude of tasks. Numpy is also inextricably linked to SciPy, a powerful library for scientific computing with capabilities for fitting, linear algebra, machine learning, etc. Here we are just going to cover some of the basics of numpy, but I encourage you to check out the numpy documentation pages (https://numpy.org/doc/stable/) to get an idea of the variety of things you can do.

Arrays are a data type which is fundamental to Numpy. In some ways Numpy arrays are like Python lists:
    - both are used for storing data/objects
    - both are mutable
    - items can be extracted from both using indexing and slicing
    - both can be iterated over

However there are key aspects of arrays that make them very different:
    - most operators act on the elements of an array instead of the array as a whole
    - arrays can only hold data of a single type
    - arrays can efficiently store large amounts of data in memory
_____no_output_____

import numpy as np

# create some sample lists
xlist = [1, 2, 3, 4]
ylist = [1, 4, 9, 16]

# create some sample arrays
x = np.array([1, 2, 3, 4])
y = np.array([1, 4, 9, 16])_____no_output_____

First let's checkout the different behaviors between lists and arrays_____no_output_____

print(xlist * 4)

print(x * 4)

print(x / 4)

print(xlist / 4)[1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4]
[ 4  8 12 16]
[0.25 0.5  0.75 1.  ]

Notice how the list was repeated 4 times, whereas each element of the array was multiplied by 4 and the result ended up being the same length.

Division works element-wise for arrays, but division is not defined and produces an error when performed on a list._____no_output_____## Iterating, indexing, and slicing

Iterating over a 1D array looks just like iterating over a list_____no_output_____

for val in xlist:
    print(val)

for val in x:
    print(val)1
2
3
4
1
2
3
4

Iterating an N-dimensional array will iterate over slices along the first dimension._____no_output_____

y = np.zeros((5, 5))

for val in y:
    print(val)

print()
# you could accomplish the same thing like this (but you probably shouldn't)
for i in range(y.shape[0]):
    val = y[i, :]
    print(val)[0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0.]

[0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0.]

We can also select subsets of the array using conditionals._____no_output_____

xs = x[x < 2]
xs_____no_output_____

## Pandas Tables

Pandas is a powerful data analysis package that provides tools for manipulating tabular data. This is particularly useful in many astronomical applications, such as spectroscopy, and time-series. Data is organized into rows and columns where the columns are named and recalled using arbitrary Python objects (strings are the most convenient). This is in contrast to Numpy arrays where columns can only be accessed using integer indices (however, see record arrays https://docs.scipy.org/doc/numpy-1.10.1/user/basics.rec.html).

Sorting, querying, merging, and aggregation are some of the most useful Pandas features, but this tutorial will only scratch the surface. See https://pandas.pydata.org/docs/ for the full documentation.

Pandas is most useful for dealing with heterogeneous and/or large datasets, when merging or complex queries are needed, or if you have metadata associated with columns (e.g. strings as labels)._____no_output_____The basic units/objects in Pandas are the Series and DataFrame objects._____no_output_____

import pandas as pd

# Lets create a sample Series object
x = [1.0, 2.0, 4.4, 4.5, 8.8, 9.1, 8.7, 2.3, 2.4, 3.1, 5.9]
s = pd.Series(x)

print(s)0     1.0
1     2.0
2     4.4
3     4.5
4     8.8
5     9.1
6     8.7
7     2.3
8     2.4
9     3.1
10    5.9
dtype: float64

We populated a Series starting from a list of floating point numbers. Notice that two columns are printed in the output. Every entry in a Series has a corresponding integer index, generally these indices are created automatically. The data type of the series is printed below the Series itself. Series objects can only store data of a single type, but any data type can be stored.

A Series is like a single column of data in a table. A DataFrame is the Pandas object that represents a full table. Each column in the table is a Series.

There are several ways to construct a Pandas DataFrame, including from Numpy arrays, Python dictionaries, a list of Series objects, reading from a CSV, reading from a URL, etc.

Let's first construct a single-column DataFrame from our series `s`._____no_output_____

df = pd.DataFrame(s, columns=['sample'])
df_____no_output_____

Jupyter has special support for displaying DataFrames, simply typing the variable name of a DataFrame at the end of the cell will present a nicely formatted view of the table.

Let's add some more columns to our DataFrame._____no_output_____

df['sample_base'] = df['sample'] // 1
df['sample_plus1'] = df['sample'] + 1
df['sample_squared'] = df['sample']**2
df_____no_output_____

Notice that we can access the values in a column using two different syntax.

Now sort by the `sample_squared` column_____no_output_____

df = df.sort_values(by='sample_squared')
df_____no_output_____

Notice that the indices were re-ordered as well. The indices retain information about the original ordering._____no_output_____We can also select subsets of the data using conditionals similar to Numpy arrays._____no_output_____

q1 = df[df['sample'] <= 4]
q1_____no_output_____

The `.groupby` method is used to create Pandas `DataFrameGroupBy` object which can be used to calculate statistics within the groups._____no_output_____

# groups that share a common sample_base field
g = df.groupby('sample_base')

# count number of rows within each group
print(g.count())             sample  sample_plus1  sample_squared
sample_base
1.0               1             1               1
2.0               3             3               3
3.0               1             1               1
4.0               2             2               2
5.0               1             1               1
8.0               2             2               2
9.0               1             1               1

We can also merge DataFrames together using a common column.

Lets create a second DataFrame from the same original list of numbers and calculate the `sample_base` field again. We will also calculate a new column called `sample_sqrt`_____no_output_____

df2 = pd.DataFrame(x, columns=['sample'])

df2['sample_base'] = df2['sample'] // 1
df2['sample_sqrt'] = np.sqrt(df2['sample'])
df2_____no_output_____

Now we can add this new column into the original DataFrame by matching up the values on a shared column. In this case we want to match up on the original `sample` column.

Sometimes we have multiple DataFrames with one or more overlapping columns and we need to combine into a single DataFrame. This is where merging comes in.

Merging is a powerful and complex subject. I frequently find myself here: https://pandas.pydata.org/pandas-docs/stable/user_guide/merging.html to lookup various functionalities._____no_output_____

merged = pd.merge(df, df2, on='sample', suffixes=['_original', '_new'])
merged_____no_output_____

If a column name appears in both DataFrames but is not the column that you are merging on, the strings defined in the `suffixes` argument will be appended to the end of the column names.

DataFrames can be written and read from files very easily. Many formats are supported but comma separated values (CSV) is the most commonly used in astronomy. The `read_csv` function can actually read a variety of text file formats by specifying the `delimiter` argument.

You can also load a CSV directly from a URL._____no_output_____

merged.to_csv('sample.csv')

!cat sample.csv

from_csv = pd.read_csv('sample.csv', index_col=0)
from_csv,sample,sample_base_original,sample_plus1,sample_squared,sample_base_new,sample_sqrt
0,1.0,1.0,2.0,1.0,1.0,1.0
1,2.0,2.0,3.0,4.0,2.0,1.4142135623730951
2,2.3,2.0,3.3,5.289999999999999,2.0,1.51657508881031
3,2.4,2.0,3.4,5.76,2.0,1.5491933384829668
4,3.1,3.0,4.1,9.610000000000001,3.0,1.760681686165901
5,4.4,4.0,5.4,19.360000000000003,4.0,2.0976176963403033
6,4.5,4.0,5.5,20.25,4.0,2.1213203435596424
7,5.9,5.0,6.9,34.81,5.0,2.4289915602982237
8,8.7,8.0,9.7,75.68999999999998,8.0,2.949576240750525
9,8.8,8.0,9.8,77.44000000000001,8.0,2.9664793948382653
10,9.1,9.0,10.1,82.80999999999999,9.0,3.0166206257996713

### Best practices

It's almost always better to avoid loops when working with Numpy arrays or Pandas DataFrames. If I'm working on a complicated problem and I'm unsure whether to use a loop or array/DataFrame operations I usually write it up in a loop first so that I can conceptualize the problem a little easier then convert later to remove as many loops as possible._____no_output_____## Activity #1

Let's see how much faster it is to work with Numpy arrays over Python lists._____no_output_____

import time
def computevals(x):
    t1 = time.time()
    for i in range(len(x)):
        x[i] = (x[i] + 1) / 2
    t2 = time.time()

    return (t2-t1)_____no_output_____

# First we'll create a long list
lengths=[1e5,1e6,1e7,1e8]
times=[]
for length in lengths:
#length = 10000000  # must be an int
    x = list(range(int(length)))
    t=computevals(x)
    # now lets loop over all of the elements and add one then divide by two
    # we will also use the time package to time how long it takes

    print("Updated {:d} elements in {:4.3f} seconds.".format(int(length), t))
    times.append(t)Updated 100000 elements in 0.015 seconds.
Updated 1000000 elements in 0.121 seconds.
Updated 10000000 elements in 1.079 seconds.
Updated 100000000 elements in 11.383 seconds.
def computevals(x):
    t1 = time.time()
    #for i in range(len(x)):
        #x[i] = (x + 1) / 2
    x=(x+1)/2
    t2 = time.time()

    return (t2-t1)_____no_output_____times_array=[]
for length in lengths[0::]:
#length = 10000000  # must be an int
    x = np.array((range(int(length))))
    #print(len(x))
    # now lets loop over all of the elements and add one then divide by two
    # we will also use the time package to time how long it takes
    t=computevals(x)

    print("Updated {:d} elements in {:4.3f} seconds.".format(int(length), t))
    times_array.append(t)Updated 100000 elements in 0.001 seconds.
Updated 1000000 elements in 0.004 seconds.
Updated 10000000 elements in 0.067 seconds.
Updated 100000000 elements in 0.905 seconds.
from matplotlib import pyplot as plt
plt.plot(lengths,times[:],label='list time')
plt.plot(lengths[0::],times_array,label='array time')
plt.xscale('log')

plt.legend()
plt.show()_____no_output_____

1. Change the length of the array and keep track of how long the calculation takes as a function of that length.

1. Plot the time as a function of list length.

1. Now construct a Numpy array from the list `x` and perform the same calculation for several different array lengths.

1. (bonus) Plot the calculation time as a function of array length and add this line to the plot created in step #2._____no_output_____## Activity #2

Lets load a couple files into a Pandas DataFrame and rearrange and merge them into a single file in a more useful format. `example_data/star_names.json` contains a list star names. The `primary_name` column is the primary ID for the star. For each unique `primary_name` there are many `other_names` associated with it. Each `primary_name`+`other_name` combination is stored in a separate row.

1. First load the file `example_data/star_names.json` into a Pandas DataFrame. The file is in JSON format so you might look into the `pandas.read_json` function.

1. Group the DataFrame on the `primary_name` column and create a custom aggregation function that takes all of the values in the `other_name` column that have the same `primary_name` and converts them into a single string deliminated with a pipe (`|`).

1. Load the `example_data/star_props.csv` file into a separate DataFrame and merge this with the grouped DataFrame from step #2.

1. Save the result as a new CSV file. The resulting file should look like `example_data/stars_merged.csv`. You may also load this file into Pandas to see what the final DataFrame should look like before saving to a CSV._____no_output_____

import pandas as pd
star_names=pd.read_json("../example_data/star_names.json")
star_names['other_name']=star_names['other_name']+"|"+star_names['primary_name']_____no_output_____star_names_____no_output_____star_props=pd.read_csv("../example_data/star_props.csv")
stars_merged=pd.merge(star_names,star_props,on='primary_name')_____no_output_____stars_merged.drop_duplicates('primary_name').sort_values('primary_name')_____no_output_____stars_merged_fromcsv=pd.read_csv("../example_data/stars_merged.csv")_____no_output_____stars_merged_fromcsv_____no_output_____