# Notebook from abhi10010/introtodeeplearning
Path: MySolutions/MIT_6S_191_Part1_TensorFlow.ipynb

_____no_output_____

      Visit MIT Deep Learning

        Run in Google Colab

        View Source on GitHub

# Copyright Information
_____no_output_____

# Copyright 2020 MIT 6.S191 Introduction to Deep Learning. All Rights Reserved.
#
# Licensed under the MIT License. You may not use this file except in compliance
# with the License. Use and/or modification of this code outside of 6.S191 must
# reference:
#
# © MIT 6.S191: Introduction to Deep Learning
# http://introtodeeplearning.com
#_____no_output_____

# Lab 1: Intro to TensorFlow and Music Generation with RNNs

In this lab, you'll get exposure to using TensorFlow and learn how it can be used for solving deep learning tasks. Go through the code and run each cell. Along the way, you'll encounter several ***TODO*** blocks -- follow the instructions to fill them out before running those cells and continuing.

# Part 1: Intro to TensorFlow

## 0.1 Install TensorFlow

TensorFlow is a software library extensively used in machine learning. Here we'll learn how computations are represented and how to define a simple neural network in TensorFlow. For all the labs in 6.S191 2020, we'll be using the latest version of TensorFlow, TensorFlow 2, which affords great flexibility and the ability to imperatively execute operations, just like in Python. You'll notice that TensorFlow 2 is quite similar to Python in its syntax and imperative execution. Let's install TensorFlow and a couple of dependencies.
_____no_output_____

%tensorflow_version 2.x
import tensorflow as tf

# Download and import the MIT 6.S191 package
!pip install mitdeeplearning
import mitdeeplearning as mdl

import numpy as np
import matplotlib.pyplot as pltCollecting mitdeeplearning
  Downloading https://files.pythonhosted.org/packages/8b/3b/b9174b68dc10832356d02a2d83a64b43a24f1762c172754407d22fc8f960/mitdeeplearning-0.1.2.tar.gz (2.1MB)
     |████████████████████████████████| 2.1MB 15.2MB/s
Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from mitdeeplearning) (1.18.4)
Requirement already satisfied: regex in /usr/local/lib/python3.6/dist-packages (from mitdeeplearning) (2019.12.20)
Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from mitdeeplearning) (4.41.1)
Requirement already satisfied: gym in /usr/local/lib/python3.6/dist-packages (from mitdeeplearning) (0.17.1)
Requirement already satisfied: pyglet<=1.5.0,>=1.4.0 in /usr/local/lib/python3.6/dist-packages (from gym->mitdeeplearning) (1.5.0)
Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from gym->mitdeeplearning) (1.4.1)
Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from gym->mitdeeplearning) (1.12.0)
Requirement already satisfied: cloudpickle<1.4.0,>=1.2.0 in /usr/local/lib/python3.6/dist-packages (from gym->mitdeeplearning) (1.3.0)
Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from pyglet<=1.5.0,>=1.4.0->gym->mitdeeplearning) (0.16.0)
Building wheels for collected packages: mitdeeplearning
  Building wheel for mitdeeplearning (setup.py) ... done
  Created wheel for mitdeeplearning: filename=mitdeeplearning-0.1.2-cp36-none-any.whl size=2114586 sha256=2250480de45631a3e94b2c8a19b1d3dc7aed8cf4b9e81b0f5a4eda94972d5b89
  Stored in directory: /root/.cache/pip/wheels/27/e1/73/5f01c787621d8a3c857f59876c79e304b9b64db9ff5bd61b74
Successfully built mitdeeplearning
Installing collected packages: mitdeeplearning
Successfully installed mitdeeplearning-0.1.2

## 1.1 Why is TensorFlow called TensorFlow?

TensorFlow is called 'TensorFlow' because it handles the flow (node/mathematical operation) of Tensors, which are data structures that you can think of as multi-dimensional arrays. Tensors are represented as n-dimensional arrays of base dataypes such as a string or integer -- they provide a way to generalize vectors and matrices to higher dimensions.

The ```shape``` of a Tensor defines its number of dimensions and the size of each dimension. The ```rank``` of a Tensor provides the number of dimensions (n-dimensions) -- you can also think of this as the Tensor's order or degree.

Let's first look at 0-d Tensors, of which a scalar is an example:_____no_output_____

sport = tf.constant("Tennis", tf.string)
number = tf.constant(1.41421356237, tf.float64)

print("`sport` is a {}-d Tensor".format(tf.rank(sport).numpy()))
print("`number` is a {}-d Tensor".format(tf.rank(number).numpy()))`sport` is a 0-d Tensor
`number` is a 0-d Tensor

Vectors and lists can be used to create 1-d Tensors:_____no_output_____

sports = tf.constant(["Tennis", "Basketball"], tf.string)
numbers = tf.constant([3.141592, 1.414213, 2.71821], tf.float64)

print("`sports` is a {}-d Tensor with shape: {}".format(tf.rank(sports).numpy(), tf.shape(sports)))
print("`numbers` is a {}-d Tensor with shape: {}".format(tf.rank(numbers).numpy(), tf.shape(numbers)))`sports` is a 1-d Tensor with shape: [2]
`numbers` is a 1-d Tensor with shape: [3]

Next we consider creating 2-d (i.e., matrices) and higher-rank Tensors. For examples, in future labs involving image processing and computer vision, we will use 4-d Tensors. Here the dimensions correspond to the number of example images in our batch, image height, image width, and the number of color channels._____no_output_____

### Defining higher-order Tensors ###

'''TODO: Define a 2-d Tensor'''
matrix = tf.constant([[2,3,4,5],[12,13,14,15]], tf.float64)

assert isinstance(matrix, tf.Tensor), "matrix must be a tf Tensor object"
assert tf.rank(matrix).numpy() == 2_____no_output_____'''TODO: Define a 4-d Tensor.'''
# Use tf.zeros to initialize a 4-d Tensor of zeros with size 10 x 256 x 256 x 3.
#   You can think of this as 10 images where each image is RGB 256 x 256.
images = tf.zeros([10,256,256,3], tf.int32)

assert isinstance(images, tf.Tensor), "matrix must be a tf Tensor object"
assert tf.rank(images).numpy() == 4, "matrix must be of rank 4"
assert tf.shape(images).numpy().tolist() == [10, 256, 256, 3], "matrix is incorrect shape"_____no_output_____

As you have seen, the ```shape``` of a Tensor provides the number of elements in each Tensor dimension. The ```shape``` is quite useful, and we'll use it often. You can also use slicing to access subtensors within a higher-rank Tensor:_____no_output_____

row_vector = matrix[1]
column_vector = matrix[:,2]
scalar = matrix[1, 2]

print("`row_vector`: {}".format(row_vector.numpy()))
print("`column_vector`: {}".format(column_vector.numpy()))
print("`scalar`: {}".format(scalar.numpy()))`row_vector`: [12. 13. 14. 15.]
`column_vector`: [ 4. 14.]
`scalar`: 14.0

## 1.2 Computations on Tensors

A convenient way to think about and visualize computations in TensorFlow is in terms of graphs. We can define this graph in terms of Tensors, which hold data, and the mathematical operations that act on these Tensors in some order. Let's look at a simple example, and define this computation using TensorFlow:

![alt text](https://raw.githubusercontent.com/aamini/introtodeeplearning/master/lab1/img/add-graph.png)_____no_output_____

# Create the nodes in the graph, and initialize values
a = tf.constant(15)
b = tf.constant(61)

# Add them!
c1 = tf.add(a,b)
c2 = a + b # TensorFlow overrides the "+" operation so that it is able to act on Tensors
print(c1)
print(c2)tf.Tensor(76, shape=(), dtype=int32)
tf.Tensor(76, shape=(), dtype=int32)

Notice how we've created a computation graph consisting of TensorFlow operations, and how  the output is a Tensor with value 76 -- we've just created a computation graph consisting of operations, and it's executed them and given us back the result.

Now let's consider a slightly more complicated example:

![alt text](https://raw.githubusercontent.com/aamini/introtodeeplearning/master/lab1/img/computation-graph.png)

Here, we take two inputs, `a, b`, and compute an output `e`. Each node in the graph represents an operation that takes some input, does some computation, and passes its output to another node.

Let's define a simple function in TensorFlow to construct this computation function:_____no_output_____

### Defining Tensor computations ###

# Construct a simple computation function
def func(a,b):
  '''TODO: Define the operation for c, d, e (use tf.add, tf.subtract, tf.multiply).'''
  c = tf.add(a,b) # TODO
  d = tf.subtract(b,1) # TODO
  e = tf.multiply(c,d) # TODO
  return e_____no_output_____

Now, we can call this function to execute the computation graph given some inputs `a,b`:_____no_output_____

# Consider example values for a,b
a, b = 1.5, 2.5
# Execute the computation
e_out = func(a,b)
print(e_out)tf.Tensor(6.0, shape=(), dtype=float32)

Notice how our output is a Tensor with value defined by the output of the computation, and that the output has no shape as it is a single scalar value._____no_output_____## 1.3 Neural networks in TensorFlow
We can also define neural networks in TensorFlow. TensorFlow uses a high-level API called [Keras](https://www.tensorflow.org/guide/keras) that provides a powerful, intuitive framework for building and training deep learning models.

Let's first consider the example of a simple perceptron defined by just one dense layer: $ y = \sigma(Wx + b)$, where $W$ represents a matrix of weights, $b$ is a bias, $x$ is the input, $\sigma$ is the sigmoid activation function, and $y$ is the output. We can also visualize this operation using a graph:

![alt text](https://raw.githubusercontent.com/aamini/introtodeeplearning/master/lab1/img/computation-graph-2.png)

Tensors can flow through abstract types called [```Layers```](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Layer) -- the building blocks of neural networks. ```Layers``` implement common neural networks operations, and are used to update weights, compute losses, and define inter-layer connectivity. We will first define a ```Layer``` to implement the simple perceptron defined above._____no_output_____

### Defining a network Layer ###

# n_output_nodes: number of output nodes
# input_shape: shape of the input
# x: input to the layer

class OurDenseLayer(tf.keras.layers.Layer):
  def __init__(self, n_output_nodes):
    super(OurDenseLayer, self).__init__()
    self.n_output_nodes = n_output_nodes

  def build(self, input_shape):
    d = int(input_shape[-1])
    # Define and initialize parameters: a weight matrix W and bias b
    # Note that parameter initialization is random!
    self.W = self.add_weight("weight", shape=[d, self.n_output_nodes]) # note the dimensionality
    #self.W = self.add_weight("weight", shape=[2, 3])
    print(self.W)
    self.b = self.add_weight("bias", shape=[1, self.n_output_nodes]) # note the dimensionality

  def call(self, x):
    '''TODO: define the operation for z (hint: use tf.matmul)'''
    z = tf.matmul(x, self.W) + self.b# TODO
    '''TODO: define the operation for out (hint: use tf.sigmoid)'''
    y = tf.sigmoid(z)# TODO
    return y

# Since layer parameters are initialized randomly, we will set a random seed for reproducibility
tf.random.set_seed(1)
layer = OurDenseLayer(3)
layer.build((1,2))
x_input = tf.constant([[1,2.]], shape=(1,2))
y = layer.call(x_input)

# test the output!
print(y.numpy())
mdl.lab1.test_custom_dense_layer_output(y)<tf.Variable 'weight:0' shape=(2, 3) dtype=float32, numpy=
array([[-0.73366153,  0.8796015 ,  0.28695   ],
       [-0.14340228, -0.4558388 ,  0.3122064 ]], dtype=float32)>
[[0.26978594 0.45750412 0.66536945]]
[PASS] test_custom_dense_layer_output

Conveniently, TensorFlow has defined a number of ```Layers``` that are commonly used in neural networks, for example a [```Dense```](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense?version=stable). Now, instead of using a single ```Layer``` to define our simple neural network, we'll use the  [`Sequential`](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/Sequential) model from Keras and a single [`Dense` ](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/layers/Dense) layer to define our network. With the `Sequential` API, you can readily create neural networks by stacking together layers like building blocks. _____no_output_____

### Defining a neural network using the Sequential API ###

# Import relevant packages
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense

# Define the number of outputs
n_output_nodes = 3

# First define the model
model = Sequential()

'''TODO: Define a dense (fully connected) layer to compute z'''
# Remember: dense layers are defined by the parameters W and b!
# You can read more about the initialization of W and b in the TF documentation :)
# https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense?version=stable
dense_layer = Dense(32)# TODO

# Add the dense layer to the model
model.add(dense_layer)
_____no_output_____

That's it! We've defined our model using the Sequential API. Now, we can test it out using an example input:_____no_output_____

# Test model with example input
x_input = tf.constant([[1,2.]], shape=(1,2))

'''TODO: feed input into the model and predict the output!'''
model_output = model.predict(x_input)# TODO
print(model_output)[[ 1.0654529   0.09029445 -0.15469268  0.23348746  0.36469004 -0.5414037
  -1.0837082   0.14028451  0.63658863 -0.16042271  0.02551669 -0.0863093
   1.0865586   0.67842567  0.6189458  -0.9281628  -0.29972476 -1.1307082
  -0.25566167  0.03804004  0.35974327  0.29079762 -0.6343883  -0.6152252
   0.3868208   0.5228447   0.55050933  0.1265643  -0.06868425  0.40916857
   0.14658245  0.08173624]]

In addition to defining models using the `Sequential` API, we can also define neural networks by directly subclassing the [`Model`](https://https://www.tensorflow.org/api_docs/python/tf/keras/Model?version=stable) class, which groups layers together to enable model training and inference. The `Model` class captures what we refer to as a "model" or as a "network". Using Subclassing, we can create a class for our model, and then define the forward pass through the network using the `call` function. Subclassing affords the flexibility to define custom layers, custom training loops, custom activation functions, and custom models. Let's define the same neural network as above now using Subclassing rather than the `Sequential` model._____no_output_____

### Defining a model using subclassing ###

from tensorflow.keras import Model
from tensorflow.keras.layers import Dense

class SubclassModel(tf.keras.Model):
  def __init__(self, n_output_nodes):
    super(SubclassModel, self).__init__()
    self.dense_layer = Dense(n_output_nodes)
  # In the call function, we define the Model's forward pass.
  def call(self, inputs):
    return self.dense_layer(inputs)_____no_output_____

Just like the model we built using the `Sequential` API, let's test out our `SubclassModel` using an example input.

_____no_output_____

n_output_nodes = 3
model = SubclassModel(n_output_nodes)

x_input = tf.constant([[1,2.]], shape=(1,2))

print(model.call(x_input))tf.Tensor([[-1.4054141  0.5451114  1.1903379]], shape=(1, 3), dtype=float32)

Importantly, Subclassing affords us a lot of flexibility to define custom models. For example, we can use boolean arguments in the `call` function to specify different network behaviors, for example different behaviors during training and inference. Let's suppose under some instances we want our network to simply output the input, without any perturbation. We define a boolean argument `isidentity` to control this behavior:_____no_output_____

### Defining a model using subclassing and specifying custom behavior ###

from tensorflow.keras import Model
from tensorflow.keras.layers import Dense

class IdentityModel(tf.keras.Model):

  # As before, in __init__ we define the Model's layers
  # Since our desired behavior involves the forward pass, this part is unchanged
  def __init__(self, n_output_nodes):
    super(IdentityModel, self).__init__()
    self.dense_layer = tf.keras.layers.Dense(n_output_nodes, activation='sigmoid')
  '''TODO: Implement the behavior where the network outputs the input, unchanged,
      under control of the isidentity argument.'''
  def call(self, inputs, isidentity=False):
    x = self.dense_layer(inputs)
    return x if isidentity == False else inputs
    '''TODO: Implement identity behavior'''_____no_output_____

Let's test this behavior:_____no_output_____

n_output_nodes = 3
model = IdentityModel(n_output_nodes)

x_input = tf.constant([[1,2.]], shape=(1,2))
'''TODO: pass the input into the model and call with and without the input identity option.'''
out_activate = model.call(x_input)# TODO
out_identity = model.call(x_input, True)# TODO

print("Network output with activation: {}; network identity output: {}".format(out_activate.numpy(), out_identity.numpy()))Network output with activation: [[0.9139061  0.3266745  0.83923274]]; network identity output: [[1. 2.]]

Now that we have learned how to define `Layers` as well as neural networks in TensorFlow using both the `Sequential` and Subclassing APIs, we're ready to turn our attention to how to actually implement network training with backpropagation._____no_output_____## 1.4 Automatic differentiation in TensorFlow

[Automatic differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation)
is one of the most important parts of TensorFlow and is the backbone of training with
[backpropagation](https://en.wikipedia.org/wiki/Backpropagation). We will use the TensorFlow GradientTape [`tf.GradientTape`](https://www.tensorflow.org/api_docs/python/tf/GradientTape?version=stable) to trace operations for computing gradients later.

When a forward pass is made through the network, all forward-pass operations get recorded to a "tape"; then, to compute the gradient, the tape is played backwards. By default, the tape is discarded after it is played backwards; this means that a particular `tf.GradientTape` can only
compute one gradient, and subsequent calls throw a runtime error. However, we can compute multiple gradients over the same computation by creating a ```persistent``` gradient tape.

First, we will look at how we can compute gradients using GradientTape and access them for computation. We define the simple function $ y = x^2$ and compute the gradient:_____no_output_____

### Gradient computation with GradientTape ###

# y = x^2
# Example: x = 3.0
x = tf.Variable(3.0)

# Initiate the gradient tape
with tf.GradientTape() as tape:
  # Define the function
  y = x * x
# Access the gradient -- derivative of y with respect to x
dy_dx = tape.gradient(y, x)
assert dy_dx.numpy() == 6.0_____no_output_____

In training neural networks, we use differentiation and stochastic gradient descent (SGD) to optimize a loss function. Now that we have a sense of how `GradientTape` can be used to compute and access derivatives, we will look at an example where we use automatic differentiation and SGD to find the minimum of $L=(x-x_f)^2$. Here $x_f$ is a variable for a desired value we are trying to optimize for; $L$ represents a loss that we are trying to  minimize. While we can clearly solve this problem analytically ($x_{min}=x_f$), considering how we can compute this using `GradientTape` sets us up nicely for future labs where we use gradient descent to optimize entire neural network losses._____no_output_____

### Function minimization with automatic differentiation and SGD ###

# Initialize a random value for our initial x
x = tf.Variable([tf.random.normal([1])])
print("Initializing x={}".format(x.numpy()))

learning_rate = 1e-2 # learning rate for SGD
history = []
# Define the target value
x_f = 4

# We will run SGD for a number of iterations. At each iteration, we compute the loss,
#   compute the derivative of the loss with respect to x, and perform the SGD update.
for i in range(500):
  with tf.GradientTape() as tape:
    '''TODO: define the loss as described above'''
    loss = (x-x_f)*(x-x_f)# TODO

  # loss minimization using gradient tape
  grad = tape.gradient(loss, x) # compute the derivative of the loss with respect to x
  new_x = x - learning_rate*grad # sgd update
  x.assign(new_x) # update the value of x
  history.append(x.numpy()[0])

# Plot the evolution of x as we optimize towards x_f!
plt.plot(history)
plt.plot([0, 500],[x_f,x_f])
plt.legend(('Predicted', 'True'))
plt.xlabel('Iteration')
plt.ylabel('x value')Initializing x=[[-0.3974985]]

`GradientTape` provides an extremely flexible framework for automatic differentiation. In order to back propagate errors through a neural network, we track forward passes on the Tape, use this information to determine the gradients, and then use these gradients for optimization using SGD._____no_output_____


# Notebook from MarcoAvoaka/Big_Data_Challenge
Path: level_one_solution-2.ipynb

import os
# Find the latest version of spark 3.0  from http://www.apache.org/dist/spark/ and enter as the spark version
# For example:
spark_version = 'spark-3.0.3'
# spark_version = 'spark-3.<enter version>'
os.environ['SPARK_VERSION']=spark_version

# Install Spark and Java
!apt-get update
!apt-get install openjdk-8-jdk-headless -qq > /dev/null
!wget -q http://www.apache.org/dist/spark/$SPARK_VERSION/$SPARK_VERSION-bin-hadoop2.7.tgz
!tar xf $SPARK_VERSION-bin-hadoop2.7.tgz
!pip install -q findspark

# Set Environment Variables
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["SPARK_HOME"] = f"/content/{spark_version}-bin-hadoop2.7"

# Start a SparkSession
import findspark
findspark.init()
0% [Working]

Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease

0% [Connecting to archive.ubuntu.com (91.189.88.142)] [Waiting for headers] [Co

Hit:2 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/ InRelease

0% [Connecting to archive.ubuntu.com (91.189.88.142)] [Connecting to ppa.launch
0% [1 InRelease gpgv 88.7 kB] [Connecting to archive.ubuntu.com (91.189.88.142)

Ign:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease

0% [1 InRelease gpgv 88.7 kB] [Waiting for headers] [Waiting for headers] [Wait

Hit:4 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic InRelease
Hit:5 http://archive.ubuntu.com/ubuntu bionic InRelease
Ign:6 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  InRelease
Hit:7 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Release
Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:9 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release
Hit:10 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
Hit:11 http://ppa.launchpad.net/cran/libgit2/ubuntu bionic InRelease
Hit:12 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic InRelease
Hit:13 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic InRelease
Reading package lists... Done
!wget https://jdbc.postgresql.org/download/postgresql-42.2.9.jar--2022-02-20 01:19:21--  https://jdbc.postgresql.org/download/postgresql-42.2.9.jar
Resolving jdbc.postgresql.org (jdbc.postgresql.org)... 72.32.157.228, 2001:4800:3e1:1::228
Connecting to jdbc.postgresql.org (jdbc.postgresql.org)|72.32.157.228|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 914037 (893K) [application/java-archive]
Saving to: ‘postgresql-42.2.9.jar’

postgresql-42.2.9.j 100%[===================>] 892.61K  5.21MB/s    in 0.2s

2022-02-20 01:19:22 (5.21 MB/s) - ‘postgresql-42.2.9.jar’ saved [914037/914037]

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("BigData-HW-1").config("spark.driver.extraClassPath","/content/postgresql-42.2.9.jar").getOrCreate()_____no_output_____

# Load Amazon Data into Spark DataFrame_____no_output_____

from pyspark import SparkFiles
url = "https://s3.amazonaws.com/amazon-reviews-pds/tsv/amazon_reviews_us_Video_Games_v1_00.tsv.gz"
spark.sparkContext.addFile(url)
video_games_df = spark.read.csv(SparkFiles.get("amazon_reviews_us_Video_Games_v1_00.tsv.gz"), sep="\t", header=True, inferSchema=True)
video_games_df.show()+-----------+-----------+--------------+----------+--------------+--------------------+----------------+-----------+-------------+-----------+----+-----------------+--------------------+--------------------+-----------+
|marketplace|customer_id|     review_id|product_id|product_parent|       product_title|product_category|star_rating|helpful_votes|total_votes|vine|verified_purchase|     review_headline|         review_body|review_date|
+-----------+-----------+--------------+----------+--------------+--------------------+----------------+-----------+-------------+-----------+----+-----------------+--------------------+--------------------+-----------+
|         US|   12039526| RTIS3L2M1F5SM|B001CXYMFS|     737716809|Thrustmaster T-Fl...|     Video Games|          5|            0|          0|   N|                Y|an amazing joysti...|Used this for Eli...| 2015-08-31|
|         US|    9636577| R1ZV7R40OLHKD|B00M920ND6|     569686175|Tonsee 6 buttons ...|     Video Games|          5|            0|          0|   N|                Y|Definitely a sile...|Loved it,  I didn...| 2015-08-31|
|         US|    2331478|R3BH071QLH8QMC|B0029CSOD2|      98937668|Hidden Mysteries:...|     Video Games|          1|            0|          1|   N|                Y|            One Star|poor quality work...| 2015-08-31|
|         US|   52495923|R127K9NTSXA2YH|B00GOOSV98|      23143350|GelTabz Performan...|     Video Games|          3|            0|          0|   N|                Y|good, but could b...|nice, but tend to...| 2015-08-31|
|         US|   14533949|R32ZWUXDJPW27Q|B00Y074JOM|     821342511|Zero Suit Samus a...|     Video Games|          4|            0|          0|   N|                Y|   Great but flawed.|Great amiibo, gre...| 2015-08-31|
|         US|    2377552|R3AQQ4YUKJWBA6|B002UBI6W6|     328764615|Psyclone Recharge...|     Video Games|          1|            0|          0|   N|                Y|            One Star|The remote consta...| 2015-08-31|
|         US|   17521011|R2F0POU5K6F73F|B008XHCLFO|      24234603|Protection for yo...|     Video Games|          5|            0|          0|   N|                Y|              A Must|I have a 2012-201...| 2015-08-31|
|         US|   19676307|R3VNR804HYSMR6|B00BRA9R6A|     682267517|   Nerf 3DS XL Armor|     Video Games|          5|            0|          0|   N|                Y|          Five Stars|Perfect, kids lov...| 2015-08-31|
|         US|     224068| R3GZTM72WA2QH|B009EPWJLA|     435241890|One Piece: Pirate...|     Video Games|          5|            0|          0|   N|                Y|          Five Stars|            Excelent| 2015-08-31|
|         US|   48467989| RNQOY62705W1K|B0000AV7GB|     256572651|Playstation 2 Dan...|     Video Games|          4|            0|          0|   N|                Y|          Four Stars|Slippery but expe...| 2015-08-31|
|         US|     106569|R1VTIA3JTYBY02|B00008KTNN|     384411423|Metal Arms: Glitc...|     Video Games|          5|            0|          0|   N|                N|          Five Stars|Love the game. Se...| 2015-08-31|
|         US|   48269642|R29DOU8791QZL8|B000A3IA0Y|     472622859|72 Pin Connector ...|     Video Games|          1|            0|          0|   N|                Y| Game will get stuck|Does not fit prop...| 2015-08-31|
|         US|   52738710|R15DUT1VIJ9RJZ|B0053BQN34|     577628462|uDraw Gametablet ...|     Video Games|          2|            0|          0|   N|                Y|We have tried it ...|This was way too ...| 2015-08-31|
|         US|   10556786|R3IMF2MQ3OU9ZM|B002I0HIMI|     988218515|NBA 2K12(Covers M...|     Video Games|          4|            0|          0|   N|                Y|          Four Stars|Works great good ...| 2015-08-31|
|         US|    2963837|R23H79DHOZTYAU|B0081EH12M|     770100932|New Trigger Grips...|     Video Games|          1|            1|          1|   N|                Y|Now i have to buy...|It did not fit th...| 2015-08-31|
|         US|   23092109| RIV24EQAIXA4O|B005FMLZQQ|      24647669|Xbox 360 Media Re...|     Video Games|          5|            0|          0|   N|                Y|          Five Stars|perfect lightweig...| 2015-08-31|
|         US|   23091728|R3UCNGYDVN24YB|B002BSA388|      33706205|Super Mario Galaxy 2|     Video Games|          5|            0|          0|   N|                Y|          Five Stars|               great| 2015-08-31|
|         US|   10712640| RUL4H4XTTN2DY|B00BUSLSAC|     829667834|Nintendo 3DS XL -...|     Video Games|          5|            0|          0|   N|                Y|          Five Stars|Works beautifully...| 2015-08-31|
|         US|   17455376|R20JF7Z4DHTNX5|B00KWF38AW|     110680188|Captain Toad:  Tr...|     Video Games|          5|            0|          0|   N|                Y|          Five Stars|Kids loved the ga...| 2015-08-31|
|         US|   14754850|R2T1AJ5MFI2260|B00BRQJYA8|     616463426|Lego Batman 2: DC...|     Video Games|          4|            0|          0|   N|                Y|          Four Stars|           Goodngame| 2015-08-31|
+-----------+-----------+--------------+----------+--------------+--------------------+----------------+-----------+-------------+-----------+----+-----------------+--------------------+--------------------+-----------+
only showing top 20 rows

# Size of Data_____no_output_____

video_games_df.count()_____no_output_____

# Cleaned up DataFrames to match tables_____no_output_____

from pyspark.sql.functions import to_date
# Review DataFrame
review_id_df = video_games_df.select(["review_id", "customer_id", "product_id", "product_parent", to_date("review_date", 'yyyy-MM-dd').alias("review_date")])
review_id_df.show()+--------------+-----------+----------+--------------+-----------+
|     review_id|customer_id|product_id|product_parent|review_date|
+--------------+-----------+----------+--------------+-----------+
| RTIS3L2M1F5SM|   12039526|B001CXYMFS|     737716809| 2015-08-31|
| R1ZV7R40OLHKD|    9636577|B00M920ND6|     569686175| 2015-08-31|
|R3BH071QLH8QMC|    2331478|B0029CSOD2|      98937668| 2015-08-31|
|R127K9NTSXA2YH|   52495923|B00GOOSV98|      23143350| 2015-08-31|
|R32ZWUXDJPW27Q|   14533949|B00Y074JOM|     821342511| 2015-08-31|
|R3AQQ4YUKJWBA6|    2377552|B002UBI6W6|     328764615| 2015-08-31|
|R2F0POU5K6F73F|   17521011|B008XHCLFO|      24234603| 2015-08-31|
|R3VNR804HYSMR6|   19676307|B00BRA9R6A|     682267517| 2015-08-31|
| R3GZTM72WA2QH|     224068|B009EPWJLA|     435241890| 2015-08-31|
| RNQOY62705W1K|   48467989|B0000AV7GB|     256572651| 2015-08-31|
|R1VTIA3JTYBY02|     106569|B00008KTNN|     384411423| 2015-08-31|
|R29DOU8791QZL8|   48269642|B000A3IA0Y|     472622859| 2015-08-31|
|R15DUT1VIJ9RJZ|   52738710|B0053BQN34|     577628462| 2015-08-31|
|R3IMF2MQ3OU9ZM|   10556786|B002I0HIMI|     988218515| 2015-08-31|
|R23H79DHOZTYAU|    2963837|B0081EH12M|     770100932| 2015-08-31|
| RIV24EQAIXA4O|   23092109|B005FMLZQQ|      24647669| 2015-08-31|
|R3UCNGYDVN24YB|   23091728|B002BSA388|      33706205| 2015-08-31|
| RUL4H4XTTN2DY|   10712640|B00BUSLSAC|     829667834| 2015-08-31|
|R20JF7Z4DHTNX5|   17455376|B00KWF38AW|     110680188| 2015-08-31|
|R2T1AJ5MFI2260|   14754850|B00BRQJYA8|     616463426| 2015-08-31|
+--------------+-----------+----------+--------------+-----------+
only showing top 20 rows

products_df = video_games_df.select(["product_id", "product_title"]).drop_duplicates()_____no_output_____reviews_df = video_games_df.select(["review_id", "review_headline", "review_body"])
reviews_df.show(10)+--------------+--------------------+--------------------+
|     review_id|     review_headline|         review_body|
+--------------+--------------------+--------------------+
| RTIS3L2M1F5SM|an amazing joysti...|Used this for Eli...|
| R1ZV7R40OLHKD|Definitely a sile...|Loved it,  I didn...|
|R3BH071QLH8QMC|            One Star|poor quality work...|
|R127K9NTSXA2YH|good, but could b...|nice, but tend to...|
|R32ZWUXDJPW27Q|   Great but flawed.|Great amiibo, gre...|
|R3AQQ4YUKJWBA6|            One Star|The remote consta...|
|R2F0POU5K6F73F|              A Must|I have a 2012-201...|
|R3VNR804HYSMR6|          Five Stars|Perfect, kids lov...|
| R3GZTM72WA2QH|          Five Stars|            Excelent|
| RNQOY62705W1K|          Four Stars|Slippery but expe...|
+--------------+--------------------+--------------------+
only showing top 10 rows

customers_df = video_games_df.groupby("customer_id").agg({"customer_id": "count"}).withColumnRenamed("count(customer_id)", "customer_count")
customers_df.show()+-----------+--------------+
|customer_id|customer_count|
+-----------+--------------+
|   48670265|             1|
|   49103216|             2|
|    1131200|             1|
|   43076447|             2|
|   46261368|             1|
|    4883305|             5|
|   41192649|             1|
|   40985731|             7|
|   10437900|             2|
|   22245671|             1|
|    2574873|             1|
|    4696154|             1|
|    5621202|             1|
|    5871933|             2|
|   44089812|             1|
|    2845910|             1|
|    5274369|             1|
|   39069693|             2|
|     137793|             1|
|   31914942|             3|
+-----------+--------------+
only showing top 20 rows

vine_df = video_games_df.select(["review_id", "star_rating", "helpful_votes", "total_votes", "vine"])
vine_df.show(10)+--------------+-----------+-------------+-----------+----+
|     review_id|star_rating|helpful_votes|total_votes|vine|
+--------------+-----------+-------------+-----------+----+
| RTIS3L2M1F5SM|          5|            0|          0|   N|
| R1ZV7R40OLHKD|          5|            0|          0|   N|
|R3BH071QLH8QMC|          1|            0|          1|   N|
|R127K9NTSXA2YH|          3|            0|          0|   N|
|R32ZWUXDJPW27Q|          4|            0|          0|   N|
|R3AQQ4YUKJWBA6|          1|            0|          0|   N|
|R2F0POU5K6F73F|          5|            0|          0|   N|
|R3VNR804HYSMR6|          5|            0|          0|   N|
| R3GZTM72WA2QH|          5|            0|          0|   N|
| RNQOY62705W1K|          4|            0|          0|   N|
+--------------+-----------+-------------+-----------+----+
only showing top 10 rows

# Push to AWS RDS instance_____no_output_____

mode = "append"
jdbc_url="jdbc:postgresql://<endpoint>:5432/my_data_class_db"
config = {"user":"postgres", "password": "<password>", "driver":"org.postgresql.Driver"}_____no_output_____# Write review_id_df to table in RDS
review_id_df.write.jdbc(url=jdbc_url, table='review_id_table', mode=mode, properties=config)_____no_output_____# Write products_df to table in RDS
products_df.write.jdbc(url=jdbc_url, table='products', mode=mode, properties=config)_____no_output_____# Write customers_df to table in RDS
customers_df.write.jdbc(url=jdbc_url, table='customers', mode=mode, properties=config)_____no_output_____# Write vine_df to table in RDS
vine_df.write.jdbc(url=jdbc_url, table='vines', mode=mode, properties=config)_____no_output_____


# Notebook from Therealchainman/Algorithms-for-Automated-Driving
Path: book/CameraCalibration/VanishingPointCameraCalibration.ipynb

# Extrinsic Camera Calibration

Sensor calibration is one of the most important aspects of building a robot. Different sensors have different calibration procedures. Typically one can distinguish between intrinsic and extrinsic calibration. Intrinsic camera calibration tries to estimate the camera matrix $K$ (and potentially other camera-specific parameters like distortion coefficients). You can learn more about it in this [lecture by Cyril Stachniss](https://www.youtube.com/watch?v=Ou9Uj75DJX0), and this [OpenCV python tutorial](https://docs.opencv.org/4.5.2/dc/dbb/tutorial_py_calibration.html). For this chapter, we will be focusing on extrinsic camera calibration.

Refer back to the [From Pixels to meters chapter](../LaneDetection/InversePerspectiveMapping.ipynb). If you did the exercises of that chapter, you may recall that to convert the lane line pixels detected by the neural network to meters, you needed the camera height $h$ and the rotation matrix $\mathbf{R_{cr}}$, which describes how the camera is oriented with respect to the road reference frame.

```{admonition} Reminder: Road reference frame
:class: tip
The "road reference frame" was defined in [the chapter on camera basics](../LaneDetection/CameraBasics.ipynb), and it is attached to the vehicle.  The $Z$ direction in the road reference frame is always identical to the vehicle's forwards axis and the $Y$ axis is identical to the vehicle's "downwards axis". The name "road reference frame" could be a bit misleading. The only reason we used the word "road" for this reference frame, is because the $Z-X$-plane in this reference frame is identical to the road plane, which we approximate as flat.
```

$\mathbf{R_{cr}}$ was very easy to obtain through the CARLA simulator. But this is not that easy in real life. When you mount the camera to your vehicle, you can try to make it perfectly straight, which would correspond to $\mathbf{R_{cr}} = \mathbf{1}$, but you will always make a small mistake, which makes calibration necessary. The orientation with respect to the road could also change dynamically, which makes calibration challenging:
The road is not always flat. It can be banked, elevated or just uneven and therefore, the orientation of the car and the camera keeps changing with respect to the flat road frame. The orientation of the car and the camera may also change with respect to the road if the car breaks are hit hard or due to the car suspension. _____no_output_____Proper extrinsic calibration is essential for good vehicle control. The following plot shows two simulations in which the camera was mounted with a yaw of 4 degrees and a pitch of -4 degrees. In the first simulation, these yaw and pitch values were passed correctly to the `CameraGeometry` object, which was implemented in the chapter on Lane Detection. Hence, in this first simulation the calibraton was *perfect*. In the second simulation, default values of zero degrees were incorrectly assumed for both pitch and yaw. Hence, there was no calibration at all. _____no_output_____

import numpy as np
# numpy array with 474 elements
# the npy file format saves the numpy array data such as shape and data
# so that it can be reconstructed on any architecture.
cte_no_calib = np.load("../../data/cte_yaw_4_no_calib.npy")
cte_with_calib = np.load("../../data/cte_yaw_4_with_calib.npy")
cte_perfect_calib = np.load("../../data/cte_yaw_4_perfect_calib.npy")

import matplotlib.pyplot as plt
%matplotlib inline_____no_output_____get_time = lambda x: np.array(range(len(x)))/30
# create a numpy array with 474 elements ranging rom 1 to 473.
# Divide by 30 I guess because there are 30 fps.
# So that will give you the time for each frame.
plt.plot(get_time(cte_no_calib), cte_no_calib, label="no calib.")
plt.plot(get_time(cte_perfect_calib), cte_perfect_calib, label="perfect calib.")
plt.legend()
plt.xlabel("Time (seconds)")
plt.ylabel("Cross Track Error (meters)")
plt.xlim(0,15.8)
plt.tight_layout()_____no_output_____

As you can see in the plot, having no calibration at all makes the vehicle leave the center of the lane until it has a constant offset of around 40 cm from the middle of the lane. Why does this happen? The wrong yaw and pitch values lead to the computation of a wrong reference path, which is then passed to the controller.  In case of *perfect calibration* the results looks different. Here, the vehicle stays in the middle of the lane within an error of around 5 cm (except for one small disturbance at around 4 seconds which is due to a bad detection of the neural network).

But as already mentioned, in the real world we do not know the exact mounting of the camera. So what we will need to do is to estimate the mounting. In this chapter you will learn how to estimate the pitch and yaw angle. The method you will learn leads to simulation results denoted as "VP calib." in the plot below:_____no_output_____

get_time = lambda x: np.array(range(len(x)))/30
plt.plot(get_time(cte_no_calib), cte_no_calib, label="no calib.")
plt.plot(get_time(cte_perfect_calib), cte_perfect_calib, label="perfect calib.")
plt.plot(get_time(cte_with_calib), cte_with_calib, label="VP calib.")
plt.legend()
plt.xlabel("Time (seconds)")
plt.ylabel("Cross Track Error (meters)")
plt.xlim(0,15.8)
plt.tight_layout()_____no_output_____

First, the vehicle leaves the middle of the lane, but then it has gathered enough data to estimate yaw and pitch angles. Consequently, it can recover and stay close to the middle of the lane. Let's learn how this "VP calib." method works!_____no_output_____## Vanishing Point Method

Our aim is to find the rotation matrix $\mathbf{R_{cr}}$, which describes how the camera is rotated

We know that the rail tracks or the lane lines are mostly parallel, however, if we take an image of the track or road from a camera, we would observe that the track lines or the lanes are not parallel in the image. The point where these lines intersect in an image is known as the vanishing point.

```{figure} images/vanishing_point.png
---
width: 50%
name: vanishing_point_carla
---
```

```{figure} images/Vanishing_point.svg
---
width: 50%
name: vanishing_point
---
Image Source [wikipedia](https://en.wikipedia.org/wiki/Vanishing_point)
```_____no_output_____We know that in the world coordinate system, these parallel lines won't ever intersect. So we say that the vanishing point is at infinity. Assuming that the car's forwards direction is aligned with the lane lines, we can say that the lane lines intersect at $Z=\infty$, where $(X,Y,Z)$ are coordinates of a point in the road reference frame (reminder: the road reference frame is attached to the vehicle, and $Z$ is the vehicle's forwards axis).
```{admonition} Important assumption!
:class: warning
The assumption we just made is very important: The vehicle is aligned with the lane, and the lane is straight. Then, the intersection of the lane lines in the image (vanishing point), will give us information about the camera mounting, i.e., the orientation of the camera with respect to vehicle. Otherwise it would only tell us something about the orientation of the vehicle with respect to the lane lines.
```

Now recall the camera projection equation

$$
    \lambda \begin{pmatrix} u \\ v\\ 1 \end{pmatrix} = \mathbf{K} \begin{pmatrix} \mathbf{R_{cr}} | \mathbf{t} \end{pmatrix}  \begin{pmatrix} X \\ Y \\ Z \\ 1 \end{pmatrix}
$$

Since this is an equation in homogeneous coordinates, we can multiply both sides by $1/Z$ and absorb this number in $\lambda$ on the left hand side:

$$
    \lambda \begin{pmatrix} u \\ v\\ 1 \end{pmatrix} = \mathbf{K} \begin{pmatrix} \mathbf{R_{cr}} | \mathbf{t} \end{pmatrix}  \begin{pmatrix} X/Z \\ Y/Z \\ 1 \\ 1/Z \end{pmatrix}
$$

If we let $Z$ go to infinity, we obtain the coordinates $(u,v)$ of the vanishing point in image space:

$$
    \lambda \begin{pmatrix} u \\ v\\ 1 \end{pmatrix} = \mathbf{K} \begin{pmatrix} \mathbf{R_{cr}} | \mathbf{t} \end{pmatrix} \begin{pmatrix} 0 \\ 0 \\ 1 \\ 0 \end{pmatrix}
$$

We define $\mathbf{p_{\infty}} = (u,v,1)^T$ for the vanishing point and we write down the transformation matrix components to obtain

$$
    \lambda \mathbf{p_{\infty}} = \mathbf{K} \begin{pmatrix} R_{xx}  R_{xy}  R_{xz}  t_x \\ R_{yx}  R_{yy}  R_{yz}  t_y \\ R_{zx}  R_{zy}  R_{zz}  t_z \end{pmatrix} \begin{pmatrix} 0 \\ 0 \\ 1 \\ 0 \end{pmatrix}
$$

_____no_output_____Now, multiplying the transformation matrix with $(0,0,1,0)^T$ will eliminate the 1st, 2nd and 4th columnn, leaving us with only the 3rd column whose values are $R_{xz}$, $R_{yz}$  and $R_{zz}$. Let us denote this column by $\mathbf{r_{3}}$. Then

$$
    \lambda \mathbf{p_{\infty}} = \mathbf{K} \mathbf{r_{3}}
$$

Columns of a rotation matrix are always vectors with length 1 (unit vectors), so $\|\mathbf{r_{3}}\|=1$. Thus, $\lambda$ equals $\frac{1} {\| \mathbf{K}^{-1} \mathbf{p_{\infty}}\| }$ and we obtain

$$
    \mathbf{r_{3}} = \frac{\mathbf{K}^{-1} \mathbf{p_{\infty}}} {\| \mathbf{K}^{-1} \mathbf{p_{\infty}}\| }
$$ (label_r3_equals_invKpinf)

_____no_output_____The idea of the vanishing point method for extrinsic calibration is the following: We determine the vanishing point $(u,v)$ in the image as the point were the lanes lines intersect. This yields $\mathbf{p_{\infty}} = (u,v,1)^T$. Since we know the intrinsic matrix $\mathbf{K}$, we can compute $\mathbf{r_3}$ using the formula above (Eq. {eq}`label_r3_equals_invKpinf`). Of course $\mathbf{r_3}$ is only one of the three columns of the rotation matrix, but as we will show later, $\mathbf{r_3}$ contains enough information to determine the yaw and pitch angle of the rotation. If we then assume that the roll angle is zero (of course an approximation), we can compute the whole rotation matrix. _____no_output_____With this vanishing point method, we are able to only recover **yaw** and **pitch** of the camera. It makes intuitive sense that the vanishing point method is not able to recover **roll** and **translation** because the vanishing point is not affected by these two! Imagine looking at a star. The star's position is only going to change if you change the yaw and pitch of your head and is not affected by roll and translation.

```{figure} images/car_rpy.svg
---
width: 60%
name: coordinate_system
---
```_____no_output_____Let's understand this by an example from CARLA. First, let us inspect of changing the roll angle:

````{tabbed} roll = -20°
```{figure} images/images_vp/p-0-y-0-r-20.png
---
width: 90%
name: roll_minus_20
---
```
````

````{tabbed} roll = 0°
```{figure} images/images_vp/p-0-y-0.png
---
width: 90%
name: roll_0
---
```
````

````{tabbed} roll = 20°
```{figure} images/images_vp/p-0-y-0-r--20.png
---
width: 90%
name: roll_20
---
```
````

You can see that the vanishing point didn't change. Similarly, let's see if this also holds true for translation:_____no_output_____

````{tabbed} translation = -2 m
```{figure} images/images_vp/trans--2.png
---
width: 90%
name: translation_minus_2
---
```
````

````{tabbed} translation = 0 m
```{figure} images/images_vp/p-0-y-0.png
---
width: 90%
name: translation_0
---
```
````

````{tabbed} translation = 2 m
```{figure} images/images_vp/trans-2.png
---
width: 90%
name: translation_2
---
```
````

You can see that the vanishing point lies at the same position for all the above images. This shows that roll and translation won't affect the vanishing point._____no_output_____
Now let's see the effect of changing pitch

````{tabbed} pitch = -5°
```{figure} images/images_vp/p--5-y-0.png
---
width: 90%
name: pitch_minus_5
---
```
````

````{tabbed} pitch = 0°
```{figure} images/images_vp/p-0-y-0.png
---
width: 90%
name: pitch_0
---
```
````

````{tabbed} pitch = 5°
```{figure} images/images_vp/p-5-y-0.png
---
width: 90%
name: pitch_5
---
```
_____no_output_____and yaw:_____no_output_____

````{tabbed} yaw = -5°
```{figure} images/images_vp/p-0-y--10.png
---
width: 90%
name: yaw_minus_5
---
```
````

````{tabbed} yaw = 0°
```{figure} images/images_vp/p-0-y-0.png
---
width: 90%
name: yaw_0
---
```
````

````{tabbed} yaw = 5°
```{figure} images/images_vp/p-0-y-10.png
---
width: 90%
name: yaw_5
---
```
_____no_output_____As you can see, the vanishing point shifted in this case. Next let us explore how we can determine yaw and pitch from $\mathbf{r_3}$_____no_output_____## Deriving yaw and pitch from the vanishing point_____no_output_____We discussed the rotation matrix in detail in the section [Basis of Image Formation](../LaneDetection/CameraBasics.ipynb) within the Lane Detection chapter. The rotation matrix parametrized by yaw, pitch and roll is given by

$$
    {R} = R_{yaw}R_{pitch}R_{roll}
$$

$$
   {R} = \begin{pmatrix} \cos(\gamma)\cos(\beta) + \sin(\alpha)\sin(\gamma)\sin(\beta)  & \cos(\gamma)\sin(\alpha)\sin(\beta) -\cos(\beta)\sin(\gamma) & -\cos(\alpha)\sin(\beta)  \\
                          \cos(\alpha)\sin(\gamma) & \cos(\alpha)\cos(\gamma) & \sin(\alpha)  \\
                          \cos(\gamma)\sin(\beta) -\cos(\beta)\sin(\alpha)\sin(\gamma)  & -\cos(\gamma)\cos(\beta)\sin(\alpha) -\sin(\gamma)\sin(\beta)  & \cos(\alpha)\cos(\beta) \\
                            \end{pmatrix}
$$

$\alpha, \beta, \gamma$ are the pitch, yaw, and roll angles respectively._____no_output_____The third column of this rotation matrix is

$$
    \begin{pmatrix}  -\cos(\alpha)\sin(\beta) \\
                            \sin(\alpha) \\
                            \cos(\alpha)\cos(\beta) \\
                            \end{pmatrix}  = \mathbf{r_{3}} = \begin{pmatrix} R_{xz} \\ R_{yz} \\ R_{zz} \end{pmatrix}
$$ (label_r3_RRR)
_____no_output_____If we determine the vanishing point $(u,v)$ in the image, we know $\mathbf{p_\infty} = (u,v,1)^T$ and hence we can compute the value of $\mathbf{r_3}= (R_{xz}, R_{yz}, R_{zz})^T$ from Eq. {eq}`label_r3_equals_invKpinf`:  $\mathbf{r_{3}} = \mathbf{K}^{-1} \mathbf{p_{\infty}} / \| \mathbf{K}^{-1} \mathbf{p_{\infty}}\| $._____no_output_____Solving Eq. {eq}`label_r3_RRR` for $\alpha$ and $\beta$ we get:

$$
    \alpha = \sin^{-1}{R_{yz}}
$$ (label_eq_alpha)

$$
    \beta = -\tan^{-1}\frac{R_{xz}}{{R_{zz}}}
$$ (label_eq_beta)

We have thus derived pitch and yaw from the vanishing point! Next, let us see how we can determine the vanishing point from an image using our lane detector._____no_output_____## Vanishing point from lane lines_____no_output_____

import cv2
import numpy as np
import copy
%matplotlib inline
import matplotlib.pyplot as plt_____no_output_____img = cv2.imread('images/test.png')[:, :, (2, 1, 0)]
plt.imshow(img);_____no_output_____

This image was collected from CARLA having a pitch of -5 degrees and yaw of -2 degrees. We aim to obtain these values of pitch and yaw just from the image. For that we will be using our lane detector network trained previously to find the left and right lane boundaries. The intersection of them will lead to the vanishing point. _____no_output_____

import sys, copy
sys.path.append('../../code')
from solutions.lane_detection.lane_detector import LaneDetector

# It is a MobileNetV3 which is for semantic segmentation
model_path = "../../code/solutions/lane_detection/fastai_model.pth"
ld = LaneDetector(model_path=model_path)_____no_output_____# print(ld.model)
print(img.shape, img.dtype)(512, 1024, 3) uint8

First let us run our neural network to compute for each pixel the probability whether it is background, left boundary or right boundary:_____no_output_____

background_prob, left_prob, right_prob = ld.detect(img)_____no_output_____print(left_prob.shape)(512, 1024)

All pixels where `left_prob` is large (let's say larger than `0.5`), can be considered as pixels of the left boundary. Let us make all those pixels blue to get a nice visualization. Similarly we make the right boundary pixels red:_____no_output_____

img_with_detection = copy.copy(img)
img_with_detection[left_prob > 0.5] = [0,0,255] # blue
img_with_detection[right_prob > 0.5] = [255,0,0] # red
plt.imshow(img_with_detection)
plt.xlabel('$u$'); plt.ylabel('$v$');_____no_output_____

To determine the intersection of the blue and red line, we describe each of them by an equation $v(u)=m u+c$, where $u$ and $v$ are the pixel coordinates (see the axes labels in the image above). Then, it is a high school math problem to find the intersection point.

Getting the line equation $v(u)=m u+c$ for the left lane line from our `prob_left` matrix is actually just two lines of python code:_____no_output_____

# get a list of all u and v values for which left_prob[v,u] > 0.5
v_list, u_list = np.nonzero(left_prob >0.5)

# Fit a polynomial of degree 1 (which is a line) to the list of pixels
poly_left = np.poly1d(np.polyfit(u_list, v_list, deg=1))_____no_output_____print(v_list)
print(u_list)
print(poly_left)
c, res= np.polynomial.polynomial.Polynomial.fit(u_list, v_list, deg=1, full=True)
print(res[0].item())
print(np.polynomial.polynomial.Polynomial(c))[172 172 172 ... 511 511 511]
[517 518 519 ...  75  76  77]

-0.7446 x + 560.8
27282.202454342998
341.90919916400634 - 168.29046804507675·x¹

Let's add two more lines of code for the right boundary_____no_output_____

v_list, u_list = np.nonzero(right_prob >0.5)
poly_right = np.poly1d(np.polyfit(u_list, v_list, deg=1))_____no_output_____

And finally some code to visualize our lines_____no_output_____

def show_img_and_lines():
    plt.imshow(img)
    u = np.arange(0, 1024)
    # this uses inputs for the u to get the v value, and so we are using
    # the polynomial equations to compute all the vs.
    # then plot them
    v_left = poly_left(u)
    v_right = poly_right(u)
    plt.plot(u, v_left, color="b")
    plt.plot(u, v_right, color="r")
    plt.xlim(0, 1024); plt.ylim(512, 0)
    plt.xlabel('$u$'); plt.ylabel('$v$');
show_img_and_lines()_____no_output_____

In the exercises you will write code that determines the coordinates of the intersection points of two lines. We can import it here to determine the vanishing point:_____no_output_____

from exercises.camera_calibration.calibrated_lane_detector import get_intersection
u_i, v_i = get_intersection(poly_left, poly_right)
# These are the coordinates of the intersection x, y
show_img_and_lines()
plt.scatter([u_i], [v_i], marker="o", s=100, color="y", zorder=3);_____no_output_____

Finally, we can find pitch and yaw of the camera form the vanishing point using Eqs. {eq}`label_r3_equals_invKpinf`, {eq}`label_eq_alpha`, {eq}`label_eq_beta`. Again you will write some code for this in the exercises._____no_output_____

from exercises.camera_calibration.calibrated_lane_detector import get_py_from_vp
pitch, yaw = get_py_from_vp(u_i, v_i, ld.cg.intrinsic_matrix)

print(f'yaw degrees:   %.2f' % np.rad2deg(yaw))
print(f'pitch degrees: %.2f' % np.rad2deg(pitch))yaw degrees:   1.98
pitch degrees: -5.00

This is very close to the correct values of -2 and -5! You can expect small errors in yaw and pitch because small errors in lane detection can lead to large errors for the vanishing point. Also the car will not always be perfectly aligned with the lane lines as it was in our example image. Therefore, the readings are mostly taken for a period of time from a video stream and then averaged.

This is why several automated driving systems like Tesla's autopilot or comma ai's openpilot require you to drive straight for a while to calibrate the cameras before you can engage the software to drive the car._____no_output_____For the exercise, you will write a `CalibratedLaneDetector` class which inherits from the `LaneDetector` class. In each cycle it determines the vanishing points and uses that to estimate yaw and pitch. It stores this information to perform an averaging over time. Additionally, the `CalibratedLaneDetector` makes sure not to use images for averaging when the lane lines are curved, since the vanishing point should only be computed from straight lines. _____no_output_____## Exercise

We provide you with a video. Your task is to use the lane detection network to calibrate the camera and output the yaw and pitch values. The video can be found in the `data` folder and is named `calibration_video.mp4`.

To start working on the exercises, open `code/tests/camera_calibration/calibrated_lane_detector.ipynb` and follow the instructions in that notebook._____no_output_____If you completed the tasks from that notebook, you can optionally check your code with a CARLA simulation. Go to the parent folder of the `code` folder and run

```bash
python -m code.tests.camera_calibration.carla_sim --help
```

to see instructions on how to run the simulation. _____no_output_____


# Notebook from mjosch/Born-to-slide
Path: Analysis_software/Figures_1_S1/Figures_1H_S1I-K.ipynb

# Born to slide: mobile origin licensing factors confer resistance to transcription conflicts

## Jupyter notebooks

#### Figures:
+ 1H
+ S1I
+ S1J
+ S1K

#### Archives:
+ MCM_high-salt_diffusion.yama

#### Additional data:
+ none_____no_output_____### Initialize ImageJ_____no_output_____

import imagej
ij = imagej.init('/Applications/Fiji.app')
ij.getVersion()_____no_output_____

### Imports_____no_output_____

import sys
sys.path.insert(0, '..')
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import re
import awesome_data
from marspy.convert.archive import DnaMoleculeArchive, instantiate_archive, describe_archives
from marspy.stats import bootstrap, calc_ci
from tqdm.notebook import tqdm
from functools import reducemarspy initialized.

### Select archives_____no_output_____

selected_archives = ['MCM_high-salt_diffusion.yama']_____no_output_____

### Figure style and output directory_____no_output_____

from figure_style import *_____no_output_____set_style_paper()_____no_output_____dir_out = '/Volumes/pool-duderstadt/Matthias/Manuscript/Figures/SVGs/Figure1/'
dir_out_____no_output_____

### Instantiate selected archives_____no_output_____

for archive in tqdm(selected_archives):
    instantiate_archive(archive, awesome_data.datasets)_____no_output_____

### Archives Overview_____no_output_____

describe_archives(DnaMoleculeArchive.instances)_____no_output_____

### Functions_____no_output_____

def calc_msd(df, prefix, msd_col, time_col, tau=1, start=np.nan, end=np.nan):
    '''
    Calculates mean square displacement of molecules.
    Remove pairs with missing slices in interval, do not drop NaN values!
    *df: dataframe
    *prefix: protein prefix
    *msd_col: column in df to calculate msd (without protein prefix!)
    *time_col: time column in df (without protein prefix!)
    *tau: time delta used to calculate msd (without protein prefix!)

    returns (msd, dtime)
    msd: mean squared error
    dtime: mean time difference underlying msd calculation
    '''
    # if not start / end was specified by used, set to max range
    if np.isnan(start):
        start=df[prefix+time_col].iloc[0]

    if np.isnan(end):
        end=df[prefix+time_col].iloc[-1]

    # will be a more accurate way to calculate MSD due to existing NaN values
    diffs = []
    dtimes = []
    for row in df.index:
            try:
                # check if sliding window range
                if (df.loc[row, prefix+time_col] >= start and df.loc[row+tau, prefix+time_col] <= end):

                    # check if we have consecutive frames (slices) which are not null
                    if (((df.loc[row+tau, prefix+'slice'] - df.loc[row, prefix+'slice']) == tau) and not
                        ((np.isnan(df.loc[row+tau, prefix+msd_col]) or np.isnan(df.loc[row, prefix+msd_col])))):

                        diffs.append(df.loc[row+tau, prefix+msd_col] - df.loc[row, prefix+msd_col])
                        dtimes.append(df.loc[row+tau, prefix+time_col] - df.loc[row, prefix+time_col])

                    else:
                        #skip to next row
                        continue

            # end of df
            except KeyError:
                break

    return np.mean(np.square(diffs)), np.mean(dtimes)_____no_output_____

### Generate main df_____no_output_____

def setup_pandas_df(archive_instances):
    '''Sets up a 2-level df with all relevant information'''
    #indices will be UIDs
    #outside: set comprehension to cover all possible prefixes in all instances
    prefixes = set()
    for archive in archive_instances:
        prefixes.update(archive.prefixes)
    col_1 = sorted(list(prefixes))
    #inside
    col_2 = ['lifetime', 'initial_intensity', 'sliding_start', 'sliding_end', 'msd', 'dtime', 'd_coeff']

    hier_index = pd.MultiIndex.from_product([col_1,col_2],names=['molecule','properties'])

    return pd.DataFrame(columns=hier_index)
    _____no_output_____df = setup_pandas_df(DnaMoleculeArchive.instances)

# sort MultiIndexCols for performance
df.sort_index(axis=1, inplace=True)
df.head()
_____no_output_____# fill dataframe
for archive in tqdm(DnaMoleculeArchive.instances):

    for molecule in archive.molecules:
        # general columns first
        # this could be generalized with mol prefixes
        df.loc[molecule.uid,'number_mcm'] = molecule.params['Number_MCM']
        df.loc[molecule.uid,'tags'] = reduce(lambda tag1,tag2: tag1+','+tag2, molecule.tags)
        df.loc[molecule.uid,'nucleotide'] = archive.nucleotide
        df.loc[molecule.uid,'dna_length'] = molecule.calc_length_dna()

        for prefix in molecule.prefixes:

            # lifetime (in frames) defined on tracking length
            df.loc[molecule.uid,(prefix,'lifetime')] = len(molecule.df[prefix+'y'].dropna())

            # take intensity from mean of first 20 frames
            df.loc[molecule.uid,(prefix,'initial_intensity')] = molecule.df[prefix+'Intensity'].iloc[0:20].mean()

            # a specific region was defined
            if 'sliding_'+prefix in list(filter(lambda region: 'sliding' in region.name, molecule.regions))[0].name:

                # temporary variables
                start = archive.archive_link.get(molecule.uid).getRegion('sliding_'+prefix).getStart()
                end = archive.archive_link.get(molecule.uid).getRegion('sliding_'+prefix).getEnd()

                # sliding window start & end
                df.loc[molecule.uid,(prefix,'sliding_start')] = start
                df.loc[molecule.uid,(prefix,'sliding_end')] = end

                # MSD (kbp^2) & dtime (s)

                msd, dtime = calc_msd(df=molecule.df, prefix=prefix, msd_col='Position_on_DNA', time_col='Time (s)',
                                      tau=1, start=start, end=end)
                df.loc[molecule.uid,(prefix,'msd')] = msd/(1000**2)
                df.loc[molecule.uid,(prefix,'dtime')] = dtime

                # diffusion coefficient (kpb^2/s)
                df.loc[molecule.uid,(prefix,'d_coeff')] = df.loc[molecule.uid,(prefix,'msd')] / (2*df.loc[molecule.uid,(prefix,'dtime')])

            #no region => molecule was not sliding
            else:
                pass

# convert tags back to list
df['tags'] = df['tags'].apply(lambda tags: tags.split(','))

# again sort MultiIndexCols for performance
df.sort_index(axis=1, inplace=True)

# fix data types
df = df.infer_objects()
_____no_output_____df.head()_____no_output_____

## Figures_____no_output_____### Figure 1H: Diffusion coefficient salt-sliding MCMs_____no_output_____

protein = 'MCM'

fig,axes = plt.subplots(figsize=(0.5,2))

sns.barplot(y=df['MCM_1_']['d_coeff'],palette=palettes[protein][1:2], lw=1 , edgecolor='.15', capsize=.3,facecolor=(1, 1, 1, 0),
            errwidth=1,ax=axes,n_boot=10000, seed=42, ci=68, errcolor='.15')
sns.swarmplot(y=df['MCM_1_']['d_coeff'],palette=palettes[protein][1:2],ax=axes)

axes.set_ylim(0,50)
#axes.set_title('Observed MCM diffusion in 0.5 M NaCl')
axes.set_ylabel('D (kbp2/s)')
axes.text(0.5, 0.95, f"n = {len(df['MCM_1_']['d_coeff'])}",
        verticalalignment='top', horizontalalignment='center',
        transform=axes.transAxes)
axes.text(0.6, 0.7, f"D = {round(df['MCM_1_']['d_coeff'].mean(),1)} +/- \n{round(df['MCM_1_']['d_coeff'].sem(),1)} kbp2/s",
        verticalalignment='top', horizontalalignment='left',
        transform=axes.transAxes)
axes.set_xlabel('MCM diffusion\ncoefficient')
# tidy up
#plt.tight_layout()
sns.despine()

# save
plt.savefig(dir_out+"1H_MCM_diffusionCoeff.svg",transparent=True)_____no_output_____

### Figure S1I: Probability of MCM sliding in high salt_____no_output_____

import scyjava as sc_____no_output_____total_mcm_count = 0
for metadata in DnaMoleculeArchive.instances[0].metadata_uids:
    total_mcm_count += dict(sc.to_python(DnaMoleculeArchive.instances[0].archive_link.getMetadata(metadata).
                                         getParameters()))['TotalMcmCount']
total_mcm_count_____no_output_____df_pslide = pd.DataFrame(data={'non-diffusive':(total_mcm_count-len(DnaMoleculeArchive.instances[0])) / total_mcm_count,
                               'diffusive':len(DnaMoleculeArchive.instances[0]) / total_mcm_count,
                               'total':total_mcm_count}.items(),
                         columns = ['population','count'])
df_pslide.set_index('population',inplace=True)_____no_output_____df_pslide_____no_output_____protein = 'MCM'
fig,axes = plt.subplots()

df_pslide.iloc[0:-1].T.plot(kind='bar',stacked=True, figsize=(0.5,2),ax=axes,color=palettes['qualitative'][0::2],lw=0.5,edgecolor='.15')
axes.set_ylabel("Probability")
#axes.set_title("MCM populations")
axes.set_ylim([0,1.19])
axes.legend(title='MCM population',frameon=False, bbox_to_anchor=[1, 1], loc='upper left')
for item in axes.get_xticklabels():
    item.set_rotation(45)

labels = [int(df_pslide.loc['total',column]) for column in df_pslide.columns]

#calculate y offset to label bars on top
dy = []
for column in df_pslide.columns:
    temp_sum = 0
    for index,row in df_pslide.iterrows():
        if (index == 'total'):
            continue
        else:
            temp_sum+=df_pslide.loc[index,column]
    dy.append(temp_sum)
rects = axes.patches
counter = 0
for rect, label in zip(rects, labels):
    height = dy[counter]
    axes.text(rect.get_x() + rect.get_width()/2, height+0.05, s=f"n = {label}", ha='center', va='bottom')
    axes.text(rect.get_x() + rect.get_width()/2, 0.45, s=f"{round(df_pslide.loc['non-diffusive','count']*100,1)} %", ha='center', va='bottom')
    axes.text(rect.get_x() + rect.get_width()/2, 0.95, s=f"{round(df_pslide.loc['diffusive','count']*100,1)} %", ha='center', va='bottom')
    counter+=1

axes.set_xticklabels([])
#tidy up
#fig.tight_layout()
sns.despine()

#save
plt.savefig(dir_out+"S1I_MCM_Populations.svg",transparent=True)_____no_output_____# estimate standard error by bootstrapping (skip)_____no_output_____df_pslide.columns = ['probability']
df_pslide_____no_output_____for row in df_pslide.index[:-1]:
    bootstrap_data = ([0 for _ in range(int((1-df_pslide.loc[row,'probability'])*df_pslide.loc['total','probability']))] +
                          [1 for _ in range(int(df_pslide.loc[row,'probability']*df_pslide.loc['total','probability']))])
    bootstrap_means = bootstrap(bootstrap_data,n_boot=10000)
    df_pslide.loc[row,'bs_mean'] = np.mean(bootstrap_means)
    df_pslide.loc[row,'standard_error'] = np.std(bootstrap_means)
    df_pslide.loc[row,'68_ci_lower'], df_pslide.loc[row,'68_ci_upper'] = calc_ci(bootstrap_means,ci=68)
    df_pslide.loc[row,'95_ci_lower'], df_pslide.loc[row,'95_ci_upper'] = calc_ci(bootstrap_means,ci=95)_____no_output_____df_pslide_____no_output_____

### Figure S1J: MSD vs dtime_____no_output_____

# Generate separate dfs with MSD for different tau
dfs_msd = []
for molecule in DnaMoleculeArchive.instances[0].molecules:
    df_temp = pd.DataFrame()
    for i in range(0,11):

        msd, dtime = calc_msd(molecule.df, prefix='MCM_1_',
                              msd_col='Position_on_DNA', time_col='Time (s)',
                              tau=i, start=df.loc[molecule.uid,'MCM_1_']['sliding_start'],
                              end=df.loc[molecule.uid,'MCM_1_']['sliding_end'])
        df_temp.loc[i,'dtime'] = dtime
        df_temp.loc[i,'msd'] = msd/(1000**2)
    dfs_msd.append(df_temp)_____no_output_____sns.set_palette(palettes['MCM'])_____no_output_____protein = 'MCM'
fig,axes = plt.subplots()

for item in dfs_msd[:]:
    sns.lineplot(x='dtime', y='msd', data=item, ax=axes)
axes.set_xlim(0,1)
axes.set_ylim(0,70)
#axes.set_title('MSD vs $\Delta$ t')
axes.set_xlabel('dt (s)')
axes.set_ylabel('MSD (kbp2)')
axes.text(0.05, 0.95, f'n = {len(dfs_msd[:])}',
        verticalalignment='top', horizontalalignment='left',
        transform=axes.transAxes)

sns.despine(trim=True)

#save
plt.savefig(dir_out+"S1J_MSD_vs_time.svg",transparent=True)
_____no_output_____

### Figure S1K: Example trajectories diffusive MCM DH_____no_output_____

protein = 'MCM'

fig,axes = plt.subplots(1,2,figsize=(5.34,2),sharey=True)

# panel A - Example Molecule free

for prefix in DnaMoleculeArchive.instances[0].molecules[11].prefixes:
    try:
        sns.lineplot(x=prefix + 'Time (s)', y=DnaMoleculeArchive.instances[0].molecules[11].df[prefix + 'Position_on_DNA']/1000, data=DnaMoleculeArchive.instances[0].molecules[11].df,ax=axes[0],
                    color=palettes[protein][1])
    except:
        continue
#axes[0].set_xlim(0,25)
axes[0].set_ylim(0,21.5)
#axes[0].set_title('Example Molecule 1')
axes[0].set_xlabel('Time (s)')
axes[0].set_ylabel('Position on DNA (kb)')

# panel B -  Example Molecule with second MCM

for prefix in DnaMoleculeArchive.instances[0].molecules[15].prefixes:
    try:
        sns.lineplot(x=prefix + 'Time (s)', y=DnaMoleculeArchive.instances[0].molecules[15].df[prefix + 'Position_on_DNA']/1000, data=DnaMoleculeArchive.instances[0].molecules[15].df,ax=axes[1],
                    color=palettes[protein][1])
    except:
        continue
#axes[1].set_xlim(0,25)
axes[1].set_ylim(0,21.5)
#axes[1].set_title('Example Molecule 2')
axes[1].set_xlabel('Time (s)')
axes[1].set_ylabel('Position on DNA (kb)')

sns.despine(trim=True)

#save
plt.savefig(dir_out+"S1K_ExampleMolecule_sliding.svg",transparent=True)
_____no_output_____


# Notebook from isabella232/jupyterlab-omnisci
Path: notebooks/interactive-slider.ipynb

# Ibis+Vega+Altair using an interactive slider

We will try to reproduce [this](https://altair-viz.github.io/gallery/us_population_over_time.html)
example from the Altair gallery, but with lazily fetching data as the user interacts with the slider.
To keep ourselves honest, we'll be putting the data in a SQLite database.

First, let's show the original example, without any modifications:_____no_output_____

import altair as alt
from vega_datasets import data

source = data.population.url

pink_blue = alt.Scale(domain=('Male', 'Female'),
                      range=["steelblue", "salmon"])

slider = alt.binding_range(min=1900, max=2000, step=10)
select_year = alt.selection_single(name="year", fields=['year'],
                                   bind=slider, init={'year': 2000})

alt.Chart(source).mark_bar().encode(
    x=alt.X('sex:N', title=None),
    y=alt.Y('people:Q', scale=alt.Scale(domain=(0, 12000000))),
    color=alt.Color('sex:N', scale=pink_blue),
    column='age:O'
).properties(
    width=20
).add_selection(
    select_year
).transform_calculate(
    "sex", alt.expr.if_(alt.datum.sex == 1, "Male", "Female")
).transform_filter(
    select_year
).configure_facet(
    spacing=8
)_____no_output_____

## Loading the data into a database

We begin our lazy-fetching example by downloading the data and putting it into a SQLite database:_____no_output_____

import sqlalchemy
dbfile = 'population.db'
engine = sqlalchemy.create_engine(f'sqlite:///{dbfile}')

import pandas as pd
df = pd.read_json(data.population.url)
df.to_sql('population', engine, if_exists='replace')_____no_output_____

Now, let's create an ibis connection to this database and verify that the data is there:_____no_output_____

import ibis

connection = ibis.sqlite.connect(dbfile)
connection.list_tables()_____no_output_____

We can use inspect the data using this ibis connection:_____no_output_____

pop = connection.table('population')
pop.head().execute()_____no_output_____

## Making an interactive plot

We are now ready to make an interactive plot using this database connection.
We can reuse the same objects for `pink_blue`, `slider`, and `select_year`, as they are independent of the data source.
The `Chart` specifiation is completely identical, except that instead of the pandas dataframe,
we supply it with the Ibis sqlite connection:_____no_output_____

import jupyterlab_omnisci.vega_ibis

alt.renderers.enable('ibis')
alt.data_transformers.enable('ibis')

alt.Chart(pop).mark_bar().encode(
    x=alt.X('sex:N', title=None),
    y=alt.Y('people:Q', scale=alt.Scale(domain=(0, 12000000))),
    color=alt.Color('sex:N', scale=pink_blue),
    column='age:O'
).properties(
    width=20
).add_selection(
    select_year
).transform_calculate(
    "sex", alt.expr.if_(alt.datum.sex == 1, "Male", "Female")
).transform_filter(
    select_year
).configure_facet(
    spacing=8
)_____no_output_____