# Notebook from vishwatejharer/basics
Path: notebooks/07_Linear_Regression/07_TF_Linear_Regression.ipynb

# Linear Regression

In this lesson we will learn about linear regression. We will understand the basic math behind it, implement it in just NumPy and then in native [TensorFlow](https://www.tensorflow.org/install) and then with Keras._____no_output_____

_____no_output_____# Overview_____no_output_____Our goal is to learn a linear model $\hat{y}$ that models $y$ given $X$.

$\hat{y} = XW + b$
* $\hat{y}$ = predictions | $\in \mathbb{R}^{NX1}$ ($N$ is the number of samples)
* $X$ = inputs | $\in \mathbb{R}^{NXD}$ ($D$ is the number of features)
* $W$ = weights | $\in \mathbb{R}^{DX1}$
* $b$ = bias | $\in \mathbb{R}^{1}$ _____no_output_____* **Objective:**  Use inputs $X$ to predict the output $\hat{y}$ using a linear model. The model will be a line of best fit that minimizes the distance between the predicted (model's output) and target (ground truth) values. Training data $(X, y)$ is used to train the model and learn the weights $W$ using gradient descent.
* **Advantages:**
  * Computationally simple.
  * Highly interpretable.
  * Can account for continuous and categorical features.
* **Disadvantages:**
  * The model will perform well only when the data is linearly separable (for classification).
  * Usually not used for classification and only for regression.
* **Miscellaneous:** You can also use linear regression for binary classification tasks where if the predicted continuous value is above a threshold, it belongs to a certain class. But we will cover better techniques for classification in future lessons and will focus on linear regression for continuous regression tasks only.
_____no_output_____# Generate data_____no_output_____We're going to create some simple dummy data to apply linear regression on. It's going to create roughly linear data (`y = 3.5X + noise`); the random noise is added to create realistic data that doesn't perfectly align in a line. Our goal is to have the model converge to a similar linear equation (there will be slight variance since we added some noise)._____no_output_____

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt_____no_output_____SEED = 1234
NUM_SAMPLES = 50_____no_output_____# Set seed for reproducibility
np.random.seed(SEED)_____no_output_____# Generate synthetic data
def generate_data(num_samples):
    """Generate dummy data for linear regression."""
    X = np.array(range(num_samples))
    random_noise = np.random.uniform(-10,20,size=num_samples)
    y = 3.5*X + random_noise # add some noise
    return X, y_____no_output_____# Generate random (linear) data
X, y = generate_data(num_samples=NUM_SAMPLES)
data = np.vstack([X, y]).T
print (data[:5])[[ 0.         -4.25441649]
 [ 1.         12.16326313]
 [ 2.         10.13183217]
 [ 3.         24.06075751]
 [ 4.         27.39927424]]
# Load into a Pandas DataFrame
df = pd.DataFrame(data, columns=['X', 'y'])
X = df[['X']].values
y = df[['y']].values
df.head()_____no_output_____# Scatter plot
plt.title("Generated data")
plt.scatter(x=df['X'], y=df['y'])
plt.show()_____no_output_____

# NumPy_____no_output_____Now that we have our data prepared, we'll first implement linear regression using just NumPy. This will let us really understand the underlying operations._____no_output_____## Split data_____no_output_____Since our task is a regression task, we will randomly split our dataset into **three** sets: train, validation and test data splits.

* train: used to train our model.
* val : used to validate our model's performance during training.
* test: used to do an evaluation of our fully trained model._____no_output_____

TRAIN_SIZE = 0.7
VAL_SIZE = 0.15
TEST_SIZE = 0.15
SHUFFLE = True_____no_output_____# Shuffle data
if SHUFFLE:
    indices = list(range(NUM_SAMPLES))
    np.random.shuffle(indices)
    X = X[indices]
    y = y[indices]_____no_output_____

**NOTE**: Be careful not to shuffle X and y separately because then the inputs won't correspond to the outputs!_____no_output_____

# Split indices
train_start = 0
train_end = int(0.7*NUM_SAMPLES)
val_start = train_end
val_end = int((TRAIN_SIZE+VAL_SIZE)*NUM_SAMPLES)
test_start = val_end_____no_output_____# Split data
X_train = X[train_start:train_end]
y_train = y[train_start:train_end]
X_val = X[val_start:val_end]
y_val = y[val_start:val_end]
X_test = X[test_start:]
y_test = y[test_start:]
print (f"X_train: {X_train.shape}, y_train: {y_train.shape}")
print (f"X_val: {X_val.shape}, y_test: {y_val.shape}")
print (f"X_test: {X_test.shape}, y_test: {y_test.shape}")X_train: (35, 1), y_train: (35, 1)
X_val: (7, 1), y_test: (7, 1)
X_test: (8, 1), y_test: (8, 1)

## Standardize data_____no_output_____We need to standardize our data (zero mean and unit variance) or our models can optimize quickly when we are training.

$z = \frac{x_i - \mu}{\sigma}$
* $z$ = standardized value
* $x_i$ = inputs
* $\mu$ = mean
* $\sigma$ = standard deviation_____no_output_____

def standardize_data(data, mean, std):
    return (data - mean)/std_____no_output_____# Determine means and stds
X_mean = np.mean(X_train)
X_std = np.std(X_train)
y_mean = np.mean(y_train)
y_std = np.std(y_train)_____no_output_____

We need to treat the validation and test sets as if they were hidden datasets. So we only use the train set to determine the mean and std to avoid biasing our training process._____no_output_____

# Standardize
X_train = standardize_data(X_train, X_mean, X_std)
y_train = standardize_data(y_train, y_mean, y_std)
X_val = standardize_data(X_val, X_mean, X_std)
y_val = standardize_data(y_val, y_mean, y_std)
X_test = standardize_data(X_test, X_mean, X_std)
y_test = standardize_data(y_test, y_mean, y_std)_____no_output_____# Check (means should be ~0 and std should be ~1)
print (f"X_train: mean: {np.mean(X_train, axis=0)[0]:.1f}, std: {np.std(X_train, axis=0)[0]:.1f}")
print (f"y_train: mean: {np.mean(y_train, axis=0)[0]:.1f}, std: {np.std(y_train, axis=0)[0]:.1f}")
print (f"X_val: mean: {np.mean(X_val, axis=0)[0]:.1f}, std: {np.std(X_val, axis=0)[0]:.1f}")
print (f"y_val: mean: {np.mean(y_val, axis=0)[0]:.1f}, std: {np.std(y_val, axis=0)[0]:.1f}")
print (f"X_test: mean: {np.mean(X_test, axis=0)[0]:.1f}, std: {np.std(X_test, axis=0)[0]:.1f}")
print (f"y_test: mean: {np.mean(y_test, axis=0)[0]:.1f}, std: {np.std(y_test, axis=0)[0]:.1f}")X_train: mean: -0.0, std: 1.0
y_train: mean: 0.0, std: 1.0
X_val: mean: -0.5, std: 0.5
y_val: mean: -0.6, std: 0.5
X_test: mean: -0.6, std: 0.9
y_test: mean: -0.6, std: 0.9

## Weights_____no_output_____Our goal is to learn a linear model $\hat{y}$ that models $y$ given $X$.

$\hat{y} = XW + b$
* $\hat{y}$ = predictions | $\in \mathbb{R}^{NX1}$ ($N$ is the number of samples)
* $X$ = inputs | $\in \mathbb{R}^{NXD}$ ($D$ is the number of features)
* $W$ = weights | $\in \mathbb{R}^{DX1}$
* $b$ = bias | $\in \mathbb{R}^{1}$ _____no_output_____1. Randomly initialize the model's weights $W$._____no_output_____

INPUT_DIM = X_train.shape[1] # X is 1-dimensional
OUTPUT_DIM = y_train.shape[1] # y is 1-dimensional_____no_output_____# Initialize random weights
W = 0.01 * np.random.randn(INPUT_DIM, OUTPUT_DIM)
b = np.zeros((1, 1))
print (f"W: {W.shape}")
print (f"b: {b.shape}")W: (1, 1)
b: (1, 1)

## Model_____no_output_____2. Feed inputs $X$ into the model to receive the predictions $\hat{y}$.
  * $\hat{y} = XW + b$_____no_output_____

# Forward pass [NX1] · [1X1] = [NX1]
y_pred = np.dot(X_train, W) + b
print (f"y_pred: {y_pred.shape}")y_pred: (35, 1)

## Loss_____no_output_____3. Compare the predictions $\hat{y}$ with the actual target values $y$ using the objective (cost) function to determine the loss $J$. A common objective function for linear regression is mean squarred error (MSE). This function calculates the difference between the predicted and target values and squares it.

  * $J(\theta) = MSE = \frac{1}{N} \sum_{i-1}^{N} (y_i - \hat{y}_i)^2 $
    * ${y}$ = ground truth | $\in \mathbb{R}^{NX1}$
    * $\hat{y}$ = predictions | $\in \mathbb{R}^{NX1}$_____no_output_____

# Loss
N = len(y_train)
loss = (1/N) * np.sum((y_train - y_pred)**2)
print (f"loss: {loss:.2f}")loss: 0.99

## Gradients_____no_output_____4. Calculate the gradient of loss $J(\theta)$ w.r.t to the model weights.
  * $J(\theta) = \frac{1}{N} \sum_i (y_i - \hat{y}_i)^2  = \frac{1}{N}\sum_i (y_i - X_iW)^2 $
    * $\frac{\partial{J}}{\partial{W}} = -\frac{2}{N} \sum_i (y_i - X_iW) X_i = -\frac{2}{N} \sum_i (y_i - \hat{y}_i) X_i$
    * $\frac{\partial{J}}{\partial{b}} = -\frac{2}{N} \sum_i (y_i - X_iW)1 = -\frac{2}{N} \sum_i (y_i - \hat{y}_i)1$_____no_output_____

# Backpropagation
dW = -(2/N) * np.sum((y_train - y_pred) * X_train)
db = -(2/N) * np.sum((y_train - y_pred) * 1)_____no_output_____

**NOTE**: The gradient is the derivative, or the rate of change of a function. It's a vector that points in the direction of greatest increase of a function. For example the gradient of our loss function ($J$) with respect to our weights ($W$) will tell us how to change W so we can maximize $J$. However, we want to minimize our loss so we subtract the gradient from $W$. _____no_output_____## Update weights_____no_output_____5. Update the weights $W$ using a small learning rate $\alpha$.
  * $W = W - \alpha\frac{\partial{J}}{\partial{W}}$
  * $b = b - \alpha\frac{\partial{J}}{\partial{b}}$_____no_output_____

LEARNING_RATE = 1e-1_____no_output_____# Update weights
W += -LEARNING_RATE * dW
b += -LEARNING_RATE * db_____no_output_____

**NOTE**: The learning rate $\alpha$ is a way to control how much we update the weights by. If we choose a small learning rate, it may take a long time for our model to train. However, if we choose a large learning rate, we may overshoot and our training will never converge. The specific learning rate depends on our data and the type of models we use but it's typically good to explore in the range of $[1e^{-8}, 1e^{-1}]$. We'll explore learning rate update stratagies in later lessons._____no_output_____## Training_____no_output_____6. Repeat steps 2 - 5 to minimize the loss and train the model._____no_output_____

NUM_EPOCHS = 50_____no_output_____# Initialize random weights
W = 0.01 * np.random.randn(INPUT_DIM, OUTPUT_DIM)
b = np.zeros((1, ))

# Training loop
for epoch_num in range(NUM_EPOCHS):

    # Forward pass [NX1] · [1X1] = [NX1]
    y_pred = np.dot(X_train, W) + b

    # Loss
    loss = (1/len(y_train)) * np.sum((y_train - y_pred)**2)

    # show progress
    if epoch_num%10 == 0:
        print (f"Epoch: {epoch_num}, loss: {loss:.3f}")

    # Backpropagation
    dW = -(2/N) * np.sum((y_train - y_pred) * X_train)
    db = -(2/N) * np.sum((y_train - y_pred) * 1)

    # Update weights
    W += -LEARNING_RATE * dW
    b += -LEARNING_RATE * dbEpoch: 0, loss: 0.990
Epoch: 10, loss: 0.039
Epoch: 20, loss: 0.028
Epoch: 30, loss: 0.028
Epoch: 40, loss: 0.028

## Evaluation_____no_output_____

# Predictions
pred_train = W*X_train + b
pred_test = W*X_test + b_____no_output_____# Train and test MSE
train_mse = np.mean((y_train - pred_train) ** 2)
test_mse = np.mean((y_test - pred_test) ** 2)
print (f"train_MSE: {train_mse:.2f}, test_MSE: {test_mse:.2f}")train_MSE: 0.03, test_MSE: 0.01
# Figure size
plt.figure(figsize=(15,5))

# Plot train data
plt.subplot(1, 2, 1)
plt.title("Train")
plt.scatter(X_train, y_train, label='y_train')
plt.plot(X_train, pred_train, color='red', linewidth=1, linestyle='-', label='model')
plt.legend(loc='lower right')

# Plot test data
plt.subplot(1, 2, 2)
plt.title("Test")
plt.scatter(X_test, y_test, label='y_test')
plt.plot(X_test, pred_test, color='red', linewidth=1, linestyle='-', label='model')
plt.legend(loc='lower right')

# Show plots
plt.show()_____no_output_____

## Interpretability_____no_output_____Since we standardized our inputs and outputs, our weights were fit to those standardized values. So we need to unstandardize our weights so we can compare it to our true weight (3.5).

Note that both X and y were standardized.

$\hat{y}_{scaled} = b_{scaled} + \sum_{j=1}^{k}W_{{scaled}_j}x_{{scaled}_j}$
* $y_{scaled} = \frac{\hat{y} - \bar{y}}{\sigma_y}$
* $x_{scaled} = \frac{x_j - \bar{x}_j}{\sigma_j}$

$\frac{\hat{y} - \bar{y}}{\sigma_y} = b_{scaled} + \sum_{j=1}^{k}W_{{scaled}_j}\frac{x_j - \bar{x}_j}{\sigma_j}$

$ \hat{y}_{scaled} = \frac{\hat{y}_{unscaled} - \bar{y}}{\sigma_y} = {b_{scaled}} + \sum_{j=1}^{k} {W}_{{scaled}_j} (\frac{x_j - \bar{x}_j}{\sigma_j}) $

$\hat{y}_{unscaled} = b_{scaled}\sigma_y + \bar{y} - \sum_{j=1}^{k} {W}_{{scaled}_j}(\frac{\sigma_y}{\sigma_j})\bar{x}_j + \sum_{j=1}^{k}{W}_{{scaled}_j}(\frac{\sigma_y}{\sigma_j})x_j $

In the expression above, we can see the expression $\hat{y}_{unscaled} = W_{unscaled}x + b_{unscaled} $ where

* $W_{unscaled} = \sum_{j=1}^{k}{W}_j(\frac{\sigma_y}{\sigma_j}) $

* $b_{unscaled} = b_{scaled}\sigma_y + \bar{y} - \sum_{j=1}^{k} {W}_j(\frac{\sigma_y}{\sigma_j})\bar{x}_j$_____no_output_____

# Unscaled weights
W_unscaled = W * (y_std/X_std)
b_unscaled = b * y_std + y_mean - np.sum(W_unscaled*X_mean)
print ("[actual] y = 3.5X + noise")
print (f"[model] y_hat = {W_unscaled[0][0]:.1f}X + {b_unscaled[0]:.1f}") [actual] y = 3.5X + noise
[model] y_hat = 3.4X + 7.8

# TensorFlow_____no_output_____We will first implement linear regression using native TensorFlow without the ease of Keras. This will help us understand the hidden operations and also appreciate all the work that Keras does for us._____no_output_____

%tensorflow_version 2.x
import tensorflow as tf_____no_output_____# Set seed for reproducibility
tf.random.set_seed(SEED)_____no_output_____

## Split data_____no_output_____When we're working with TensorFlow we normally use the scikit learn's [splitting functions](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection) to split our data._____no_output_____

from sklearn.model_selection import train_test_split_____no_output_____TRAIN_SIZE = 0.7
VAL_SIZE = 0.15
TEST_SIZE = 0.15
SHUFFLE = True_____no_output_____def train_val_test_split(X, y, val_size, test_size, shuffle):
    """Split data into train/val/test datasets.
    """
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=test_size, shuffle=shuffle)
    X_train, X_val, y_train, y_val = train_test_split(
        X_train, y_train, test_size=val_size, shuffle=shuffle)
    return X_train, X_val, X_test, y_train, y_val, y_test_____no_output_____

The `train_val_test_split` function essentially splits our data twice. First, we separate out the test set. And then we separate the remaining other set into train and validation sets. _____no_output_____

# Create data splits
X_train, X_val, X_test, y_train, y_val, y_test = train_val_test_split(
    X, y, val_size=VAL_SIZE, test_size=TEST_SIZE, shuffle=SHUFFLE)
print (f"X_train: {X_train.shape}, y_train: {y_train.shape}")
print (f"X_val: {X_val.shape}, y_test: {y_val.shape}")
print (f"X_test: {X_test.shape}, y_test: {y_test.shape}")X_train: (35, 1), y_train: (35, 1)
X_val: (7, 1), y_test: (7, 1)
X_test: (8, 1), y_test: (8, 1)

## Standardize data_____no_output_____We can also use scikit learn to do [preprocessing and normalization](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.preprocessing). _____no_output_____

from sklearn.preprocessing import StandardScaler_____no_output_____# Standardize the data (mean=0, std=1) using training data
X_scaler = StandardScaler().fit(X_train)
y_scaler = StandardScaler().fit(y_train)_____no_output_____# Apply scaler on training and test data
X_train = X_scaler.transform(X_train)
y_train = y_scaler.transform(y_train).ravel().reshape(-1, 1)
X_val = X_scaler.transform(X_val)
y_val = y_scaler.transform(y_val).ravel().reshape(-1, 1)
X_test = X_scaler.transform(X_test)
y_test = y_scaler.transform(y_test).ravel().reshape(-1, 1)_____no_output_____# Check (means should be ~0 and std should be ~1)
print (f"X_train: mean: {np.mean(X_train, axis=0)[0]:.1f}, std: {np.std(X_train, axis=0)[0]:.1f}")
print (f"y_train: mean: {np.mean(y_train, axis=0)[0]:.1f}, std: {np.std(y_train, axis=0)[0]:.1f}")
print (f"X_val: mean: {np.mean(X_val, axis=0)[0]:.1f}, std: {np.std(X_val, axis=0)[0]:.1f}")
print (f"y_val: mean: {np.mean(y_val, axis=0)[0]:.1f}, std: {np.std(y_val, axis=0)[0]:.1f}")
print (f"X_test: mean: {np.mean(X_test, axis=0)[0]:.1f}, std: {np.std(X_test, axis=0)[0]:.1f}")
print (f"y_test: mean: {np.mean(y_test, axis=0)[0]:.1f}, std: {np.std(y_test, axis=0)[0]:.1f}")X_train: mean: -0.0, std: 1.0
y_train: mean: 0.0, std: 1.0
X_val: mean: 0.1, std: 0.6
y_val: mean: 0.1, std: 0.7
X_test: mean: -0.3, std: 0.7
y_test: mean: -0.3, std: 0.6

## Weights_____no_output_____

# Weights
W = tf.Variable(0.)
b = tf.Variable(0.)_____no_output_____

## Model_____no_output_____

# Model
def model(x):
    return x*W + b_____no_output_____# Forward pass
print (X_train[0])
print (model(X_train[0]))[0.96849907]
tf.Tensor([0.], shape=(1,), dtype=float32)

## Loss_____no_output_____

# Loss
def mean_squared_error(y_pred, y_true):
    return tf.reduce_mean(tf.square(y_pred-y_true))_____no_output_____# Sample loss
print (f"loss: {mean_squared_error(model(X_train), y_train)}")loss: 1.0

## Training_____no_output_____

for epoch in range(NUM_EPOCHS):
    with tf.GradientTape() as tape:
        # Forward pass
        y_pred = model(X_train)

        # Loss
        train_loss = mean_squared_error(y_pred=y_pred, y_true=y_train)

    # Gradients
    gradients = tape.gradient(train_loss, [W, b])

    # Update weights
    W.assign_sub(gradients[0] * LEARNING_RATE)
    b.assign_sub(gradients[1] * LEARNING_RATE)

    # Metrics
    if (epoch%5) == 0:
        print (f"Epoch: {epoch} | train_loss: {train_loss.numpy():.2f}")Epoch: 0 | train_loss: 1.00
Epoch: 5 | train_loss: 0.13
Epoch: 10 | train_loss: 0.03
Epoch: 15 | train_loss: 0.02
Epoch: 20 | train_loss: 0.02
Epoch: 25 | train_loss: 0.02
Epoch: 30 | train_loss: 0.02
Epoch: 35 | train_loss: 0.02
Epoch: 40 | train_loss: 0.02
Epoch: 45 | train_loss: 0.02

## Evaluation_____no_output_____

# Predictions
pred_train = W*X_train + b
pred_test = W*X_test + b_____no_output_____# Train and test MSE
train_mse = np.mean((y_train - pred_train) ** 2)
test_mse = np.mean((y_test - pred_test) ** 2)
print (f"train_MSE: {train_mse:.2f}, test_MSE: {test_mse:.2f}")train_MSE: 0.02, test_MSE: 0.01
# Figure size
plt.figure(figsize=(15,5))

# Plot train data
plt.subplot(1, 2, 1)
plt.title("Train")
plt.scatter(X_train, y_train, label='y_train')
plt.plot(X_train, model(X_train), color='red', linewidth=1, linestyle='-', label='model')
plt.legend(loc='lower right')

# Plot test data
plt.subplot(1, 2, 2)
plt.title("Test")
plt.scatter(X_test, y_test, label='y_test')
plt.plot(X_test, model(X_test), color='red', linewidth=1, linestyle='-', label='model')
plt.legend(loc='lower right')

# Show plots
plt.show()_____no_output_____

## Interpretability_____no_output_____

# Unscaled weights
W_unscaled = W.numpy() * (y_std/X_std)
b_unscaled = b.numpy() * y_std + y_mean - np.sum(W_unscaled*X_mean)
print ("[actual] y = 3.5X + noise")
print (f"[model] y_hat = {W_unscaled:.1f}X + {b_unscaled:.1f}") [actual] y = 3.5X + noise
[model] y_hat = 3.4X + 7.6

# TensorFlow + Keras_____no_output_____Now that we've implemented linear regression with Numpy and native TensorFlow, let's do the same with TensorFlow + Keras._____no_output_____## Weights_____no_output_____We will be using [Dense layers](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/layers/Dense) in our MLP implementation. The layer applies an activation function on the dot product of the layer's inputs and its weights.

$ z = \text{activation}(XW)$ _____no_output_____

from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import Input_____no_output_____x = Input(shape=(INPUT_DIM,))
fc = Dense(units=OUTPUT_DIM, activation='linear')
z = fc(x)
W, b = fc.weights
print (f"z {z.shape} = x {x.shape} · W {W.shape} + b {b.shape}")z (None, 1) = x (None, 1) · W (1, 1) + b (1,)

## Model_____no_output_____Our goal is to learn a linear model $\hat{y}$ that models $y$ given $X$.

$\hat{y} = XW + b$
* $\hat{y}$ = predictions | $\in \mathbb{R}^{NX1}$ ($N$ is the number of samples)
* $X$ = inputs | $\in \mathbb{R}^{NXD}$ ($D$ is the number of features)
* $W$ = weights | $\in \mathbb{R}^{DX1}$
* $b$ = bias | $\in \mathbb{R}^{1}$ _____no_output_____

from tensorflow.keras.models import Model
from tensorflow.keras.utils import plot_model_____no_output_____class LinearRegression(Model):
    def __init__(self, output_dim):
        super(LinearRegression, self).__init__(name='linear_regression')
        self.fc1 = Dense(units=output_dim, activation='linear', name='W')

    def call(self, x_in, training=False):
        y_pred = self.fc1(x_in)
        return y_pred

    def summary(self, input_shape):
        x_in = Input(shape=input_shape, name='X')
        summary = Model(inputs=x_in, outputs=self.call(x_in), name=self.name)
        return plot_model(summary, show_shapes=True) # forward pass_____no_output_____# Initialize model
model = LinearRegression(output_dim=OUTPUT_DIM)
model.summary(input_shape=(INPUT_DIM,))_____no_output_____

## Loss_____no_output_____

from tensorflow.keras.losses import MeanSquaredError_____no_output_____mse = MeanSquaredError()
loss = mse([0., 0., 1., 1.], [1., 1., 1., 0.])
print('Loss: ', loss.numpy()) Loss:  0.75

## Metrics_____no_output_____

from tensorflow.keras.metrics import MeanAbsolutePercentageError_____no_output_____metric = MeanAbsolutePercentageError()
metric.update_state([0.5, 0.5, 1., 1.], [0.5, 1., 1., 0.])
print('Final result: ', metric.result().numpy())Final result:  50.0

## Optimizer_____no_output_____When we implemented linear regression with just NumPy, we used batch gradient descent to update our weights. But there are actually many different [gradient descent optimization algorithms](https://ruder.io/optimizing-gradient-descent/) to choose from and it depends on the situation. However, the [ADAM optimizer](https://ruder.io/optimizing-gradient-descent/index.html#gradientdescentoptimizationalgorithms/#adam) has become a standard algorithm for most cases. _____no_output_____

from tensorflow.keras.optimizers import Adam_____no_output_____# Optimizer
optimizer = Adam(lr=LEARNING_RATE)_____no_output_____

## Training_____no_output_____Here are the full list of options for [optimizer](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/optimizers), [loss](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/losses) and [metrics](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/metrics)._____no_output_____

# Compile
model.compile(optimizer=Adam(lr=LEARNING_RATE),
              loss=MeanSquaredError(),
              metrics=[MeanAbsolutePercentageError()])_____no_output_____

When we implemented linear regression from scratch, we used batch gradient descent to update our weights. This means that we calculated the gradients using the entire training dataset. We also could've updated our weights using stochastic gradient descent (SGD) where we pass in one training example one at a time. The current standard is mini-batch gradient descent, which strikes a balance between batch and stochastic GD, where we update the weights using a mini-batch of n (`BATCH_SIZE`) samples._____no_output_____

BATCH_SIZE = 10_____no_output_____# Training
history = model.fit(x=X_train,
                    y=y_train,
                    validation_data=(X_val, y_val),
                    epochs=NUM_EPOCHS,
                    batch_size=BATCH_SIZE,
                    shuffle=False,
                    verbose=1)Train on 35 samples, validate on 7 samples
Epoch 1/50
35/35 [==============================] - 0s 8ms/sample - loss: 2.2032 - mean_absolute_percentage_error: 144.8060 - val_loss: 0.7335 - val_mean_absolute_percentage_error: 227.3441
Epoch 2/50
35/35 [==============================] - 0s 450us/sample - loss: 1.1970 - mean_absolute_percentage_error: 108.1093 - val_loss: 0.3955 - val_mean_absolute_percentage_error: 328.6021
Epoch 3/50
35/35 [==============================] - 0s 401us/sample - loss: 0.5289 - mean_absolute_percentage_error: 74.9418 - val_loss: 0.1668 - val_mean_absolute_percentage_error: 206.9364
Epoch 4/50
35/35 [==============================] - 0s 403us/sample - loss: 0.1644 - mean_absolute_percentage_error: 43.4698 - val_loss: 0.0530 - val_mean_absolute_percentage_error: 718.5472
Epoch 5/50
35/35 [==============================] - 0s 434us/sample - loss: 0.0381 - mean_absolute_percentage_error: 26.2976 - val_loss: 0.0190 - val_mean_absolute_percentage_error: 983.8639
Epoch 6/50
35/35 [==============================] - 0s 377us/sample - loss: 0.0444 - mean_absolute_percentage_error: 29.6342 - val_loss: 0.0161 - val_mean_absolute_percentage_error: 962.9730
Epoch 7/50
35/35 [==============================] - 0s 396us/sample - loss: 0.0828 - mean_absolute_percentage_error: 36.9853 - val_loss: 0.0176 - val_mean_absolute_percentage_error: 846.5008
Epoch 8/50
35/35 [==============================] - 0s 393us/sample - loss: 0.0976 - mean_absolute_percentage_error: 40.1439 - val_loss: 0.0161 - val_mean_absolute_percentage_error: 814.3967
Epoch 9/50
35/35 [==============================] - 0s 398us/sample - loss: 0.0808 - mean_absolute_percentage_error: 37.4335 - val_loss: 0.0139 - val_mean_absolute_percentage_error: 860.1505
Epoch 10/50
35/35 [==============================] - 0s 536us/sample - loss: 0.0519 - mean_absolute_percentage_error: 31.4122 - val_loss: 0.0150 - val_mean_absolute_percentage_error: 873.4119
Epoch 11/50
35/35 [==============================] - 0s 394us/sample - loss: 0.0309 - mean_absolute_percentage_error: 27.1245 - val_loss: 0.0203 - val_mean_absolute_percentage_error: 802.0851
Epoch 12/50
35/35 [==============================] - 0s 389us/sample - loss: 0.0239 - mean_absolute_percentage_error: 25.0748 - val_loss: 0.0277 - val_mean_absolute_percentage_error: 699.3417
Epoch 13/50
35/35 [==============================] - 0s 418us/sample - loss: 0.0261 - mean_absolute_percentage_error: 25.5582 - val_loss: 0.0335 - val_mean_absolute_percentage_error: 646.6029
Epoch 14/50
35/35 [==============================] - 0s 416us/sample - loss: 0.0295 - mean_absolute_percentage_error: 26.0150 - val_loss: 0.0349 - val_mean_absolute_percentage_error: 665.0445
Epoch 15/50
35/35 [==============================] - 0s 416us/sample - loss: 0.0298 - mean_absolute_percentage_error: 25.7870 - val_loss: 0.0326 - val_mean_absolute_percentage_error: 707.7242
Epoch 16/50
35/35 [==============================] - 0s 398us/sample - loss: 0.0277 - mean_absolute_percentage_error: 25.4551 - val_loss: 0.0284 - val_mean_absolute_percentage_error: 727.8849
Epoch 17/50
35/35 [==============================] - 0s 409us/sample - loss: 0.0255 - mean_absolute_percentage_error: 25.5023 - val_loss: 0.0244 - val_mean_absolute_percentage_error: 725.5954
Epoch 18/50
35/35 [==============================] - 0s 372us/sample - loss: 0.0244 - mean_absolute_percentage_error: 25.5995 - val_loss: 0.0218 - val_mean_absolute_percentage_error: 727.2172
Epoch 19/50
35/35 [==============================] - 0s 399us/sample - loss: 0.0243 - mean_absolute_percentage_error: 25.7011 - val_loss: 0.0207 - val_mean_absolute_percentage_error: 742.0944
Epoch 20/50
35/35 [==============================] - 0s 395us/sample - loss: 0.0245 - mean_absolute_percentage_error: 25.6953 - val_loss: 0.0208 - val_mean_absolute_percentage_error: 754.7386
Epoch 21/50
35/35 [==============================] - 0s 387us/sample - loss: 0.0244 - mean_absolute_percentage_error: 25.5661 - val_loss: 0.0215 - val_mean_absolute_percentage_error: 751.1937
Epoch 22/50
35/35 [==============================] - 0s 432us/sample - loss: 0.0241 - mean_absolute_percentage_error: 25.4915 - val_loss: 0.0225 - val_mean_absolute_percentage_error: 737.0789
Epoch 23/50
35/35 [==============================] - 0s 418us/sample - loss: 0.0240 - mean_absolute_percentage_error: 25.4577 - val_loss: 0.0235 - val_mean_absolute_percentage_error: 727.1889
Epoch 24/50
35/35 [==============================] - 0s 481us/sample - loss: 0.0241 - mean_absolute_percentage_error: 25.4853 - val_loss: 0.0242 - val_mean_absolute_percentage_error: 727.0169
Epoch 25/50
35/35 [==============================] - 0s 436us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4768 - val_loss: 0.0244 - val_mean_absolute_percentage_error: 730.5203
Epoch 26/50
35/35 [==============================] - 0s 422us/sample - loss: 0.0243 - mean_absolute_percentage_error: 25.4726 - val_loss: 0.0242 - val_mean_absolute_percentage_error: 731.6016
Epoch 27/50
35/35 [==============================] - 0s 433us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4893 - val_loss: 0.0237 - val_mean_absolute_percentage_error: 731.0724
Epoch 28/50
35/35 [==============================] - 0s 380us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.5046 - val_loss: 0.0233 - val_mean_absolute_percentage_error: 732.3527
Epoch 29/50
35/35 [==============================] - 0s 415us/sample - loss: 0.0241 - mean_absolute_percentage_error: 25.4985 - val_loss: 0.0231 - val_mean_absolute_percentage_error: 735.3268
Epoch 30/50
35/35 [==============================] - 0s 365us/sample - loss: 0.0241 - mean_absolute_percentage_error: 25.4811 - val_loss: 0.0230 - val_mean_absolute_percentage_error: 736.9933
Epoch 31/50
35/35 [==============================] - 0s 423us/sample - loss: 0.0241 - mean_absolute_percentage_error: 25.4728 - val_loss: 0.0231 - val_mean_absolute_percentage_error: 736.0269
Epoch 32/50
35/35 [==============================] - 0s 410us/sample - loss: 0.0241 - mean_absolute_percentage_error: 25.4779 - val_loss: 0.0233 - val_mean_absolute_percentage_error: 734.1309
Epoch 33/50
35/35 [==============================] - 0s 370us/sample - loss: 0.0241 - mean_absolute_percentage_error: 25.4853 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 733.2979
Epoch 34/50
35/35 [==============================] - 0s 419us/sample - loss: 0.0241 - mean_absolute_percentage_error: 25.4873 - val_loss: 0.0235 - val_mean_absolute_percentage_error: 733.5831
Epoch 35/50
35/35 [==============================] - 0s 420us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4869 - val_loss: 0.0235 - val_mean_absolute_percentage_error: 733.9268
Epoch 36/50
35/35 [==============================] - 0s 387us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4891 - val_loss: 0.0235 - val_mean_absolute_percentage_error: 733.9293
Epoch 37/50
35/35 [==============================] - 0s 595us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4926 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.0235
Epoch 38/50
35/35 [==============================] - 0s 408us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4937 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.4495
Epoch 39/50
35/35 [==============================] - 0s 402us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4919 - val_loss: 0.0233 - val_mean_absolute_percentage_error: 734.8490
Epoch 40/50
35/35 [==============================] - 0s 367us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4904 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.8605
Epoch 41/50
35/35 [==============================] - 0s 393us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4911 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.6075
Epoch 42/50
35/35 [==============================] - 0s 374us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4928 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.4434
Epoch 43/50
35/35 [==============================] - 0s 541us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4940 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.4872
Epoch 44/50
35/35 [==============================] - 0s 414us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4945 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.5970
Epoch 45/50
35/35 [==============================] - 0s 394us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4951 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.6555
Epoch 46/50
35/35 [==============================] - 0s 422us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4960 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.6952
Epoch 47/50
35/35 [==============================] - 0s 401us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4966 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.7706
Epoch 48/50
35/35 [==============================] - 0s 400us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4968 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.8504
Epoch 49/50
35/35 [==============================] - 0s 383us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4969 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.8776
Epoch 50/50
35/35 [==============================] - 0s 362us/sample - loss: 0.0242 - mean_absolute_percentage_error: 25.4974 - val_loss: 0.0234 - val_mean_absolute_percentage_error: 734.8608
# Training metrics
print (f"metrics: {list(history.history.keys())}")
print (f"final val_loss: {history.history['val_loss'][-1]:.2f}")metrics: ['loss', 'mean_absolute_percentage_error', 'val_loss', 'val_mean_absolute_percentage_error']
final val_loss: 0.02

## Evaluation_____no_output_____

# Predictions
pred_train = model.predict(X_train)
pred_test = model.predict(X_test)_____no_output_____# Train metrics
train_mse = tf.keras.metrics.MeanSquaredError()
train_mse.update_state(y_train, pred_train)
print(f'train_mse: {train_mse.result().numpy(): .2f}')train_mse:  0.02
# Test metrics
test_mse = tf.keras.metrics.MeanSquaredError()
test_mse.update_state(y_test, pred_test)
print(f'test_mse: {test_mse.result().numpy(): .2f}')test_mse:  0.01

Since we only have one feature, it's easy to visually inspect the model._____no_output_____

# Figure size
plt.figure(figsize=(15,5))

# Plot train data
plt.subplot(1, 2, 1)
plt.title("Train")
plt.scatter(X_train, y_train, label='y_train')
plt.plot(X_train, pred_train, color='red', linewidth=1, linestyle='-', label='model')
plt.legend(loc='lower right')

# Plot test data
plt.subplot(1, 2, 2)
plt.title("Test")
plt.scatter(X_test, y_test, label='y_test')
plt.plot(X_test, pred_test, color='red', linewidth=1, linestyle='-', label='model')
plt.legend(loc='lower right')

# Show plots
plt.show()_____no_output_____

## Inference_____no_output_____After training a model, we can use it to predict on new data._____no_output_____

# Feed in your own inputs
sample_indices = [10, 15, 25]
X_infer = np.array(sample_indices, dtype=np.float32)
X_infer = X_scaler.transform(X_infer.reshape(-1, 1))_____no_output_____

Recall that we need to unstandardize our predictions.

$ \hat{y}_{scaled} = \frac{\hat{y} - \mu_{\hat{y}}}{\sigma_{\hat{y}}} $

$ \hat{y} = \hat{y}_{scaled} * \sigma_{\hat{y}} + \mu_{\hat{y}} $_____no_output_____

# Unstandardize predictions
pred_infer = model.predict(X_infer) * np.sqrt(y_scaler.var_) + y_scaler.mean_
for i, index in enumerate(sample_indices):
    print (f"{df.iloc[index]['y']:.2f} (actual) → {pred_infer[i][0]:.2f} (predicted)")35.73 (actual) → 43.36 (predicted)
59.34 (actual) → 60.37 (predicted)
97.04 (actual) → 94.40 (predicted)

## Interpretability_____no_output_____Linear regression offers the great advantage of being highly interpretable. Each feature has a coefficient which signifies its importance/impact on the output variable y. We can interpret our coefficient as follows: by increasing X by 1 unit, we increase y by $W$ (~3.65) units. _____no_output_____

# Unstandardize coefficients
W = model.layers[0].get_weights()[0][0][0]
b = model.layers[0].get_weights()[1][0]
W_unscaled = W * (y_scaler.scale_/X_scaler.scale_)
b_unscaled = b * y_scaler.scale_ + y_scaler.mean_ - np.sum(W_unscaled*X_scaler.mean_)
print ("[actual] y = 3.5X + noise")
print (f"[model] y_hat = {W_unscaled[0]:.1f}X + {b_unscaled[0]:.1f}") [actual] y = 3.5X + noise
[model] y_hat = 3.4X + 9.3

# Regularization_____no_output_____Regularization helps decrease overfitting. Below is L2 regularization (ridge regression). There are many forms of regularization but they all work to reduce overfitting in our models. With L2 regularization, we are penalizing the weights with large magnitudes by decaying them. Having certain weights with high magnitudes will lead to preferential bias with the inputs and we want the model to work with all the inputs and not just a select few. There are also other types of regularization like L1 (lasso regression) which is useful for creating sparse models where some feature cofficients are zeroed out, or elastic which combines L1 and L2 penalties.

**Note**: Regularization is not just for linear regression. You can use it to regularize any model's weights including the ones we will look at in future lessons._____no_output_____$ J(\theta) = = \frac{1}{2}\sum_{i}(X_iW - y_i)^2 + \frac{\lambda}{2}W^TW$

$ \frac{\partial{J}}{\partial{W}}  = X (\hat{y} - y) + \lambda W $

$W = W- \alpha\frac{\partial{J}}{\partial{W}}$
* $\lambda$ is the regularzation coefficient_____no_output_____

from tensorflow.keras.regularizers import l2_____no_output_____L2_LAMBDA = 1e-2_____no_output_____class L2LinearRegression(Model):
    def __init__(self, l2_lambda, output_dim):
        super(L2LinearRegression, self).__init__(name='l2_linear_regression')
        self.fc1 = Dense(units=output_dim, activation='linear',
                         kernel_regularizer=l2(l=l2_lambda), name='W')

    def call(self, x_in, training=False):
        y_pred = self.fc1(x_in)
        return y_pred

    def summary(self, input_shape):
        x_in = Input(shape=input_shape, name='X')
        summary = Model(inputs=x_in, outputs=self.call(x_in), name=self.name)
        return plot_model(summary, show_shapes=True) # forward pass_____no_output_____# Initialize model
model = L2LinearRegression(l2_lambda=L2_LAMBDA, output_dim=OUTPUT_DIM)
model.summary(input_shape=(INPUT_DIM,))_____no_output_____# Compile
model.compile(optimizer=Adam(lr=LEARNING_RATE),
              loss=MeanSquaredError(),
              metrics=[MeanAbsolutePercentageError()])_____no_output_____# Training
model.fit(x=X_train,
          y=y_train,
          validation_data=(X_val, y_val),
          epochs=NUM_EPOCHS,
          batch_size=BATCH_SIZE,
          shuffle=SHUFFLE,
          verbose=1)Train on 35 samples, validate on 7 samples
Epoch 1/50
35/35 [==============================] - 0s 9ms/sample - loss: 0.3463 - mean_absolute_percentage_error: 64.7264 - val_loss: 0.0935 - val_mean_absolute_percentage_error: 132.8669
Epoch 2/50
35/35 [==============================] - 0s 430us/sample - loss: 0.0612 - mean_absolute_percentage_error: 28.1441 - val_loss: 0.0314 - val_mean_absolute_percentage_error: 882.2163
Epoch 3/50
35/35 [==============================] - 0s 451us/sample - loss: 0.0474 - mean_absolute_percentage_error: 27.2072 - val_loss: 0.0277 - val_mean_absolute_percentage_error: 866.8756
Epoch 4/50
35/35 [==============================] - 0s 358us/sample - loss: 0.0819 - mean_absolute_percentage_error: 36.9764 - val_loss: 0.0243 - val_mean_absolute_percentage_error: 359.9240
Epoch 5/50
35/35 [==============================] - 0s 394us/sample - loss: 0.0596 - mean_absolute_percentage_error: 34.5615 - val_loss: 0.0262 - val_mean_absolute_percentage_error: 682.9621
Epoch 6/50
35/35 [==============================] - 0s 401us/sample - loss: 0.0370 - mean_absolute_percentage_error: 25.2826 - val_loss: 0.0396 - val_mean_absolute_percentage_error: 922.7436
Epoch 7/50
35/35 [==============================] - 0s 406us/sample - loss: 0.0383 - mean_absolute_percentage_error: 24.8166 - val_loss: 0.0468 - val_mean_absolute_percentage_error: 588.8998
Epoch 8/50
35/35 [==============================] - 0s 412us/sample - loss: 0.0424 - mean_absolute_percentage_error: 26.9451 - val_loss: 0.0457 - val_mean_absolute_percentage_error: 533.3515
Epoch 9/50
35/35 [==============================] - 0s 397us/sample - loss: 0.0383 - mean_absolute_percentage_error: 26.8012 - val_loss: 0.0374 - val_mean_absolute_percentage_error: 558.5472
Epoch 10/50
35/35 [==============================] - 0s 589us/sample - loss: 0.0341 - mean_absolute_percentage_error: 25.9799 - val_loss: 0.0312 - val_mean_absolute_percentage_error: 716.1393
Epoch 11/50
35/35 [==============================] - 0s 445us/sample - loss: 0.0336 - mean_absolute_percentage_error: 25.6114 - val_loss: 0.0284 - val_mean_absolute_percentage_error: 545.0917
Epoch 12/50
35/35 [==============================] - 0s 430us/sample - loss: 0.0371 - mean_absolute_percentage_error: 28.5534 - val_loss: 0.0287 - val_mean_absolute_percentage_error: 401.7839
Epoch 13/50
35/35 [==============================] - 0s 515us/sample - loss: 0.0340 - mean_absolute_percentage_error: 26.2566 - val_loss: 0.0334 - val_mean_absolute_percentage_error: 804.1356
Epoch 14/50
35/35 [==============================] - 0s 441us/sample - loss: 0.0346 - mean_absolute_percentage_error: 25.6177 - val_loss: 0.0363 - val_mean_absolute_percentage_error: 803.2571
Epoch 15/50
35/35 [==============================] - 0s 433us/sample - loss: 0.0347 - mean_absolute_percentage_error: 25.0646 - val_loss: 0.0356 - val_mean_absolute_percentage_error: 731.2593
Epoch 16/50
35/35 [==============================] - 0s 417us/sample - loss: 0.0340 - mean_absolute_percentage_error: 25.0223 - val_loss: 0.0338 - val_mean_absolute_percentage_error: 612.5067
Epoch 17/50
35/35 [==============================] - 0s 489us/sample - loss: 0.0331 - mean_absolute_percentage_error: 26.7277 - val_loss: 0.0328 - val_mean_absolute_percentage_error: 451.5154
Epoch 18/50
35/35 [==============================] - 0s 409us/sample - loss: 0.0338 - mean_absolute_percentage_error: 27.5179 - val_loss: 0.0328 - val_mean_absolute_percentage_error: 466.8643
Epoch 19/50
35/35 [==============================] - 0s 450us/sample - loss: 0.0333 - mean_absolute_percentage_error: 27.1883 - val_loss: 0.0327 - val_mean_absolute_percentage_error: 533.8986
Epoch 20/50
35/35 [==============================] - 0s 394us/sample - loss: 0.0331 - mean_absolute_percentage_error: 26.6978 - val_loss: 0.0326 - val_mean_absolute_percentage_error: 672.0956
Epoch 21/50
35/35 [==============================] - 0s 469us/sample - loss: 0.0337 - mean_absolute_percentage_error: 25.4493 - val_loss: 0.0343 - val_mean_absolute_percentage_error: 837.8459
Epoch 22/50
35/35 [==============================] - 0s 458us/sample - loss: 0.0347 - mean_absolute_percentage_error: 25.9778 - val_loss: 0.0322 - val_mean_absolute_percentage_error: 570.5813
Epoch 23/50
35/35 [==============================] - 0s 530us/sample - loss: 0.0334 - mean_absolute_percentage_error: 27.0696 - val_loss: 0.0329 - val_mean_absolute_percentage_error: 472.2430
Epoch 24/50
35/35 [==============================] - 0s 405us/sample - loss: 0.0338 - mean_absolute_percentage_error: 27.3810 - val_loss: 0.0322 - val_mean_absolute_percentage_error: 607.1464
Epoch 25/50
35/35 [==============================] - 0s 559us/sample - loss: 0.0336 - mean_absolute_percentage_error: 27.4095 - val_loss: 0.0324 - val_mean_absolute_percentage_error: 473.0745
Epoch 26/50
35/35 [==============================] - 0s 406us/sample - loss: 0.0337 - mean_absolute_percentage_error: 27.4562 - val_loss: 0.0316 - val_mean_absolute_percentage_error: 583.5190
Epoch 27/50
35/35 [==============================] - 0s 369us/sample - loss: 0.0335 - mean_absolute_percentage_error: 25.6790 - val_loss: 0.0351 - val_mean_absolute_percentage_error: 847.4083
Epoch 28/50
35/35 [==============================] - 0s 405us/sample - loss: 0.0346 - mean_absolute_percentage_error: 25.2173 - val_loss: 0.0348 - val_mean_absolute_percentage_error: 662.8919
Epoch 29/50
35/35 [==============================] - 0s 415us/sample - loss: 0.0332 - mean_absolute_percentage_error: 25.9801 - val_loss: 0.0332 - val_mean_absolute_percentage_error: 469.8535
Epoch 30/50
35/35 [==============================] - 0s 410us/sample - loss: 0.0339 - mean_absolute_percentage_error: 27.8425 - val_loss: 0.0326 - val_mean_absolute_percentage_error: 414.7518
Epoch 31/50
35/35 [==============================] - 0s 550us/sample - loss: 0.0332 - mean_absolute_percentage_error: 26.9420 - val_loss: 0.0340 - val_mean_absolute_percentage_error: 710.4662
Epoch 32/50
35/35 [==============================] - 0s 380us/sample - loss: 0.0339 - mean_absolute_percentage_error: 25.4025 - val_loss: 0.0347 - val_mean_absolute_percentage_error: 688.7251
Epoch 33/50
35/35 [==============================] - 0s 417us/sample - loss: 0.0342 - mean_absolute_percentage_error: 26.9076 - val_loss: 0.0317 - val_mean_absolute_percentage_error: 487.5617
Epoch 34/50
35/35 [==============================] - 0s 466us/sample - loss: 0.0334 - mean_absolute_percentage_error: 26.8543 - val_loss: 0.0310 - val_mean_absolute_percentage_error: 613.7100
Epoch 35/50
35/35 [==============================] - 0s 459us/sample - loss: 0.0336 - mean_absolute_percentage_error: 26.1343 - val_loss: 0.0310 - val_mean_absolute_percentage_error: 599.0154
Epoch 36/50
35/35 [==============================] - 0s 518us/sample - loss: 0.0337 - mean_absolute_percentage_error: 27.5218 - val_loss: 0.0323 - val_mean_absolute_percentage_error: 403.6833
Epoch 37/50
35/35 [==============================] - 0s 421us/sample - loss: 0.0336 - mean_absolute_percentage_error: 27.4454 - val_loss: 0.0349 - val_mean_absolute_percentage_error: 461.2988
Epoch 38/50
35/35 [==============================] - 0s 413us/sample - loss: 0.0334 - mean_absolute_percentage_error: 26.8901 - val_loss: 0.0338 - val_mean_absolute_percentage_error: 650.4165
Epoch 39/50
35/35 [==============================] - 0s 463us/sample - loss: 0.0344 - mean_absolute_percentage_error: 25.0298 - val_loss: 0.0335 - val_mean_absolute_percentage_error: 733.0479
Epoch 40/50
35/35 [==============================] - 0s 441us/sample - loss: 0.0350 - mean_absolute_percentage_error: 26.8224 - val_loss: 0.0308 - val_mean_absolute_percentage_error: 331.2224
Epoch 41/50
35/35 [==============================] - 0s 373us/sample - loss: 0.0348 - mean_absolute_percentage_error: 28.9593 - val_loss: 0.0323 - val_mean_absolute_percentage_error: 562.0246
Epoch 42/50
35/35 [==============================] - 0s 411us/sample - loss: 0.0338 - mean_absolute_percentage_error: 25.8618 - val_loss: 0.0352 - val_mean_absolute_percentage_error: 705.7799
Epoch 43/50
35/35 [==============================] - 0s 430us/sample - loss: 0.0343 - mean_absolute_percentage_error: 25.7046 - val_loss: 0.0333 - val_mean_absolute_percentage_error: 538.3356
Epoch 44/50
35/35 [==============================] - 0s 385us/sample - loss: 0.0336 - mean_absolute_percentage_error: 26.7710 - val_loss: 0.0327 - val_mean_absolute_percentage_error: 509.3722
Epoch 45/50
35/35 [==============================] - 0s 523us/sample - loss: 0.0345 - mean_absolute_percentage_error: 28.7642 - val_loss: 0.0327 - val_mean_absolute_percentage_error: 384.9824
Epoch 46/50
35/35 [==============================] - 0s 453us/sample - loss: 0.0338 - mean_absolute_percentage_error: 25.9567 - val_loss: 0.0345 - val_mean_absolute_percentage_error: 814.8223
Epoch 47/50
35/35 [==============================] - 0s 422us/sample - loss: 0.0352 - mean_absolute_percentage_error: 24.8184 - val_loss: 0.0358 - val_mean_absolute_percentage_error: 634.1329
Epoch 48/50
35/35 [==============================] - 0s 457us/sample - loss: 0.0348 - mean_absolute_percentage_error: 27.6794 - val_loss: 0.0312 - val_mean_absolute_percentage_error: 303.8320
Epoch 49/50
35/35 [==============================] - 0s 476us/sample - loss: 0.0361 - mean_absolute_percentage_error: 28.6131 - val_loss: 0.0304 - val_mean_absolute_percentage_error: 588.8770
Epoch 50/50
35/35 [==============================] - 0s 521us/sample - loss: 0.0332 - mean_absolute_percentage_error: 24.9838 - val_loss: 0.0354 - val_mean_absolute_percentage_error: 851.1025
# Predictions
pred_train = model.predict(X_train)
pred_test = model.predict(X_test)_____no_output_____# Train metrics
train_mse = tf.keras.metrics.MeanSquaredError()
train_mse.update_state(y_train, pred_train)
print(f'train_mse: {train_mse.result().numpy(): .2f}')train_mse:  0.02
# Test metrics
test_mse = tf.keras.metrics.MeanSquaredError()
test_mse.update_state(y_test, pred_test)
print(f'test_mse: {test_mse.result().numpy(): .2f}')test_mse:  0.01

Regularization didn't make a difference in performance with this specific example because our data is generated from a perfect linear equation but for large realistic data, regularization can help our model generalize well._____no_output_____---
Share and discover ML projects at Made With ML.

             _____no_output_____


# Notebook from moo100/langtangen
Path: Exercises 3.ipynb

# Exercise 3.1: Implement a simple mathematical function_____no_output_____

from math import exp, sin, pi

def g(t):
    return exp(-t) * sin(pi * t)_____no_output_____print(g(0), g(1))0.0 4.505223801027239e-17

# Exercise 3.2: Implement a simple mathematical function with a parameter_____no_output_____

def h(t, a=1):
    return exp(-a * t) * sin(pi * t)_____no_output_____print(h(0, 10), h(1, 10))0.0 5.559887866514173e-21

# Exercise 3.3: Explain how a program works_____no_output_____

def add(A, B):
    C = A + B
    return C

a = 3
b = 2
print(add(a, b))          # 5
print(add(2*a, b+1)*3)    # 275
27

# Exercise 3.4: Write a Fahrenheit-Celsius conversion function_____no_output_____

def C(F):
    return 5 / 9 * (F - 32)

def F(C):
    return 9/5*C + 32

a = 5
print(a, C(F(a)), F(C(a)))5 5.0 5.0

# Exercise 3.5: Write a test function for Exercise 3.4_____no_output_____

def test_F_C():
    eps = 1e-14
    a = 12.34
    assert abs(C(F(a)) - a) < eps, f"a = {a}, C(F(a)) = {C(F(a))}"
    assert abs(F(C(a)) - a) < eps, f"a = {a}, F(C(a)) = {F(C(a))}"

test_F_C()_____no_output_____

# Exercise 3.6: Given a test function, write the function_____no_output_____

def test_double():
    assert double(2) == 4
    assert abs(double(0.1) - 0.2) < 1E-15
    assert double([1, 2]) == [1, 2, 1, 2]
    assert double((1, 2)) == (1, 2, 1, 2)
    assert double(3+4j) == 6+8j
    assert double('hello') == 'hellohello'

def double(x):
    return 2*x

test_double()_____no_output_____

# Exercise 3.7: Evaluate a sum and write a test function_____no_output_____

def sum_1k(M):
    return sum(1 / k for k in range(1, M+1))_____no_output_____def test_sum_1k():
    M = 3
    exact = 1 + 1/2 + 1/3
    result = sum_1k(M)
    error = abs(result - exact)
    success = error < 1e-15
    assert success, f"sum_1k({M}): result={result}, exact={exact}, error={error}"

test_sum_1k()_____no_output_____

# Exercise 3.8: Write a function for solving $a x^2 + bx + c = 0$_____no_output_____

from numpy.lib.scimath import sqrt

def roots(a, b, c):
    r1 = sqrt((b**2 / 4 / a**2) - c/a) - b/2/a
    r2 = -sqrt((b**2 / 4 / a**2) - c/a) - b/2/a
    return r1, r2_____no_output_____def test_roots_float():
    a, b, c = 2, 4, -2
    r = sorted(roots(a, b, c))
    exact = -2.4142, 0.41421
    error = (abs(r[0] - exact[0]) + abs(r[1] - exact[1])) / 2
    success = error < 1e-5
    assert success, f"Computed roots: {r}, actual roots: {exact}. Error: {error}"

def test_roots_complex():
    a, b, c = 2, 4, 3
    r = sorted(roots(a, b, c), key=lambda x: x.imag)
    exact =  -1.00000 - 0.70711j, -1.00000 + 0.70711j
    error = (abs(r[0] - exact[0]) + abs(r[1] - exact[1])) / 2
    success = error < 1e-5
    assert success, f"Computed roots: {r}, actual roots: {exact}. Error: {error}"_____no_output_____test_roots_float()
test_roots_complex()_____no_output_____

# Exercise 3.9: Implement the sum function_____no_output_____

def mysum(x):
    s = type(x[0])()
    for y in x:
        s += y
    return s_____no_output_____print(mysum([1, 2, 3]))
print(mysum(["Hello, ", "World!"]))6
Hello, World!

# Exercise 3.10: Compute a polynomial via a product_____no_output_____

from numpy import prod

def poly(x, roots):
    """
    Computes the value of a polynomial P(x) given the roots. #
    Does not work if there is a coefficient on the d-th term, d being the degree of the polynomial.
    """
    return prod([x - r for r in roots])

def test_poly():
    x = 5
    roots = -4.4495, 0.44949
    approx = poly(x, roots)
    exact = 43
    error = abs(approx - exact)
    success = error < 1e-4
    assert success, f"Computed value: {approx}, actual value: {exact}. Error: {error}"_____no_output_____test_poly()_____no_output_____

# Exercise 3.11: Integrate a function by the Trapezoidal rule_____no_output_____

def trapezint1(f, a, b):
    return (b - a) / 2 * (f(a) + f(b))_____no_output_____def trapezint2(f, a, b):
    return (b - a) / 4 * (f(a) + 2*f(a + (b - a)/2) + f(b))_____no_output_____def trapezint(f, a, b, n):
    h = (b - a) / n
    x = lambda i: a + i*h
    return sum(h / 2 * (f(x(i)) + f(x(i+1))) for i in range(0, n))    # 0 -> n-1, im buch steht 1 -> n-1_____no_output_____from math import cos, pi, sin

# trapezint = trapezint2

approx = trapezint(cos, 0, pi, 100)
exact = 0
error = abs(approx - exact)
print(f"cosx i[0, pi]: approx={approx}, exact={exact}, error={error}")

approx = trapezint(sin, 0, pi, 100)
exact = 2
error = abs(approx - exact)
print(f"sinx i[0, pi]: approx={approx}, exact={exact}, error={error}")

approx = trapezint(sin, 0, pi / 2, 100)
exact = 1
error = abs(approx - exact)
print(f"sinx i[0, pi/2]: approx={approx}, exact={exact}, error={error}")cosx i[0, pi]: approx=-4.0939474033052647e-16, exact=0, error=4.0939474033052647e-16
sinx i[0, pi]: approx=1.9998355038874438, exact=2, error=0.00016449611255620056
sinx i[0, pi/2]: approx=0.9999794382396072, exact=1, error=2.0561760392778794e-05
def test_trapezint():
    approx = trapezint(cos, 0, 2*pi, 10)
    exact = 0
    error = abs(approx - exact)
    success = error < 1e-10
    assert success, f"cosx i[0, 2*pi]: approx={approx}, exact={exact}, error={error}"_____no_output_____test_trapezint()_____no_output_____

# Exercise 3.12: Derive the general Midpoint integration rule_____no_output_____

def midpointint(f, a, b, n):
    h = (b - a) / n
    return h * sum(f(a + i*h + h/2) for i in range(0, n))_____no_output_____from math import cos, pi, sin

approx = midpointint(cos, 0, pi, 100)
exact = 0
error = abs(approx - exact)
print(f"cosx i[0, pi]: approx={approx}, exact={exact}, error={error}")

approx = midpointint(sin, 0, pi, 100)
exact = 2
error = abs(approx - exact)
print(f"sinx i[0, pi]: approx={approx}, exact={exact}, error={error}")

approx = midpointint(sin, 0, pi / 2, 100)
exact = 1
error = abs(approx - exact)
print(f"sinx i[0, pi/2]: approx={approx}, exact={exact}, error={error}")

# better for large n, worse for small n (in comparison to trapezoidal rule)cosx i[0, pi]: approx=-1.9880850438649202e-16, exact=0, error=1.9880850438649202e-16
sinx i[0, pi]: approx=2.0000822490709864, exact=2, error=8.224907098641765e-05
sinx i[0, pi/2]: approx=1.0000102809119051, exact=1, error=1.0280911905136136e-05

# Exercise 3.13: Make an adaptive Trapezoidal rule_____no_output_____

# I mean it works I guess, but I just guessed an `h_` and I don't think this is the correct solution.
def adaptive_trapezint(f, a, b, eps=1e-1):
    h_ = 1e-6
    d2f = lambda x: (f(x + h_) - 2*f(x) + f(x - h_)) / h_**2
    d2f_max = 0
    for x in range(int((b - a) // h_)):
        d2f_ = abs(d2f(x))
        if d2f_ > d2f_max:
            d2f_max = d2f_
    n = int((b - a) / sqrt(12 * eps) * sqrt((b - a) * d2f_max))
    return trapezint(f, a, b, n)_____no_output_____from math import cos, pi, sin

approx = adaptive_trapezint(cos, 0, pi)
exact = 0
error = abs(approx - exact)
print(f"cosx i[0, pi]: approx={approx}, exact={exact}, error={error}")

approx = adaptive_trapezint(sin, 0, pi)
exact = 2
error = abs(approx - exact)
print(f"sinx i[0, pi]: approx={approx}, exact={exact}, error={error}")

approx = adaptive_trapezint(sin, 0, pi / 2)
exact = 1
error = abs(approx - exact)
print(f"sinx i[0, pi/2]: approx={approx}, exact={exact}, error={error}")cosx i[0, pi]: approx=1.5959455978986625e-16, exact=0, error=1.5959455978986625e-16
sinx i[0, pi]: approx=1.9996442490081243, exact=2, error=0.0003557509918756807
sinx i[0, pi/2]: approx=0.9770486166568532, exact=1, error=0.022951383343146836

# Exercise 3.14: Simulate a program by hand_____no_output_____

def a(x):
    q = 2
    x = 3*x
    return q + x

def b(x):
    global q
    q += x
    return q + x

q = 0
x = 3
print(a(x), b(x), a(x), b(x))

# i did it yay11 6 11 9

# Exercise 3.15: Debug a given test function_____no_output_____

def triple(x):
    return x + x*2_____no_output_____def test_triple():
    assert triple(3) == 9
    assert abs(triple(0.1) - 0.3) < 1e-10
    assert triple([1, 2]) == [1, 2, 1, 2, 1, 2]
    assert triple("hello ") == "hello hello hello "

test_triple()_____no_output_____

# Exercise 3.16: Compute the area of an arbitrary triangle_____no_output_____

def triangle_area(v):
    return 1/2 * abs(v[1][0]*v[2][1] - v[2][0]*v[1][1] - v[0][0]*v[2][1] + v[2][0]*v[0][1] + v[0][0]*v[1][1] - v[1][0]*v[0][1])_____no_output_____def test_triangle_area():
    """
    Verify the area of a triangle with vertex coordinates
    (0,0), (1,0), and (0,2).
    """
    v1 = (0,0); v2 = (1,0); v3 = (0,2)
    vertices = [v1, v2, v3]
    expected = 1
    computed = triangle_area(vertices)
    tol = 1e-14
    success = abs(expected - computed) < tol    # what does `diff` mean?
    msg = "computed area=%g != %g (expected)" % (computed, expected)
    assert success, msg

test_triangle_area()_____no_output_____

# Exercise 3.17: Compute the length of a path_____no_output_____

from math import sqrt

def pathlength(x, y):
    return sum(sqrt((x[i] - x[i - 1])**2 + (y[i] - y[i - 1])**2) for i in range(1, len(x)))

def test_pathlength():
    x, y = (1, 2, 3, 4), (2, 3, 4, 5)
    approx = pathlength(x, y)
    exact = 3 * sqrt(2)
    error = abs(exact - approx)
    eps = 1e-12
    success = error < eps
    assert success, f"approx: {approx}, exact: {exact}, error: {error}"

test_pathlength()_____no_output_____

# Exercise 3.18: Approximate $\pi$_____no_output_____

xi = lambda n, i: 1/2 * cos(2 * pi * i / n)
yi = lambda n, i: 1/2 * sin(2 * pi * i / n)

for n in (2**k for k in range(2, 11)):
    x = []; y = []
    for i in range(n):
        x.append(xi(n, i))
        y.append(yi(n, i))
    path = pathlength(x, y)
    print(path, abs(path - pi))2.1213203435596424 1.0202723100301507
2.678784026555628 0.4628086270341649
2.926354830241923 0.21523782334787
3.0385313502163798 0.10306130337341335
3.0912634826273364 0.050329170962456704
3.116736022409867 0.024856631179926314
3.1292422628585688 0.012350390731224348
3.1354370557179547 0.006155597871838392
3.138519768514227 0.003072885075566134

# Exercise 3.19: Compute the area of a polygon_____no_output_____

def polygon_area(x, y):
    return 1 / 2 * sum(x[i - 1]*y[i] - y[i - 1]*x[i] for i in range(len(x)))_____no_output_____polygon_area((0, 1, 1, 0), (0, 0, 1, 1))_____no_output_____polygon_area((0, 1, 0), (0, 0, 1))_____no_output_____# i don't want to calculate the area of a pentagon.._____no_output_____

# Exercise 3.20: Write functions_____no_output_____

def hw1():
    return "Hello, World!"

def hw2():
    print("Hello, World!")

def hw3(x, y):
    return x + y

print(hw1())
hw2()
print(hw3("Hello, ", "World!"))
print(hw3("Python ", "function"))Hello, World!
Hello, World!
Hello, World!
Python function

# Exercise 3.21: Approximate a function by a sum of sines_____no_output_____

from math import pi, sin

def S(t, n, T):
    return 4 / pi * sum(sin(2 * (2*i - 1) * pi * t / T) / (2*i - 1) for i in range(1, n + 1))

def f(t, T):
    return 1 if 0 < t < T/2 else -1 if T/2 < t < T else 0

print("   n |  alpha |    error ")
print("-----+--------+----------")
T = 2*pi
for n in 1, 3, 5, 10, 30, 100:
    for alpha in 0.01, 0.25, 0.49:
        t = alpha * T
        error = f(t, T) - S(t, n, T)
        print(f" {n:3} | {alpha:6} | {error:8.4f}")   n |  alpha |    error
-----+--------+----------
   1 |   0.01 |   0.9201
   1 |   0.25 |  -0.2732
   1 |   0.49 |   0.9201
   3 |   0.01 |   0.7618
   3 |   0.25 |  -0.1035
   3 |   0.49 |   0.7618
   5 |   0.01 |   0.6086
   5 |   0.25 |  -0.0631
   5 |   0.49 |   0.6086
  10 |   0.01 |   0.2668
  10 |   0.25 |   0.0318
  10 |   0.49 |   0.2668
  30 |   0.01 |  -0.1448
  30 |   0.25 |   0.0106
  30 |   0.49 |  -0.1448
 100 |   0.01 |   0.0501
 100 |   0.25 |   0.0032
 100 |   0.49 |   0.0501

# Exercise 3.22: Implement a Gaussian function_____no_output_____

from math import sqrt, pi, exp

def gauss(x, m=0, s=1):
    return exp(-((x - m) / s)**2 / 2) / sqrt(2 * pi) / s

m = -3    # Mean
s = 2     # Standard deviation
n = 10    # No. of value pairs

print("      x |  gauss(x) ")
print("--------+-----------")
for x in [m - 5*s + i*(10 * s / (n - 1)) for i in range(n)]:
    print(f" {x:6.2f} | {gauss(x, m, s):6.3e} ")      x |  gauss(x)
--------+-----------
 -13.00 | 7.434e-07
 -10.78 | 1.037e-04
  -8.56 | 4.211e-03
  -6.33 | 4.974e-02
  -4.11 | 1.709e-01
  -1.89 | 1.709e-01
   0.33 | 4.974e-02
   2.56 | 4.211e-03
   4.78 | 1.037e-04
   7.00 | 7.434e-07

# Exercise 3.23: Wrap a formula in a function_____no_output_____

from math import pi, log

def egg(M, T0=20, Ty=70):
    """
    Calculate the perfect time to cook an egg.

    Parameters
    ----------
    M : number
        Mass of the egg [kg]
    T0 : number, optional, default 20
        Initial temperature of the egg [°C]
    Ty : number, optional, default 70
        Maximum temperature of the yolk [°C]

    Returns
    -------
    t : number
        Time to boil the egg [s]
    """

    Tw = 100             # Temperature of the boiling water [°C]
    rho = 1.038e3        # Density of the egg [kg / m^3]
    c = 3.7e3            # Specific heat capacity of the egg [J / kg / K]
    K = 0.54             # Thermal conductivity [W / K]

    t = M**(2 / 3) * c * rho**(1 / 3) / K / pi**2 / (4 * 3 / pi)**(2 / 3) * log(0.76 * (T0 - Tw) / (Ty - Tw))

    return t

print(f"""
Time to soft-boil a small egg taken from the fridge: {egg(47, T0=4, Ty=65):.2e} s
Time to hard-boil a small egg taken from the fridge: {egg(47, T0=4, Ty=75):.2e} s
Time to soft-boil a large egg taken from the fridge: {egg(67, T0=4, Ty=65):.2e} s
Time to hard-boil a large egg taken from the fridge: {egg(67, T0=4, Ty=75):.2e} s
Time to soft-boil a small egg taken from a hot room: {egg(47, T0=25, Ty=65):.2e} s
Time to hard-boil a small egg taken from a hot room: {egg(47, T0=25, Ty=75):.2e} s
Time to soft-boil a large egg taken from a hot room: {egg(67, T0=25, Ty=65):.2e} s
Time to hard-boil a large egg taken from a hot room: {egg(67, T0=25, Ty=75):.2e} s
""")
Time to soft-boil a small egg taken from the fridge: 2.75e+04 s
Time to hard-boil a small egg taken from the fridge: 4.01e+04 s
Time to soft-boil a large egg taken from the fridge: 3.49e+04 s
Time to hard-boil a large egg taken from the fridge: 5.08e+04 s
Time to soft-boil a small egg taken from a hot room: 1.83e+04 s
Time to hard-boil a small egg taken from a hot room: 3.09e+04 s
Time to soft-boil a large egg taken from a hot room: 2.31e+04 s
Time to hard-boil a large egg taken from a hot room: 3.91e+04 s

# Exercise 3.24: Write a function for numerical differentiation_____no_output_____

def diff(f, x, h=1e-5):
    """Calculate the derivative of `h` at `x`."""
    return (f(x + h) - f(x - h)) / 2 / h

def test_diff():
    """Verify `diff` using `f` = 2x^2 + 3x + 7 and `x` = -1."""
    f = lambda x: 2*x**2 + 3*x + 7
    x = -1
    expected = -1
    computed = diff(f, x)
    error = abs(expected - computed)
    tolerance = 1e-10
    success = error < tolerance
    msg = f"""
    Error testing `diff` using `f` = 2x^2 + 3x + 7 and `x` = -1.
    Expected:  {expected:13.6e}
    Computed:  {computed:13.6e}
    Error:     {error:13.6e}
    Tolerance: {tolerance:13.6e}"""
    assert success, msg_____no_output_____from math import exp, cos, pi, log

def application():
    print("exp(0):         Error = %12.6e" % abs(1 - diff(exp, 0, .01)))
    f = lambda x: exp(-2 * x**2)
    print("exp(-2 * 0**2): Error = %12.6e" % abs(0 - diff(f, 0, .01)))
    print("cos(x):         Error = %12.6e" % abs(0 - diff(cos, 2*pi, .01)))
    print("ln(x):          Error = %12.6e" % abs(1 - diff(log, 1, .01)))

application()exp(0):         Error = 1.666675e-05
exp(-2 * 0**2): Error = 0.000000e+00
cos(x):         Error = 0.000000e+00
ln(x):          Error = 3.333533e-05

# Exercise 3.25: Implement the factorial function_____no_output_____

def fact(n):
    if n < 0 or not float(n).is_integer():
        raise ValueError("Factorial numbers are only defined for positive integers.")
    if n <= 1:
        return 1
    else:
        return n * fact(n - 1)_____no_output_____def test_fact():
    n = 4
    # Check an arbitrary case n=4
    expected = 4*3*2*1
    computed = fact(n)
    assert computed == expected # Check the special cases
    assert fact(0) == 1
    assert fact(1) == 1

test_fact()_____no_output_____

# Exercise 3.26: Compute velocity and acceleration from 1D position data_____no_output_____

def kinematics(i, x, t):
    v = (x[i + 1] - x[i - 1]) / (t[i + 1] - t[i - 1])
    a = 2 / (t[i + 1] - t[i - 1]) * ((x[i + 1] - x[i]) / (t[i + 1] - t[i]) - (x[i] - x[i - 1]) / (t[i] - t[i - 1]))
    return v, a

def test_kinematics():
    v = 2
    t = 0, 0.5, 1.5, 2.2
    x = 0, 1, 3, 4.4
    expected = 2, 0, 2, 0
    computed = kinematics(1, x, t) + kinematics(2, x, t)
    error = sum(abs(expected[i] - computed[i]) for i in range(len(expected))) / len(expected)
    tolerance = 1e-10
    success = error < tolerance
    msg = f"""
    Error testing `kinematics`.
    Expected:  {expected}
    Computed:  {computed}
    Error:     {error:13.6e}
    Tolerance: {tolerance:13.6e}"""
    assert success, msg

test_kinematics()_____no_output_____

# Exercise 3.27: Find the max and min values of a function_____no_output_____

def maxmin(f, a, b, n=10000):
    values = [f(x) for x in [a + i*(b - a)/(n - 1) for i in range(n)]]
    return max(values), min(values)

from math import pi, cos
def test_maxmin():
    a = -pi/2
    b = 2*pi
    f = cos
    expected = 1, -1
    computed = maxmin(f, a, b)
    error = sum(abs(x - y) for x, y in zip(expected, computed)) / len(expected)
    tolerance = 1e-6
    success = error < tolerance
    msg = f"""
    Error testing `maxmin`.
    Expected:  {expected}
    Computed:  {computed}
    Error:     {error:13.6e}
    Tolerance: {tolerance:13.6e}"""
    assert success, msg

test_maxmin()_____no_output_____

# Exercise 3.28: Find the max and min elements in a list_____no_output_____

def max_(list_):
    m = list_[0]
    for x in list_[1:]:
        if x > m:
            m = x
    return m

def min_(list_):
    m = list_[0]
    for x in list_[1:]:
        if x < m:
            m = x
    return m

print(max_([1, 2, 3, 4, 5]))
print(min_([1, 2, 3, 4, 5]))5
1

# Exercise 3.29: Implement the Heaviside function_____no_output_____

def H(x):
    return 0 if x < 0 else 1

def test_H():
    """Verify `H` using `x` elem. {-10, -10e-15, 0, 10e-15, 10}."""
    expected = 0, 0, 1, 1, 1
    computed = H(-10), H(-10e-15), H(0), H(10e-15), H(10)
    error = sum(abs(x - y) for x, y in zip(expected, computed)) / len(expected)
    tolerance = 1e-12
    success = error < tolerance
    msg = f"""
    Error testing `H`.
    Expected:  {expected}
    Computed:  {computed}
    Error:     {error:13.6e}
    Tolerance: {tolerance:13.6e}"""
    assert success, msg

test_H()_____no_output_____

# Exercise 3.30: Implement a smoothed Heaviside function_____no_output_____

from math import pi, sin

def H_eps(x, eps=.01):
    return 0 if x < -eps else 1 if x > eps else 1/2 + x/2/eps + (pi / 2 * sin(pi * x / eps))

def test_H_eps():
    """Verify `H_eps` using `x` elem. {-10, -1, -0.01, 0, 0.01, 1, 10}."""
    x = -10, -1, -0.01, 0, 0.01, 1, 10
    expected = 0, 0, 0, 0.5, 1, 1, 1
    computed = (H_eps(y) for y in x)
    error = sum(abs(x - y) for x, y in zip(expected, computed)) / len(expected)
    tolerance = 1e-12
    success = error < tolerance
    msg = f"""
    Error testing `H`.
    Expected:  {expected}
    Computed:  {computed}
    Error:     {error:13.6e}
    Tolerance: {tolerance:13.6e}"""
    assert success, msg

test_H_eps()_____no_output_____

# Exercise 3.31: Implement an indicator function_____no_output_____

I1 = lambda x, l, r: int(l <= x <= r)
I2 = lambda x, l, r: H(x - l) * H(r - x)

def test_I():
    """Verify `I1` and `I2` using `l` = -3, `r` = 20 and `x` elem. {-5, `l`, (`l` + `r`) / 2, `r`, 30}."""
    l = -3; r = 20
    x = -5, l, (l + r) / 2, r, 30
    expected = 2 * [0, 1, 1, 1, 0]
    computed = [I1(y, l, r) for y in x] + [I2(y, l, r) for y in x]
    error = sum(abs(x - y) for x, y in zip(expected, computed)) / len(expected)
    tolerance = 1e-12
    success = error < tolerance
    msg = f"""
    Error testing `I1` and `I2`.
    Expected:  {expected}
    Computed:  {computed}
    Error:     {error:13.6e}
    Tolerance: {tolerance:13.6e}"""
    assert success, msg

test_I()_____no_output_____

# Exercise 3.32: Implement a piecewise constant function_____no_output_____

def piecewise(x, data):
    for i, (_, y) in enumerate(data, start=1):
        if len(data) == i or x < data[i][0]:
            return y

def test_piecewise():
    """Verify `picesewise` using `data` = [(0, -1), (1, 0), (1.5, 4)] and `x` elem. {-5, 0, .5, 1, 1.5, 5}."""
    data = [(0, -1), (1, 0), (1.5, 4)]
    x = -5, 0, .5, 1, 1.5, 5
    expected = -1, -1, -1, 0, 4, 4
    computed = [piecewise(x, data) for x in x]
    error = sum(abs(x - y) for x, y in zip(expected, computed)) / len(expected)
    tolerance = 1e-12
    success = error < tolerance
    msg = f"""
    Error testing `piecewise`.
    Expected:  {expected}
    Computed:  {computed}
    Error:     {error:13.6e}
    Tolerance: {tolerance:13.6e}"""
    assert success, msg

test_piecewise()_____no_output_____

# Exercise 3.33: Apply indicator functions_____no_output_____

I3 = lambda x, l, r: int(l <= x < r)    # Interval needs to be half-open for piecewise definition.

def piecewise2(x, data):
    return sum(point[1] * I3(x, point[0], data[i][0] if len(data) != i else x) for i, point in enumerate(data, start=1))

piecewise2(0, [(0, -1), (1, 0), (1.5, 4)])_____no_output_____

# Exercise 3.34: Test your understanding of branching_____no_output_____

# Expected Output:
# quadrant I or II
# quadrant II
# quadrant I or IV
# quadrant I or II
# quadrant I or IV

def where1(x, y):
    if x > 0:
        print("quadrant I or IV")
    if y > 0:
        print("quadrant I or II")
def where2(x, y):
    if x > 0:
        print("quadrant I or IV")
    elif y > 0:
        print("quadrant II")

for x, y in (-1, 1), (1, 1):
    where1(x,y)
    where2(x,y)quadrant I or II
quadrant II
quadrant I or IV
quadrant I or II
quadrant I or IV

# Exercise 3.35: Simulate nested loops by hand_____no_output_____

# Expected output:
# -1, 1, 2

# 1, 0
# 1, 1
# 1, 2
# 7, 0
# 7, 1
# 7, 2

# 2, 1
# 3, 1
# 3, 2

# 7, 4, 2
# 7, 4, 3
# 7, 6, 2
# 7, 6, 3
# 7, 6, 4
# 7, 6, 5

n = 3
for i in range(-1, n):
    if i != 0:
        print(i, end=" ")
print()
print()

for i in range(1, 13, 2*n):
    for j in range(n):
        print(i, j)
print()

for i in range(1, n+1):
    for j in range(i):
        if j:
            print(i, j)
print()

for i in range(1, 13, 2*n):
    for j in range(0, i, 2):
        for k in range(2, j, 1):
            b = i > j > k
            if b:
                print(i, j, k)

# enough thinking for today-1 1 2

1 0
1 1
1 2
7 0
7 1
7 2

2 1
3 1
3 2

7 4 2
7 4 3
7 6 2
7 6 3
7 6 4
7 6 5

# Exercise 3.36: Rewrite a mathematical function_____no_output_____$$
c_i = (1 - \frac{1}{i}) \frac{x}{1 + x} c_{i-1}
$$_____no_output_____

def L3_ci(x, epsilon=1e-6):
    s = c = x / (1 + x)
    a = lambda i: (1 - 1 / i) * x / (1 + x)
    i = 1
    while abs(c) > epsilon:
        i += 1
        c *= a(i)
        s += c
    return s, i

def L3(x, epsilon=1.0E-6):
    x = float(x)
    i = 1
    term = (1.0/i)*(x/(1+x))**i
    s = term
    while abs(term) > epsilon:
        i += 1
        term = (1.0/i)*(x/(1+x))**i
        s += term
    return s, i

def test_L3_ci():
    """Verify `L3_ci` using `x` = 2."""
    x = 2
    expected, _ = L3(x)
    computed, _ = L3_ci(x)
    error = abs(expected - computed)
    tolerance = 1e-12
    success = error < tolerance
    msg = f"""
    Error testing `piecewise`.
    Expected:  {expected:13.6e}
    Computed:  {computed:13.6e}
    Error:     {error:13.6e}
    Tolerance: {tolerance:13.6e}"""
    assert success, msg

test_L3_ci()_____no_output_____

# Exercise 3.37: Make a table for approximations of cos x_____no_output_____

def C(x, n):
    s = c = 1
    for i in range(1, n):
        c *= -x**2 / 2 / i / (2*i - 1)
        s += c
    return s

from math import pi, cos

def C_table():
    n = 5, 25, 50, 100, 200
    x = 4*pi, 6*pi, 8*pi, 10*pi
    print(" ↓x \ n→ | {:^12} | {:^12} | {:^12} | {:^12} | {:^12} ".format(*n))
    print("---------+--------------+--------------+--------------+--------------+--------------")
    for x in x:
        c = [C(x, n) for n in n]
        e = [abs(cos(x) - c) for c in c]
        print(" {:7g} | {:12e} | {:12e} | {:12e} | {:12e} | {:12e} ".format(x, *e))

C_table() ↓x \ n→ |      5       |      25      |      50      |     100      |     200
---------+--------------+--------------+--------------+--------------+--------------
 12.5664 | 1.091347e+04 | 2.826108e-10 | 6.568079e-13 | 6.568079e-13 | 6.568079e-13
 18.8496 | 3.380495e+05 | 1.686243e-01 | 4.437678e-10 | 4.437678e-10 | 4.437678e-10
 25.1327 | 3.614470e+06 | 2.722097e+05 | 4.177693e-07 | 4.177693e-07 | 4.177693e-07
 31.4159 | 2.223789e+07 | 1.716062e+10 | 3.587509e-04 | 3.587559e-04 | 3.587559e-04

# Exercise 3.38: Use None in keyword arguments_____no_output_____

def L2(x, n):
    s = 0
    for i in range(1, n+1):
        s += (1.0/i)*(x/(1.0+x))**i
    return s

def L3(x, epsilon=1.0E-6):
    x = float(x)
    i = 1
    term = (1.0/i)*(x/(1+x))**i
    s = term
    while abs(term) > epsilon:
        i += 1
        term = (1.0/i)*(x/(1+x))**i
        s += term
    return s, i

def L4(x, epsilon=1e-6, n=None, return_n=False):
    if n:
        ln = L2(x, n)
    else:
        ln, n = L3(x, epsilon)
    return (ln, n) if return_n else ln

print(L4(2))
print(L4(2, return_n=True))
print(L4(2, n=5))1.0986111084058483
(1.0986111084058483, 27)
1.0633744855967078

# Exercise 3.39: Write a sort function for a list of 4-tuples_____no_output_____

data = [
    ('Alpha Centauri A',    4.3,  0.26,      1.56),
    ('Alpha Centauri B',    4.3,  0.077,     0.45),
    ('Alpha Centauri C',    4.2,  0.00001,   0.00006),
    ("Barnard's Star",      6.0,  0.00004,   0.0005),
    ('Wolf 359',            7.7,  0.000001,  0.00002),
    ('BD +36 degrees 2147', 8.2,  0.0003,    0.006),
    ('Luyten 726-8 A',      8.4,  0.000003,  0.00006),
    ('Luyten 726-8 B',      8.4,  0.000002,  0.00004),
    ('Sirius A',            8.6,  1.00,      23.6),
    ('Sirius B',            8.6,  0.001,     0.003),
    ('Ross 154',            9.4,  0.00002,   0.0005),
]

print("Sorted with respect to distance from sun:")
print("=========================================")
for name, distance, _, _ in sorted(data, key=lambda x: -x[1]):
    print(f"{name + ':':<21}{distance}")
print()
print("Sorted with respect to apparent brightness:")
print("===========================================")
for name, _, brightness, _ in sorted(data, key=lambda x: -x[2]):
    print(f"{name + ':':<21}{brightness}")
print()
print("Sorted with respect to luminosity:")
print("==================================")
for name, _, _, luminosity in sorted(data, key=lambda x: -x[3]):
    print(f"{name + ':':<21}{luminosity}")Sorted with respect to distance from sun:
=========================================
Ross 154:            9.4
Sirius A:            8.6
Sirius B:            8.6
Luyten 726-8 A:      8.4
Luyten 726-8 B:      8.4
BD +36 degrees 2147: 8.2
Wolf 359:            7.7
Barnard's Star:      6.0
Alpha Centauri A:    4.3
Alpha Centauri B:    4.3
Alpha Centauri C:    4.2

Sorted with respect to apparent brightness:
===========================================
Sirius A:            1.0
Alpha Centauri A:    0.26
Alpha Centauri B:    0.077
Sirius B:            0.001
BD +36 degrees 2147: 0.0003
Barnard's Star:      4e-05
Ross 154:            2e-05
Alpha Centauri C:    1e-05
Luyten 726-8 A:      3e-06
Luyten 726-8 B:      2e-06
Wolf 359:            1e-06

Sorted with respect to luminosity:
==================================
Sirius A:            23.6
Alpha Centauri A:    1.56
Alpha Centauri B:    0.45
BD +36 degrees 2147: 0.006
Sirius B:            0.003
Barnard's Star:      0.0005
Ross 154:            0.0005
Alpha Centauri C:    6e-05
Luyten 726-8 A:      6e-05
Luyten 726-8 B:      4e-05
Wolf 359:            2e-05

# Exercise 3.40: Find prime numbers_____no_output_____

def sieve(n):
    a = list(range(2, n+1))
    ps = []
    while a:
        p = a[0]
        ps.append(p)
        for x in [a for a in a if a % p == 0]:
            a.remove(x)
    return ps
print(sieve(100))[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97]

# Exercise 3.41: Find pairs of characters_____no_output_____

def count_pairs(dna, pair):
    result = 0
    for a, b in zip(dna, dna[1:]):
        result += a + b == pair
    return result

count_pairs("ABCACBAB", "AB")_____no_output_____

# Exercise 3.42: Count substrings_____no_output_____

def count_subs(dna, string):
    result = 0
    d = len(string)
    for i in range(len(dna)):
        result += dna[i : i+d] == string
    return result

count_subs("ACGTTACGGAACG", "ACG")_____no_output_____

# Exercise 3.43: Resolve a problem with a function_____no_output_____

# because the function only returns something if x <= 4._____no_output_____def f(x):
    if 0 <= x <= 2:
        return x**2
    elif 2 < x <= 4:
        return 4
    elif x < 0:
        return 0

print(f(5))None

# Exercise 3.44: Determine the types of some objects_____no_output_____

def makelist(start, stop, inc):
    value = start
    result = []
    while value <= stop:
        result.append(value)
        value = value + inc
    return result

l1 = makelist(0, 100, 1)
"""
loop:
result = [0]
value = 1
...
result = [0:100:1]    # int elements
"""
print(l1)

l2 = makelist(0, 100, 1.0)
"""
loop:
result = [0]
value = 1
...
result = [0:100:1]    # float elements
"""
print(l2)

l3 = makelist(-1, 1, 0.1)
"""
loop:
result = [-1]
value = -0.9
...
result = [-1:1:0.1]    # float elements
"""
print(l3)

l4 = makelist(10, 20, 20)
"""
loop:
result = [10]
value = 30
...
result = [10]    # int element
"""
print(l4)

# l5 = makelist([1,2], [3,4], [5])
"""
loop:
result = [1, 2]
value = [1, 2, 5]
....
"""

# l6 = makelist((1,-1,1), ("myfile.dat", "yourfile.dat"))
"""
`inc` is not specified!
"""

# l7 = makelist("myfile.dat", "yourfile.dat", "herfile.dat")
"""
loop:
result = "myfile.dat"
value = "myfile.datherfile.dat"
....
"""
print()[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100]
[0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 14.0, 15.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 43.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 52.0, 53.0, 54.0, 55.0, 56.0, 57.0, 58.0, 59.0, 60.0, 61.0, 62.0, 63.0, 64.0, 65.0, 66.0, 67.0, 68.0, 69.0, 70.0, 71.0, 72.0, 73.0, 74.0, 75.0, 76.0, 77.0, 78.0, 79.0, 80.0, 81.0, 82.0, 83.0, 84.0, 85.0, 86.0, 87.0, 88.0, 89.0, 90.0, 91.0, 92.0, 93.0, 94.0, 95.0, 96.0, 97.0, 98.0, 99.0, 100.0]
[-1, -0.9, -0.8, -0.7000000000000001, -0.6000000000000001, -0.5000000000000001, -0.40000000000000013, -0.30000000000000016, -0.20000000000000015, -0.10000000000000014, -1.3877787807814457e-16, 0.09999999999999987, 0.19999999999999987, 0.2999999999999999, 0.3999999999999999, 0.4999999999999999, 0.5999999999999999, 0.6999999999999998, 0.7999999999999998, 0.8999999999999998, 0.9999999999999998]
[10]

# Exercise 3.45: Find an error in a program_____no_output_____

from math import exp, sin

def expsin(x, p, q):
    return exp(p*x)*sin(q*x)

def f(x, m, n, r, s):
    return expsin(x, r, m) + expsin(x, s, n)

x = 2.5
print(f(x, 0.1, 0.2, 1, 1))8.854595968125068


# Notebook from csbobby/STAR_Codes
Path: code/QA_Visualization.ipynb

import pickle
import json
from visualization_tools.qa_visualization import show_generated_qa,show_select_qa_____no_output_____

# Path configs_____no_output_____

annotation_dir = '../annotations/STAR_classes/'
# plase replace following data path to your local path
# all data can be download from our homepages (http://star.csail.mit.edu or https://bobbywu.com/STAR)
root_dir = '/root/user/'
video_dir= root_dir +'Charades/'
frame_dir = root_dir + 'STAR/frames/Charades_v1_480/'
pose_dir = root_dir + 'STAR/pose/'
qa_train_dir = root_dir + 'STAR_train.json'
qa_val_dir = root_dir + 'STAR/STAR_val.json'
qa_test_dir = root_dir + 'STAR/STAR_test.json'
save_video_dir = root_dir + 'visualization_tmp/'
fps = pickle.load(open(annotation_dir + 'video_fps','rb'))_____no_output_____

# Load STAR_____no_output_____

QAList = json.load(open(qa_train_dir))+json.load(open(qa_val_dir))+json.load(open(qa_test_dir))_____no_output_____

# Visulization Setting_____no_output_____

vis_kwargs = {
    # data_kwargs
    'fps':fps,
    'raw_frame_dir':frame_dir,
    'raw_video_dir': video_dir,
    'pose_dir':pose_dir,
    'save_video_dir':save_video_dir,
    # group_kwargs
    'show_by_vid': True,
    'show_by_qtype': True,
    # vis_kwargs
    'vis_meta':True,
    'vis_q_a_o':True,
    'vis_v':True,
    'vis_kf':False,
    'vis_sg':False,
    'vis_pose':False,
    'vis_box':False,
    'max_show_num':5,
    # query_kwargs work in select module only
    'vid':'',
    'qid_':['Interaction_T1_17'],
    }_____no_output_____

# Select by QID/VID_____no_output_____

show_select_qa(QAList,**vis_kwargs)_____no_output_____

# Visulization All QA_____no_output_____

show_generated_qa(QAList,**vis_kwargs)_____no_output_____


# Notebook from Grant-Steinfeld/astronomy-notebooks
Path: Interactive-Sky.ipynb

# Simple star field

Moving your mouse pointer across the display below
will rotate the celestial sphere,
showing the brightest stars down to magnitude 5._____no_output_____

<code>
%load_ext autoreload
%autoreload 2_____no_output_____from lib.downloader import download_all
download_all()Getting data file ftp://cdsarc.u-strasbg.fr/pub/cats/I/239/hip_main.dat.gz...
Getting data file ftp://cdsarc.u-strasbg.fr/pub/cats/VI/42/data.dat...
Getting data file http://ssd.jpl.nasa.gov/dat/ELEMENTS.COMET...
Getting data file http://ssd.jpl.nasa.gov/dat/ELEMENTS.NUMBR.gz...
Getting data file http://ssd.jpl.nasa.gov/dat/ELEMENTS.UNNUM.gz...
Getting data file http://www.celestrak.com/NORAD/elements/visual.txt...
Getting data file http://www.pbarbier.com/constellations/bound_verts_18.txt...
Getting data file https://raw.github.com/astronexus/HYG-Database/master/hygfull.csv...
Done!
from lib import sky
sky.starfield()_____no_output_____
</code>