# Notebook from PhysicsTeacher13/AlphaFold_updated_2022
Path: notebooks/AlphaFold.ipynb

# AlphaFold Colab

This Colab notebook allows you to easily predict the structure of a protein using a slightly simplified version of [AlphaFold v2.1.0](https://doi.org/10.1038/s41586-021-03819-2).

**Differences to AlphaFold v2.1.0**

In comparison to AlphaFold v2.1.0, this Colab notebook uses **no templates (homologous structures)** and a selected portion of the [BFD database](https://bfd.mmseqs.com/). We have validated these changes on several thousand recent PDB structures. While accuracy will be near-identical to the full AlphaFold system on many targets, a small fraction have a large drop in accuracy due to the smaller MSA and lack of templates. For best reliability, we recommend instead using the [full open source AlphaFold](https://github.com/deepmind/alphafold/), or the [AlphaFold Protein Structure Database](https://alphafold.ebi.ac.uk/).

**This Colab has a small drop in average accuracy for multimers compared to local AlphaFold installation, for full multimer accuracy it is highly recommended to run [AlphaFold locally](https://github.com/deepmind/alphafold#running-alphafold).** Moreover, the AlphaFold-Multimer requires searching for MSA for every unique sequence in the complex, hence it is substantially slower. If your notebook times-out due to slow multimer MSA search, we recommend either using Colab Pro or running AlphaFold locally.

Please note that this Colab notebook is provided as an early-access prototype and is not a finished product. It is provided for theoretical modelling only and caution should be exercised in its use.

**Citing this work**

Any publication that discloses findings arising from using this notebook should [cite](https://github.com/deepmind/alphafold/#citing-this-work) the [AlphaFold paper](https://doi.org/10.1038/s41586-021-03819-2).

**Licenses**

This Colab uses the [AlphaFold model parameters](https://github.com/deepmind/alphafold/#model-parameters-license) which are subject to the Creative Commons Attribution 4.0 International ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/legalcode)) license. The Colab itself is provided under the [Apache 2.0 license](https://www.apache.org/licenses/LICENSE-2.0). See the full license statement below.

**More information**

You can find more information about how AlphaFold works in the following papers:

*   [AlphaFold methods paper](https://www.nature.com/articles/s41586-021-03819-2)
*   [AlphaFold predictions of the human proteome paper](https://www.nature.com/articles/s41586-021-03828-1)
*   [AlphaFold-Multimer paper](https://www.biorxiv.org/content/10.1101/2021.10.04.463034v1)

FAQ on how to interpret AlphaFold predictions are [here](https://alphafold.ebi.ac.uk/faq).

If you have any questions not covered in the FAQ, please contact the AlphaFold team at alphafold@deepmind.com.

**Get in touch**

We would love to hear your feedback and understand how AlphaFold has been useful in your research. Share your stories with us at alphafold@deepmind.com.
_____no_output_____## Setup

Start by running the 2 cells below to set up AlphaFold and all required software._____no_output_____

#@title 1. Install third-party software

#@markdown Please execute this cell by pressing the _Play_ button
#@markdown on the left to download and import third-party software
#@markdown in this Colab notebook. (See the [acknowledgements](https://github.com/deepmind/alphafold/#acknowledgements) in our readme.)

#@markdown **Note**: This installs the software on the Colab
#@markdown notebook in the cloud and not on your computer.

from IPython.utils import io
import os
import subprocess
import tqdm.notebook

TQDM_BAR_FORMAT = '{l_bar}{bar}| {n_fmt}/{total_fmt} [elapsed: {elapsed} remaining: {remaining}]'

try:
  with tqdm.notebook.tqdm(total=100, bar_format=TQDM_BAR_FORMAT) as pbar:
    with io.capture_output() as captured:
      # Uninstall default Colab version of TF.
      %shell pip uninstall -y tensorflow

      %shell sudo apt install --quiet --yes hmmer
      pbar.update(6)

      # Install py3dmol.
      %shell pip install py3dmol
      pbar.update(2)

      # Install OpenMM and pdbfixer.
      %shell rm -rf /opt/conda
      %shell wget -q -P /tmp \
        https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
          && bash /tmp/Miniconda3-latest-Linux-x86_64.sh -b -p /opt/conda \
          && rm /tmp/Miniconda3-latest-Linux-x86_64.sh
      pbar.update(9)

      PATH=%env PATH
      %env PATH=/opt/conda/bin:{PATH}
      %shell conda update -qy conda \
          && conda install -qy -c conda-forge \
            python=3.7 \
            openmm=7.5.1 \
            pdbfixer
      pbar.update(80)

      # Create a ramdisk to store a database chunk to make Jackhmmer run fast.
      %shell sudo mkdir -m 777 --parents /tmp/ramdisk
      %shell sudo mount -t tmpfs -o size=9G ramdisk /tmp/ramdisk
      pbar.update(2)

      %shell wget -q -P /content \
        https://git.scicore.unibas.ch/schwede/openstructure/-/raw/7102c63615b64735c4941278d92b554ec94415f8/modules/mol/alg/src/stereo_chemical_props.txt
      pbar.update(1)
except subprocess.CalledProcessError:
  print(captured)
  raise_____no_output_____#@title 2. Download AlphaFold

#@markdown Please execute this cell by pressing the *Play* button on
#@markdown the left.

GIT_REPO = 'https://github.com/deepmind/alphafold'

SOURCE_URL = 'https://storage.googleapis.com/alphafold/alphafold_params_colab_2022-03-02.tar'
PARAMS_DIR = './alphafold/data/params'
PARAMS_PATH = os.path.join(PARAMS_DIR, os.path.basename(SOURCE_URL))

try:
  with tqdm.notebook.tqdm(total=100, bar_format=TQDM_BAR_FORMAT) as pbar:
    with io.capture_output() as captured:
      %shell rm -rf alphafold
      %shell git clone --branch main {GIT_REPO} alphafold
      pbar.update(8)
      # Install the required versions of all dependencies.
      %shell pip3 install -r ./alphafold/requirements.txt
      # Run setup.py to install only AlphaFold.
      %shell pip3 install --no-dependencies ./alphafold
      pbar.update(10)

      # Apply OpenMM patch.
      %shell pushd /opt/conda/lib/python3.7/site-packages/ && \
          patch -p0 < /content/alphafold/docker/openmm.patch && \
          popd

      # Make sure stereo_chemical_props.txt is in all locations where it could be searched for.
      %shell mkdir -p /content/alphafold/alphafold/common
      %shell cp -f /content/stereo_chemical_props.txt /content/alphafold/alphafold/common
      %shell mkdir -p /opt/conda/lib/python3.7/site-packages/alphafold/common/
      %shell cp -f /content/stereo_chemical_props.txt /opt/conda/lib/python3.7/site-packages/alphafold/common/

      %shell mkdir --parents "{PARAMS_DIR}"
      %shell wget -O "{PARAMS_PATH}" "{SOURCE_URL}"
      pbar.update(27)

      %shell tar --extract --verbose --file="{PARAMS_PATH}" \
        --directory="{PARAMS_DIR}" --preserve-permissions
      %shell rm "{PARAMS_PATH}"
      pbar.update(55)
except subprocess.CalledProcessError:
  print(captured)
  raise

import jax
if jax.local_devices()[0].platform == 'tpu':
  raise RuntimeError('Colab TPU runtime not supported. Change it to GPU via Runtime -> Change Runtime Type -> Hardware accelerator -> GPU.')
elif jax.local_devices()[0].platform == 'cpu':
  raise RuntimeError('Colab CPU runtime not supported. Change it to GPU via Runtime -> Change Runtime Type -> Hardware accelerator -> GPU.')
else:
  print(f'Running with {jax.local_devices()[0].device_kind} GPU')

# Make sure everything we need is on the path.
import sys
sys.path.append('/opt/conda/lib/python3.7/site-packages')
sys.path.append('/content/alphafold')

# Make sure all necessary environment variables are set.
import os
os.environ['TF_FORCE_UNIFIED_MEMORY'] = '1'
os.environ['XLA_PYTHON_CLIENT_MEM_FRACTION'] = '2.0'_____no_output_____

## Making a prediction

Please paste the sequence of your protein in the text box below, then run the remaining cells via _Runtime_ > _Run after_. You can also run the cells individually by pressing the _Play_ button on the left.

Note that the search against databases and the actual prediction can take some time, from minutes to hours, depending on the length of the protein and what type of GPU you are allocated by Colab (see FAQ below)._____no_output_____

#@title 3. Enter the amino acid sequence(s) to fold ⬇
#@markdown Enter the amino acid sequence(s) to fold:
#@markdown * If you enter only a single sequence, the monomer model will be used.
#@markdown * If you enter multiple sequences, the multimer model will be used.

from alphafold.notebooks import notebook_utils

sequence_1 = 'MAAHKGAEHHHKAAEHHEQAAKHHHAAAEHHEKGEHEQAAHHADTAYAHHKHAEEHAAQAAKHDAEHHAPKPH'  #@param {type:"string"}
sequence_2 = ''  #@param {type:"string"}
sequence_3 = ''  #@param {type:"string"}
sequence_4 = ''  #@param {type:"string"}
sequence_5 = ''  #@param {type:"string"}
sequence_6 = ''  #@param {type:"string"}
sequence_7 = ''  #@param {type:"string"}
sequence_8 = ''  #@param {type:"string"}

input_sequences = (sequence_1, sequence_2, sequence_3, sequence_4,
                   sequence_5, sequence_6, sequence_7, sequence_8)

MIN_SINGLE_SEQUENCE_LENGTH = 16
MAX_SINGLE_SEQUENCE_LENGTH = 2500
MAX_MULTIMER_LENGTH = 2500

# Validate the input.
sequences, model_type_to_use = notebook_utils.validate_input(
    input_sequences=input_sequences,
    min_length=MIN_SINGLE_SEQUENCE_LENGTH,
    max_length=MAX_SINGLE_SEQUENCE_LENGTH,
    max_multimer_length=MAX_MULTIMER_LENGTH)_____no_output_____#@title 4. Search against genetic databases

#@markdown Once this cell has been executed, you will see
#@markdown statistics about the multiple sequence alignment
#@markdown (MSA) that will be used by AlphaFold. In particular,
#@markdown you’ll see how well each residue is covered by similar
#@markdown sequences in the MSA.

# --- Python imports ---
import collections
import copy
from concurrent import futures
import json
import random

from urllib import request
from google.colab import files
from matplotlib import gridspec
import matplotlib.pyplot as plt
import numpy as np
import py3Dmol

from alphafold.model import model
from alphafold.model import config
from alphafold.model import data

from alphafold.data import feature_processing
from alphafold.data import msa_pairing
from alphafold.data import pipeline
from alphafold.data import pipeline_multimer
from alphafold.data.tools import jackhmmer

from alphafold.common import protein

from alphafold.relax import relax
from alphafold.relax import utils

from IPython import display
from ipywidgets import GridspecLayout
from ipywidgets import Output

# Color bands for visualizing plddt
PLDDT_BANDS = [(0, 50, '#FF7D45'),
               (50, 70, '#FFDB13'),
               (70, 90, '#65CBF3'),
               (90, 100, '#0053D6')]

# --- Find the closest source ---
test_url_pattern = 'https://storage.googleapis.com/alphafold-colab{:s}/latest/uniref90_2021_03.fasta.1'
ex = futures.ThreadPoolExecutor(3)
def fetch(source):
  request.urlretrieve(test_url_pattern.format(source))
  return source
fs = [ex.submit(fetch, source) for source in ['', '-europe', '-asia']]
source = None
for f in futures.as_completed(fs):
  source = f.result()
  ex.shutdown()
  break

JACKHMMER_BINARY_PATH = '/usr/bin/jackhmmer'
DB_ROOT_PATH = f'https://storage.googleapis.com/alphafold-colab{source}/latest/'
# The z_value is the number of sequences in a database.
MSA_DATABASES = [
    {'db_name': 'uniref90',
     'db_path': f'{DB_ROOT_PATH}uniref90_2021_03.fasta',
     'num_streamed_chunks': 59,
     'z_value': 135_301_051},
    {'db_name': 'smallbfd',
     'db_path': f'{DB_ROOT_PATH}bfd-first_non_consensus_sequences.fasta',
     'num_streamed_chunks': 17,
     'z_value': 65_984_053},
    {'db_name': 'mgnify',
     'db_path': f'{DB_ROOT_PATH}mgy_clusters_2019_05.fasta',
     'num_streamed_chunks': 71,
     'z_value': 304_820_129},
]

# Search UniProt and construct the all_seq features only for heteromers, not homomers.
if model_type_to_use == notebook_utils.ModelType.MULTIMER and len(set(sequences)) > 1:
  MSA_DATABASES.extend([
      # Swiss-Prot and TrEMBL are concatenated together as UniProt.
      {'db_name': 'uniprot',
       'db_path': f'{DB_ROOT_PATH}uniprot_2021_03.fasta',
       'num_streamed_chunks': 98,
       'z_value': 219_174_961 + 565_254},
  ])

TOTAL_JACKHMMER_CHUNKS = sum([cfg['num_streamed_chunks'] for cfg in MSA_DATABASES])

MAX_HITS = {
    'uniref90': 10_000,
    'smallbfd': 5_000,
    'mgnify': 501,
    'uniprot': 50_000,
}

def get_msa(fasta_path):
  """Searches for MSA for the given sequence using chunked Jackhmmer search."""

  # Run the search against chunks of genetic databases (since the genetic
  # databases don't fit in Colab disk).
  raw_msa_results = collections.defaultdict(list)
  with tqdm.notebook.tqdm(total=TOTAL_JACKHMMER_CHUNKS, bar_format=TQDM_BAR_FORMAT) as pbar:
    def jackhmmer_chunk_callback(i):
      pbar.update(n=1)

    for db_config in MSA_DATABASES:
      db_name = db_config['db_name']
      pbar.set_description(f'Searching {db_name}')
      jackhmmer_runner = jackhmmer.Jackhmmer(
          binary_path=JACKHMMER_BINARY_PATH,
          database_path=db_config['db_path'],
          get_tblout=True,
          num_streamed_chunks=db_config['num_streamed_chunks'],
          streaming_callback=jackhmmer_chunk_callback,
          z_value=db_config['z_value'])
      # Group the results by database name.
      raw_msa_results[db_name].extend(jackhmmer_runner.query(fasta_path))

  return raw_msa_results

features_for_chain = {}
raw_msa_results_for_sequence = {}
for sequence_index, sequence in enumerate(sequences, start=1):
  print(f'\nGetting MSA for sequence {sequence_index}')

  fasta_path = f'target_{sequence_index}.fasta'
  with open(fasta_path, 'wt') as f:
    f.write(f'>query\n{sequence}')

  # Don't do redundant work for multiple copies of the same chain in the multimer.
  if sequence not in raw_msa_results_for_sequence:
    raw_msa_results = get_msa(fasta_path=fasta_path)
    raw_msa_results_for_sequence[sequence] = raw_msa_results
  else:
    raw_msa_results = copy.deepcopy(raw_msa_results_for_sequence[sequence])

  # Extract the MSAs from the Stockholm files.
  # NB: deduplication happens later in pipeline.make_msa_features.
  single_chain_msas = []
  uniprot_msa = None
  for db_name, db_results in raw_msa_results.items():
    merged_msa = notebook_utils.merge_chunked_msa(
        results=db_results, max_hits=MAX_HITS.get(db_name))
    if merged_msa.sequences and db_name != 'uniprot':
      single_chain_msas.append(merged_msa)
      msa_size = len(set(merged_msa.sequences))
      print(f'{msa_size} unique sequences found in {db_name} for sequence {sequence_index}')
    elif merged_msa.sequences and db_name == 'uniprot':
      uniprot_msa = merged_msa

  notebook_utils.show_msa_info(single_chain_msas=single_chain_msas, sequence_index=sequence_index)

  # Turn the raw data into model features.
  feature_dict = {}
  feature_dict.update(pipeline.make_sequence_features(
      sequence=sequence, description='query', num_res=len(sequence)))
  feature_dict.update(pipeline.make_msa_features(msas=single_chain_msas))
  # We don't use templates in AlphaFold Colab notebook, add only empty placeholder features.
  feature_dict.update(notebook_utils.empty_placeholder_template_features(
      num_templates=0, num_res=len(sequence)))

  # Construct the all_seq features only for heteromers, not homomers.
  if model_type_to_use == notebook_utils.ModelType.MULTIMER and len(set(sequences)) > 1:
    valid_feats = msa_pairing.MSA_FEATURES + (
        'msa_species_identifiers',
    )
    all_seq_features = {
        f'{k}_all_seq': v for k, v in pipeline.make_msa_features([uniprot_msa]).items()
        if k in valid_feats}
    feature_dict.update(all_seq_features)

  features_for_chain[protein.PDB_CHAIN_IDS[sequence_index - 1]] = feature_dict

# Do further feature post-processing depending on the model type.
if model_type_to_use == notebook_utils.ModelType.MONOMER:
  np_example = features_for_chain[protein.PDB_CHAIN_IDS[0]]

elif model_type_to_use == notebook_utils.ModelType.MULTIMER:
  all_chain_features = {}
  for chain_id, chain_features in features_for_chain.items():
    all_chain_features[chain_id] = pipeline_multimer.convert_monomer_features(
        chain_features, chain_id)

  all_chain_features = pipeline_multimer.add_assembly_features(all_chain_features)

  np_example = feature_processing.pair_and_merge(
      all_chain_features=all_chain_features)

  # Pad MSA to avoid zero-sized extra_msa.
  np_example = pipeline_multimer.pad_msa(np_example, min_num_seq=512)_____no_output_____#@title 5. Run AlphaFold and download prediction

#@markdown Once this cell has been executed, a zip-archive with
#@markdown the obtained prediction will be automatically downloaded
#@markdown to your computer.

#@markdown In case you are having issues with the relaxation stage, you can disable it below.
#@markdown Warning: This means that the prediction might have distracting
#@markdown small stereochemical violations.

run_relax = True  #@param {type:"boolean"}

# --- Run the model ---
if model_type_to_use == notebook_utils.ModelType.MONOMER:
  model_names = config.MODEL_PRESETS['monomer'] + ('model_2_ptm',)
elif model_type_to_use == notebook_utils.ModelType.MULTIMER:
  model_names = config.MODEL_PRESETS['multimer']

output_dir = 'prediction'
os.makedirs(output_dir, exist_ok=True)

plddts = {}
ranking_confidences = {}
pae_outputs = {}
unrelaxed_proteins = {}

with tqdm.notebook.tqdm(total=len(model_names) + 1, bar_format=TQDM_BAR_FORMAT) as pbar:
  for model_name in model_names:
    pbar.set_description(f'Running {model_name}')

    cfg = config.model_config(model_name)
    if model_type_to_use == notebook_utils.ModelType.MONOMER:
      cfg.data.eval.num_ensemble = 1
    elif model_type_to_use == notebook_utils.ModelType.MULTIMER:
      cfg.model.num_ensemble_eval = 1
    params = data.get_model_haiku_params(model_name, './alphafold/data')
    model_runner = model.RunModel(cfg, params)
    processed_feature_dict = model_runner.process_features(np_example, random_seed=0)
    prediction = model_runner.predict(processed_feature_dict, random_seed=random.randrange(sys.maxsize))

    mean_plddt = prediction['plddt'].mean()

    if model_type_to_use == notebook_utils.ModelType.MONOMER:
      if 'predicted_aligned_error' in prediction:
        pae_outputs[model_name] = (prediction['predicted_aligned_error'],
                                   prediction['max_predicted_aligned_error'])
      else:
        # Monomer models are sorted by mean pLDDT. Do not put monomer pTM models here as they
        # should never get selected.
        ranking_confidences[model_name] = prediction['ranking_confidence']
        plddts[model_name] = prediction['plddt']
    elif model_type_to_use == notebook_utils.ModelType.MULTIMER:
      # Multimer models are sorted by pTM+ipTM.
      ranking_confidences[model_name] = prediction['ranking_confidence']
      plddts[model_name] = prediction['plddt']
      pae_outputs[model_name] = (prediction['predicted_aligned_error'],
                                 prediction['max_predicted_aligned_error'])

    # Set the b-factors to the per-residue plddt.
    final_atom_mask = prediction['structure_module']['final_atom_mask']
    b_factors = prediction['plddt'][:, None] * final_atom_mask
    unrelaxed_protein = protein.from_prediction(
        processed_feature_dict,
        prediction,
        b_factors=b_factors,
        remove_leading_feature_dimension=(
            model_type_to_use == notebook_utils.ModelType.MONOMER))
    unrelaxed_proteins[model_name] = unrelaxed_protein

    # Delete unused outputs to save memory.
    del model_runner
    del params
    del prediction
    pbar.update(n=1)

  # --- AMBER relax the best model ---

  # Find the best model according to the mean pLDDT.
  best_model_name = max(ranking_confidences.keys(), key=lambda x: ranking_confidences[x])

  if run_relax:
    pbar.set_description(f'AMBER relaxation')
    amber_relaxer = relax.AmberRelaxation(
        max_iterations=0,
        tolerance=2.39,
        stiffness=10.0,
        exclude_residues=[],
        max_outer_iterations=3,
        use_gpu=True)
    relaxed_pdb, _, _ = amber_relaxer.process(prot=unrelaxed_proteins[best_model_name])
  else:
    print('Warning: Running without the relaxation stage.')
    relaxed_pdb = protein.to_pdb(unrelaxed_proteins[best_model_name])
  pbar.update(n=1)  # Finished AMBER relax.

# Construct multiclass b-factors to indicate confidence bands
# 0=very low, 1=low, 2=confident, 3=very high
banded_b_factors = []
for plddt in plddts[best_model_name]:
  for idx, (min_val, max_val, _) in enumerate(PLDDT_BANDS):
    if plddt >= min_val and plddt <= max_val:
      banded_b_factors.append(idx)
      break
banded_b_factors = np.array(banded_b_factors)[:, None] * final_atom_mask
to_visualize_pdb = utils.overwrite_b_factors(relaxed_pdb, banded_b_factors)

# Write out the prediction
pred_output_path = os.path.join(output_dir, 'selected_prediction.pdb')
with open(pred_output_path, 'w') as f:
  f.write(relaxed_pdb)

# --- Visualise the prediction & confidence ---
show_sidechains = True
def plot_plddt_legend():
  """Plots the legend for pLDDT."""
  thresh = ['Very low (pLDDT < 50)',
            'Low (70 > pLDDT > 50)',
            'Confident (90 > pLDDT > 70)',
            'Very high (pLDDT > 90)']

  colors = [x[2] for x in PLDDT_BANDS]

  plt.figure(figsize=(2, 2))
  for c in colors:
    plt.bar(0, 0, color=c)
  plt.legend(thresh, frameon=False, loc='center', fontsize=20)
  plt.xticks([])
  plt.yticks([])
  ax = plt.gca()
  ax.spines['right'].set_visible(False)
  ax.spines['top'].set_visible(False)
  ax.spines['left'].set_visible(False)
  ax.spines['bottom'].set_visible(False)
  plt.title('Model Confidence', fontsize=20, pad=20)
  return plt

# Show the structure coloured by chain if the multimer model has been used.
if model_type_to_use == notebook_utils.ModelType.MULTIMER:
  multichain_view = py3Dmol.view(width=800, height=600)
  multichain_view.addModelsAsFrames(to_visualize_pdb)
  multichain_style = {'cartoon': {'colorscheme': 'chain'}}
  multichain_view.setStyle({'model': -1}, multichain_style)
  multichain_view.zoomTo()
  multichain_view.show()

# Color the structure by per-residue pLDDT
color_map = {i: bands[2] for i, bands in enumerate(PLDDT_BANDS)}
view = py3Dmol.view(width=800, height=600)
view.addModelsAsFrames(to_visualize_pdb)
style = {'cartoon': {'colorscheme': {'prop': 'b', 'map': color_map}}}
if show_sidechains:
  style['stick'] = {}
view.setStyle({'model': -1}, style)
view.zoomTo()

grid = GridspecLayout(1, 2)
out = Output()
with out:
  view.show()
grid[0, 0] = out

out = Output()
with out:
  plot_plddt_legend().show()
grid[0, 1] = out

display.display(grid)

# Display pLDDT and predicted aligned error (if output by the model).
if pae_outputs:
  num_plots = 2
else:
  num_plots = 1

plt.figure(figsize=[8 * num_plots, 6])
plt.subplot(1, num_plots, 1)
plt.plot(plddts[best_model_name])
plt.title('Predicted LDDT')
plt.xlabel('Residue')
plt.ylabel('pLDDT')

if num_plots == 2:
  plt.subplot(1, 2, 2)
  pae, max_pae = list(pae_outputs.values())[0]
  plt.imshow(pae, vmin=0., vmax=max_pae, cmap='Greens_r')
  plt.colorbar(fraction=0.046, pad=0.04)

  # Display lines at chain boundaries.
  best_unrelaxed_prot = unrelaxed_proteins[best_model_name]
  total_num_res = best_unrelaxed_prot.residue_index.shape[-1]
  chain_ids = best_unrelaxed_prot.chain_index
  for chain_boundary in np.nonzero(chain_ids[:-1] - chain_ids[1:]):
    if chain_boundary.size:
      plt.plot([0, total_num_res], [chain_boundary, chain_boundary], color='red')
      plt.plot([chain_boundary, chain_boundary], [0, total_num_res], color='red')

  plt.title('Predicted Aligned Error')
  plt.xlabel('Scored residue')
  plt.ylabel('Aligned residue')

# Save the predicted aligned error (if it exists).
pae_output_path = os.path.join(output_dir, 'predicted_aligned_error.json')
if pae_outputs:
  # Save predicted aligned error in the same format as the AF EMBL DB.
  pae_data = notebook_utils.get_pae_json(pae=pae, max_pae=max_pae.item())
  with open(pae_output_path, 'w') as f:
    f.write(pae_data)

# --- Download the predictions ---
!zip -q -r {output_dir}.zip {output_dir}
files.download(f'{output_dir}.zip')_____no_output_____

### Interpreting the prediction

In general predicted LDDT (pLDDT) is best used for intra-domain confidence, whereas Predicted Aligned Error (PAE) is best used for determining between domain or between chain confidence.

Please see the [AlphaFold methods paper](https://www.nature.com/articles/s41586-021-03819-2), the [AlphaFold predictions of the human proteome paper](https://www.nature.com/articles/s41586-021-03828-1), and the [AlphaFold-Multimer paper](https://www.biorxiv.org/content/10.1101/2021.10.04.463034v1) as well as [our FAQ](https://alphafold.ebi.ac.uk/faq) on how to interpret AlphaFold predictions._____no_output_____## FAQ & Troubleshooting

*   How do I get a predicted protein structure for my protein?
    *   Click on the _Connect_ button on the top right to get started.
    *   Paste the amino acid sequence of your protein (without any headers) into the “Enter the amino acid sequence to fold”.
    *   Run all cells in the Colab, either by running them individually (with the play button on the left side) or via _Runtime_ > _Run all._ Make sure you run all 5 cells in order.
    *   The predicted protein structure will be downloaded once all cells have been executed. Note: This can take minutes to hours - see below.
*   How long will this take?
    *   Downloading the AlphaFold source code can take up to a few minutes.
    *   Downloading and installing the third-party software can take up to a few minutes.
    *   The search against genetic databases can take minutes to hours.
    *   Running AlphaFold and generating the prediction can take minutes to hours, depending on the length of your protein and on which GPU-type Colab has assigned you.
*   My Colab no longer seems to be doing anything, what should I do?
    *   Some steps may take minutes to hours to complete.
    *   If nothing happens or if you receive an error message, try restarting your Colab runtime via _Runtime_ > _Restart runtime_.
    *   If this doesn’t help, try resetting your Colab runtime via _Runtime_ > _Factory reset runtime_.
*   How does this compare to the open-source version of AlphaFold?
    *   This Colab version of AlphaFold searches a selected portion of the BFD dataset and currently doesn’t use templates, so its accuracy is reduced in comparison to the full version of AlphaFold that is described in the [AlphaFold paper](https://doi.org/10.1038/s41586-021-03819-2) and [Github repo](https://github.com/deepmind/alphafold/) (the full version is available via the inference script).
*   What is a Colab?
    *   See the [Colab FAQ](https://research.google.com/colaboratory/faq.html).
*   I received a warning “Notebook requires high RAM”, what do I do?
    *   The resources allocated to your Colab vary. See the [Colab FAQ](https://research.google.com/colaboratory/faq.html) for more details.
    *   You can execute the Colab nonetheless.
*   I received an error “Colab CPU runtime not supported” or “No GPU/TPU found”, what do I do?
    *   Colab CPU runtime is not supported. Try changing your runtime via _Runtime_ > _Change runtime type_ > _Hardware accelerator_ > _GPU_.
    *   The type of GPU allocated to your Colab varies. See the [Colab FAQ](https://research.google.com/colaboratory/faq.html) for more details.
    *   If you receive “Cannot connect to GPU backend”, you can try again later to see if Colab allocates you a GPU.
    *   [Colab Pro](https://colab.research.google.com/signup) offers priority access to GPUs.
*   I received an error “ModuleNotFoundError: No module named ...”, even though I ran the cell that imports it, what do I do?
    *   Colab notebooks on the free tier time out after a certain amount of time. See the [Colab FAQ](https://research.google.com/colaboratory/faq.html#idle-timeouts). Try rerunning the whole notebook from the beginning.
*   Does this tool install anything on my computer?
    *   No, everything happens in the cloud on Google Colab.
    *   At the end of the Colab execution a zip-archive with the obtained prediction will be automatically downloaded to your computer.
*   How should I share feedback and bug reports?
    *   Please share any feedback and bug reports as an [issue](https://github.com/deepmind/alphafold/issues) on Github.

## Related work

Take a look at these Colab notebooks provided by the community (please note that these notebooks may vary from our validated AlphaFold system and we cannot guarantee their accuracy):

*   The [ColabFold AlphaFold2 notebook](https://colab.research.google.com/github/sokrypton/ColabFold/blob/main/AlphaFold2.ipynb) by Sergey Ovchinnikov, Milot Mirdita and Martin Steinegger, which uses an API hosted at the Södinglab based on the MMseqs2 server ([Mirdita et al. 2019, Bioinformatics](https://academic.oup.com/bioinformatics/article/35/16/2856/5280135)) for the multiple sequence alignment creation.
_____no_output_____# License and Disclaimer

This is not an officially-supported Google product.

This Colab notebook and other information provided is for theoretical modelling only, caution should be exercised in its use. It is provided ‘as-is’ without any warranty of any kind, whether expressed or implied. Information is not intended to be a substitute for professional medical advice, diagnosis, or treatment, and does not constitute medical or other professional advice.

Copyright 2021 DeepMind Technologies Limited.

## AlphaFold Code License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

## Model Parameters License

The AlphaFold parameters are made available under the terms of the Creative Commons Attribution 4.0 International (CC BY 4.0) license. You can find details at: https://creativecommons.org/licenses/by/4.0/legalcode

## Third-party software

Use of the third-party software, libraries or code referred to in the [Acknowledgements section](https://github.com/deepmind/alphafold/#acknowledgements) in the AlphaFold README may be governed by separate terms and conditions or license provisions. Your use of the third-party software, libraries or code is subject to any such terms and you should check that you can comply with any applicable restrictions or terms and conditions before use.

## Mirrored Databases

The following databases have been mirrored by DeepMind, and are available with reference to the following:
* UniProt: v2021\_03 (unmodified), by The UniProt Consortium, available under a [Creative Commons Attribution-NoDerivatives 4.0 International License](http://creativecommons.org/licenses/by-nd/4.0/).
* UniRef90: v2021\_03 (unmodified), by The UniProt Consortium, available under a [Creative Commons Attribution-NoDerivatives 4.0 International License](http://creativecommons.org/licenses/by-nd/4.0/).
* MGnify: v2019\_05 (unmodified), by Mitchell AL et al., available free of all copyright restrictions and made fully and freely available for both non-commercial and commercial use under [CC0 1.0 Universal (CC0 1.0) Public Domain Dedication](https://creativecommons.org/publicdomain/zero/1.0/).
* BFD: (modified), by Steinegger M. and Söding J., modified by DeepMind, available under a [Creative Commons Attribution-ShareAlike 4.0 International License](https://creativecommons.org/licenses/by/4.0/). See the Methods section of the [AlphaFold proteome paper](https://www.nature.com/articles/s41586-021-03828-1) for details._____no_output_____


# Notebook from MarcusFra/workshop
Path: 02_usecases/factorization_machines_recommendations/10_Recommendation_Prepare_User_Item_Dataset.ipynb

# Validating and Importing User-Item-Interaction Data

In this notebook, you will choose a dataset and prepare it for use with Amazon Personalize.

1. [Introduction](#intro)
1. [Choose a dataset or data source](#source)
1. [Prepare your data](#prepare)
1. [Create dataset groups and the interactions dataset](#group_dataset)
1. [Configure an S3 bucket and an IAM role](#bucket_role)
1. [Import the interactions data](#import)

## Introduction

For the most part, the algorithms in Amazon Personalize (called recipes) look to solve different tasks, explained here:

1. **User Personalization** - New release that supports ALL HRNN workflows / user personalization needs, it will be what we use here.
1. **HRNN & HRNN-Metadata** - Recommends items based on previous user interactions with items.
1. **HRNN-Coldstart** - Recommends new items for which interaction data is not yet available.
1. **Personalized-Ranking** - Takes a collection of items and then orders them in probable order of interest using an HRNN-like approach.
1. **SIMS (Similar Items)** - Given one item, recommends other items also interacted with by users.
1. **Popularity-Count** - Recommends the most popular items, if HRNN or HRNN-Metadata do not have an answer - this is returned by default.

No matter the use case, the algorithms all share a base of learning on user-item-interaction data which is defined by 3 core attributes:

1. **UserID** - The user who interacted
1. **ItemID** - The item the user interacted with
1. **Timestamp** - The time at which the interaction occurred

We also support event types and event values defined by:

1. **Event Type** - Categorical label of an event (browse, purchased, rated, etc).
1. **Event Value** - A value corresponding to the event type that occurred. Generally speaking, we look for normalized values between 0 and 1 over the event types. For example, if there are three phases to complete a transaction (clicked, added-to-cart, and purchased), then there would be an event_value for each phase as 0.33, 0.66, and 1.0 respectfully.

The event type and event value fields are additional data which can be used to filter the data sent for training the personalization model. In this particular exercise we will not have an event type or event value.

## Choose a dataset or data source
[Back to top](#top)

As we mentioned, the user-item-iteraction data is key for getting started with the service. This means we need to look for use cases that generate that kind of data, a few common examples are:

1. Video-on-demand applications
1. E-commerce platforms
1. Social media aggregators / platforms

There are a few guidelines for scoping a problem suitable for Personalize. We recommend the values below as a starting point, although the [official limits](https://docs.aws.amazon.com/personalize/latest/dg/limits.html) lie a little lower.

* Authenticated users
* At least 50 unique users
* At least 100 unique items
* At least 2 dozen interactions for each user

Most of the time this is easily attainable, and if you are low in one category, you can often make up for it by having a larger number in another category.

Generally speaking your data will not arrive in a perfect form for Personalize, and will take some modification to be structured correctly. This notebook looks to guide you through all of that.

To begin with, we are going to use the latest MovieLens dataset, this dataset has over 25 million interactions and a rich collection of metadata for items, there is also a smaller version of this dataset, which can be used to shorten training times, while still incorporating the same capabilities as the full dataset. Set USE_FULL_MOVIELENS to True to use the full dataset._____no_output_____![](img/01.png)_____no_output_____

USE_FULL_MOVIELENS = False_____no_output_____

First, you will download the dataset and unzip it in a new folder using the code below._____no_output_____

data_dir = "poc_data"
!mkdir $data_dir

if not USE_FULL_MOVIELENS:
    !cd $data_dir && wget http://files.grouplens.org/datasets/movielens/ml-latest-small.zip
    !cd $data_dir && unzip -o ml-latest-small.zip
    dataset_dir = data_dir + "/ml-latest-small/"
else:
    !cd $data_dir && wget http://files.grouplens.org/datasets/movielens/ml-25m.zip
    !cd $data_dir && unzip -o ml-25m.zip
    dataset_dir = data_dir + "/ml-25m/"--2020-11-18 10:38:57--  http://files.grouplens.org/datasets/movielens/ml-latest-small.zip
Resolving files.grouplens.org (files.grouplens.org)... 128.101.65.152
Connecting to files.grouplens.org (files.grouplens.org)|128.101.65.152|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 978202 (955K) [application/zip]
Saving to: ‘ml-latest-small.zip’

ml-latest-small.zip 100%[===================>] 955.28K  4.14MB/s    in 0.2s

2020-11-18 10:38:58 (4.14 MB/s) - ‘ml-latest-small.zip’ saved [978202/978202]

Archive:  ml-latest-small.zip
   creating: ml-latest-small/
  inflating: ml-latest-small/links.csv
  inflating: ml-latest-small/tags.csv
  inflating: ml-latest-small/ratings.csv
  inflating: ml-latest-small/README.txt
  inflating: ml-latest-small/movies.csv

Take a look at the data files you have downloaded._____no_output_____

!ls $dataset_dirlinks.csv  movies.csv  ratings.csv  README.txt	tags.csv

At present not much is known except that we have a few CSVs and a readme. Next we will output the readme to learn more!_____no_output_____

!pygmentize $dataset_dir/README.txtSummary
=======

This dataset (ml-latest-small) describes 5-star rating and free-text tagging activity from [MovieLens](http://movielens.org), a movie recommendation service. It contains 100836 ratings and 3683 tag applications across 9742 movies. These data were created by 610 users between March 29, 1996 and September 24, 2018. This dataset was generated on September 26, 2018.

Users were selected at random for inclusion. All selected users had rated at least 20 movies. No demographic information is included. Each user is represented by an id, and no other information is provided.

The data are contained in the files `links.csv`, `movies.csv`, `ratings.csv` and `tags.csv`. More details about the contents and use of all these files follows.

This is a *development* dataset. As such, it may change over time and is not an appropriate dataset for shared research results. See available *benchmark* datasets if that is your intent.

This and other GroupLens data sets are publicly available for download at <http://grouplens.org/datasets/>.

Usage License
=============

Neither the University of Minnesota nor any of the researchers involved can guarantee the correctness of the data, its suitability for any particular purpose, or the validity of results based on the use of the data set. The data set may be used for any research purposes under the following conditions:

* The user may not state or imply any endorsement from the University of Minnesota or the GroupLens Research Group.
* The user must acknowledge the use of the data set in publications resulting from the use of the data set (see below for citation information).
* The user may redistribute the data set, including transformations, so long as it is distributed under these same license conditions.
* The user may not use this information for any commercial or revenue-bearing purposes without first obtaining permission from a faculty member of the GroupLens Research Project at the University of Minnesota.
* The executable software scripts are provided "as is" without warranty of any kind, either expressed or implied, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. The entire risk as to the quality and performance of them is with you. Should the program prove defective, you assume the cost of all necessary servicing, repair or correction.

In no event shall the University of Minnesota, its affiliates or employees be liable to you for any damages arising out of the use or inability to use these programs (including but not limited to loss of data or data being rendered inaccurate).

If you have any further questions or comments, please email <grouplens-info@umn.edu>

Citation
========

To acknowledge use of the dataset in publications, please cite the following paper:

> F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Transactions on Interactive Intelligent Systems (TiiS) 5, 4: 19:1–19:19. <https://doi.org/10.1145/2827872>

Further Information About GroupLens
===================================

GroupLens is a research group in the Department of Computer Science and Engineering at the University of Minnesota. Since its inception in 1992, GroupLens's research projects have explored a variety of fields including:

* recommender systems
* online communities
* mobile and ubiquitious technologies
* digital libraries
* local geographic information systems

GroupLens Research operates a movie recommender based on collaborative filtering, MovieLens, which is the source of these data. We encourage you to visit <http://movielens.org> to try it out! If you have exciting ideas for experimental work to conduct on MovieLens, send us an email at <grouplens-info@cs.umn.edu> - we are always interested in working with external collaborators.

Content and Use of Files
========================

Formatting and Encoding
-----------------------

The dataset files are written as [comma-separated values](http://en.wikipedia.org/wiki/Comma-separated_values) files with a single header row. Columns that contain commas (`,`) are escaped using double-quotes (`"`). These files are encoded as UTF-8. If accented characters in movie titles or tag values (e.g. Misérables, Les (1995)) display incorrectly, make sure that any program reading the data, such as a text editor, terminal, or script, is configured for UTF-8.

User Ids
--------

MovieLens users were selected at random for inclusion. Their ids have been anonymized. User ids are consistent between `ratings.csv` and `tags.csv` (i.e., the same id refers to the same user across the two files).

Movie Ids
---------

Only movies with at least one rating or tag are included in the dataset. These movie ids are consistent with those used on the MovieLens web site (e.g., id `1` corresponds to the URL <https://movielens.org/movies/1>). Movie ids are consistent between `ratings.csv`, `tags.csv`, `movies.csv`, and `links.csv` (i.e., the same id refers to the same movie across these four data files).

Ratings Data File Structure (ratings.csv)
-----------------------------------------

All ratings are contained in the file `ratings.csv`. Each line of this file after the header row represents one rating of one movie by one user, and has the following format:

    userId,movieId,rating,timestamp

The lines within this file are ordered first by userId, then, within user, by movieId.

Ratings are made on a 5-star scale, with half-star increments (0.5 stars - 5.0 stars).

Timestamps represent seconds since midnight Coordinated Universal Time (UTC) of January 1, 1970.

Tags Data File Structure (tags.csv)
-----------------------------------

All tags are contained in the file `tags.csv`. Each line of this file after the header row represents one tag applied to one movie by one user, and has the following format:

    userId,movieId,tag,timestamp

The lines within this file are ordered first by userId, then, within user, by movieId.

Tags are user-generated metadata about movies. Each tag is typically a single word or short phrase. The meaning, value, and purpose of a particular tag is determined by each user.

Timestamps represent seconds since midnight Coordinated Universal Time (UTC) of January 1, 1970.

Movies Data File Structure (movies.csv)
---------------------------------------

Movie information is contained in the file `movies.csv`. Each line of this file after the header row represents one movie, and has the following format:

    movieId,title,genres

Movie titles are entered manually or imported from <https://www.themoviedb.org/>, and include the year of release in parentheses. Errors and inconsistencies may exist in these titles.

Genres are a pipe-separated list, and are selected from the following:

* Action
* Adventure
* Animation
* Children's
* Comedy
* Crime
* Documentary
* Drama
* Fantasy
* Film-Noir
* Horror
* Musical
* Mystery
* Romance
* Sci-Fi
* Thriller
* War
* Western
* (no genres listed)

Links Data File Structure (links.csv)
---------------------------------------

Identifiers that can be used to link to other sources of movie data are contained in the file `links.csv`. Each line of this file after the header row represents one movie, and has the following format:

    movieId,imdbId,tmdbId

movieId is an identifier for movies used by <https://movielens.org>. E.g., the movie Toy Story has the link <https://movielens.org/movies/1>.

imdbId is an identifier for movies used by <http://www.imdb.com>. E.g., the movie Toy Story has the link <http://www.imdb.com/title/tt0114709/>.

tmdbId is an identifier for movies used by <https://www.themoviedb.org>. E.g., the movie Toy Story has the link <https://www.themoviedb.org/movie/862>.

Use of the resources listed above is subject to the terms of each provider.

Cross-Validation
----------------

Prior versions of the MovieLens dataset included either pre-computed cross-folds or scripts to perform this computation. We no longer bundle either of these features with the dataset, since most modern toolkits provide this as a built-in feature. If you wish to learn about standard approaches to cross-fold computation in the context of recommender systems evaluation, see [LensKit](http://lenskit.org) for tools, documentation, and open-source code examples.

From the README, we see there is a file `ratings.csv` that should work as a proxy for our interactions data, after all rating a film definitely is a form of interacting with it. The dataset also has some genre information as some movie genome data. In this POC we will focus on the interactions and the genre data.
_____no_output_____## Prepare your data
[Back to top](#top)

The next thing to be done is to load the data and confirm the data is in a good state, then save it to a CSV where it is ready to be used with Amazon Personalize.

To get started, import a collection of Python libraries commonly used in data science._____no_output_____

import time
from time import sleep
import json
from datetime import datetime
import boto3
import pandas as pd_____no_output_____

Next,open the data file and take a look at the first several rows._____no_output_____

original_data = pd.read_csv(dataset_dir + '/ratings.csv')
original_data.head(5)_____no_output_____original_data.shape_____no_output_____original_data.describe()_____no_output_____

This shows that we have a good range of values for `userId` and `movieId`. Next, it is always a good idea to confirm the data format._____no_output_____

original_data.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 100836 entries, 0 to 100835
Data columns (total 4 columns):
 #   Column     Non-Null Count   Dtype
---  ------     --------------   -----
 0   userId     100836 non-null  int64
 1   movieId    100836 non-null  int64
 2   rating     100836 non-null  float64
 3   timestamp  100836 non-null  int64
dtypes: float64(1), int64(3)
memory usage: 3.1 MB
original_data.isnull().any()_____no_output_____

From this, you can see that there are a total of (25,000,095 for full 100836 for small) entries in the dataset, with 4 columns, and each cell stored as int64 format, with the exception of the rating whihch is a float64.

The int64 format is clearly suitable for `userId` and `movieId`. However, we need to diver deeper to understand the timestamps in the data. To use Amazon Personalize, you need to save timestamps in [Unix Epoch](https://en.wikipedia.org/wiki/Unix_time) format.

Currently, the timestamp values are not human-readable. So let's grab an arbitrary timestamp value and figure out how to interpret it._____no_output_____Do a quick sanity check on the transformed dataset by picking an arbitrary timestamp and transforming it to a human-readable format._____no_output_____

arb_time_stamp = original_data.iloc[50]['timestamp']
print(arb_time_stamp)
print(datetime.utcfromtimestamp(arb_time_stamp).strftime('%Y-%m-%d %H:%M:%S'))964982681.0
2000-07-30 18:44:41

This date makes sense as a timestamp, so we can continue formatting the rest of the data. Remember, the data we need is user-item-interaction data, which is `userId`, `movieId`, and `timestamp` in this case. Our dataset has an additional column, `rating`, which can be dropped from the dataset after we have leveraged it to focus on positive interactions._____no_output_____Since this is an explicit feedback movie rating dataset, it includes movies rated from 1 to 5, we want to include only moves that weree "liked" by the users, and simulate a implicit dataset that is more like what data would be gathered by a VOD platform. For that so we will filter out all interactions under 2 out of 5, and create an EVENT_Type of "click" and an EVENT_Type of "watch". We will then assign all movies rated 2 and above as "click" and movies above 4 and above as "click" and "watch".

Note this is to correspond with the events we are modeling, for a real data set you would actually model based on implicit feedback like clicks, watches and/or explicit feedback like ratings, likes etc._____no_output_____

watched_df = original_data.copy()
watched_df = watched_df[watched_df['rating'] > 3]
watched_df = watched_df[['userId', 'movieId', 'timestamp']]
watched_df['EVENT_TYPE']='watch'
watched_df.head()_____no_output_____clicked_df = original_data.copy()
clicked_df = clicked_df[clicked_df['rating'] > 1]
clicked_df = clicked_df[['userId', 'movieId', 'timestamp']]
clicked_df['EVENT_TYPE']='click'
clicked_df.head()_____no_output_____interactions_df = clicked_df.copy()
interactions_df = interactions_df.append(watched_df)
interactions_df.sort_values("timestamp", axis = 0, ascending = True,
                 inplace = True, na_position ='last') _____no_output_____interactions_df.info()<class 'pandas.core.frame.DataFrame'>
Int64Index: 158371 entries, 66679 to 81092
Data columns (total 4 columns):
 #   Column      Non-Null Count   Dtype
---  ------      --------------   -----
 0   userId      158371 non-null  int64
 1   movieId     158371 non-null  int64
 2   timestamp   158371 non-null  int64
 3   EVENT_TYPE  158371 non-null  object
dtypes: int64(3), object(1)
memory usage: 6.0+ MB

lets look at what the new dataset looks like _____no_output_____

interactions_df.describe()_____no_output_____

After manipulating the data, always confirm if the data format has changed._____no_output_____

interactions_df.dtypes_____no_output_____

 Amazon Personalize has default column names for users, items, and timestamp. These default column names are `USER_ID`, `ITEM_ID`, AND `TIMESTAMP`. So the final modification to the dataset is to replace the existing column headers with the default headers._____no_output_____

interactions_df.rename(columns = {'userId':'USER_ID', 'movieId':'ITEM_ID',
                              'timestamp':'TIMESTAMP'}, inplace = True) _____no_output_____

That's it! At this point the data is ready to go, and we just need to save it as a CSV file._____no_output_____

interactions_filename = "interactions.csv"
interactions_df.to_csv((data_dir+"/"+interactions_filename), index=False, float_format='%.0f')_____no_output_____

## Create dataset groups and the interactions dataset
[Back to top](#top)

The highest level of isolation and abstraction with Amazon Personalize is a *dataset group*. Information stored within one of these dataset groups has no impact on any other dataset group or models created from one - they are completely isolated. This allows you to run many experiments and is part of how we keep your models private and fully trained only on your data.

Before importing the data prepared earlier, there needs to be a dataset group and a dataset added to it that handles the interactions.

Dataset groups can house the following types of information:

* User-item-interactions
* Event streams (real-time interactions)
* User metadata
* Item metadata

Before we create the dataset group and the dataset for our interaction data, let's validate that your environment can communicate successfully with Amazon Personalize._____no_output_____

# Configure the SDK to Personalize:
personalize = boto3.client('personalize')
personalize_runtime = boto3.client('personalize-runtime')_____no_output_____

### Create the dataset group

The following cell will create a new dataset group with the name `personalize-poc-lastfm`._____no_output_____

create_dataset_group_response = personalize.create_dataset_group(
    name = "personalize-poc-movielens25m"
)

dataset_group_arn = create_dataset_group_response['datasetGroupArn']
print(json.dumps(create_dataset_group_response, indent=2)){
  "datasetGroupArn": "arn:aws:personalize:us-east-1:001723200916:dataset-group/personalize-poc-movielens25m",
  "ResponseMetadata": {
    "RequestId": "850bad28-40c4-4181-864b-3bf367addd5f",
    "HTTPStatusCode": 200,
    "HTTPHeaders": {
      "content-type": "application/x-amz-json-1.1",
      "date": "Wed, 18 Nov 2020 10:39:01 GMT",
      "x-amzn-requestid": "850bad28-40c4-4181-864b-3bf367addd5f",
      "content-length": "107",
      "connection": "keep-alive"
    },
    "RetryAttempts": 0
  }
}

Before we can use the dataset group, it must be active. This can take a minute or two. Execute the cell below and wait for it to show the ACTIVE status. It checks the status of the dataset group every second, up to a maximum of 3 hours._____no_output_____

max_time = time.time() + 3*60*60 # 3 hours
while time.time() < max_time:
    describe_dataset_group_response = personalize.describe_dataset_group(
        datasetGroupArn = dataset_group_arn
    )
    status = describe_dataset_group_response["datasetGroup"]["status"]
    print("DatasetGroup: {}".format(status))

    if status == "ACTIVE" or status == "CREATE FAILED":
        break

    time.sleep(60)DatasetGroup: CREATE PENDING
DatasetGroup: ACTIVE

Now that you have a dataset group, you can create a dataset for the interaction data._____no_output_____### Create the dataset

First, define a schema to tell Amazon Personalize what type of dataset you are uploading. There are several reserved and mandatory keywords required in the schema, based on the type of dataset. More detailed information can be found in the [documentation](https://docs.aws.amazon.com/personalize/latest/dg/how-it-works-dataset-schema.html).

Here, you will create a schema for interactions data, which needs the `USER_ID`, `ITEM_ID`, and `TIMESTAMP` fields. These must be defined in the same order in the schema as they appear in the dataset._____no_output_____

interactions_schema = schema = {
    "type": "record",
    "name": "Interactions",
    "namespace": "com.amazonaws.personalize.schema",
    "fields": [
        {
            "name": "USER_ID",
            "type": "string"
        },
        {
            "name": "ITEM_ID",
            "type": "string"
        },
        {
            "name": "EVENT_TYPE",
            "type": "string"
        },
        {
            "name": "TIMESTAMP",
            "type": "long"
        }
    ],
    "version": "1.0"
}

create_schema_response = personalize.create_schema(
    name = "personalize-poc-movielens-interactions",
    schema = json.dumps(interactions_schema)
)

schema_arn = create_schema_response['schemaArn']
print(json.dumps(create_schema_response, indent=2)){
  "schemaArn": "arn:aws:personalize:us-east-1:001723200916:schema/personalize-poc-movielens-interactions",
  "ResponseMetadata": {
    "RequestId": "36bca0fa-a24a-4841-9674-81fd286806a7",
    "HTTPStatusCode": 200,
    "HTTPHeaders": {
      "content-type": "application/x-amz-json-1.1",
      "date": "Wed, 18 Nov 2020 10:40:01 GMT",
      "x-amzn-requestid": "36bca0fa-a24a-4841-9674-81fd286806a7",
      "content-length": "104",
      "connection": "keep-alive"
    },
    "RetryAttempts": 0
  }
}
# personalize.list_schemas()_____no_output_____# personalize.delete_schema(
#     schemaArn='arn:aws:personalize:us-east-1:835319576252:schema/personalize-demo-schema'
# #     name = "personalize-poc-movielens-interactions",
# # #    schema = json.dumps(interactions_schema)
# )
_____no_output_____

With a schema created, you can create a dataset within the dataset group. Note, this does not load the data yet. This will happen a few steps later._____no_output_____

dataset_type = "INTERACTIONS"
create_dataset_response = personalize.create_dataset(
    name = "personalize-poc-movielens-ints",
    datasetType = dataset_type,
    datasetGroupArn = dataset_group_arn,
    schemaArn = schema_arn
)

interactions_dataset_arn = create_dataset_response['datasetArn']
print(json.dumps(create_dataset_response, indent=2)){
  "datasetArn": "arn:aws:personalize:us-east-1:001723200916:dataset/personalize-poc-movielens25m/INTERACTIONS",
  "ResponseMetadata": {
    "RequestId": "bdfd9830-5ad0-4031-8c6a-71dcb8f2573b",
    "HTTPStatusCode": 200,
    "HTTPHeaders": {
      "content-type": "application/x-amz-json-1.1",
      "date": "Wed, 18 Nov 2020 10:40:02 GMT",
      "x-amzn-requestid": "bdfd9830-5ad0-4031-8c6a-71dcb8f2573b",
      "content-length": "109",
      "connection": "keep-alive"
    },
    "RetryAttempts": 0
  }
}

## Configure an S3 bucket and an IAM  role
[Back to top](#top)

So far, we have downloaded, manipulated, and saved the data onto the Amazon EBS instance attached to instance running this Jupyter notebook. However, Amazon Personalize will need an S3 bucket to act as the source of your data, as well as IAM roles for accessing that bucket. Let's set all of that up.

Use the metadata stored on the instance underlying this Amazon SageMaker notebook, to determine the region it is operating in. If you are using a Jupyter notebook outside of Amazon SageMaker, simply define the region as a string below. The Amazon S3 bucket needs to be in the same region as the Amazon Personalize resources we have been creating so far._____no_output_____

with open('/opt/ml/metadata/resource-metadata.json') as notebook_info:
    data = json.load(notebook_info)
    resource_arn = data['ResourceArn']
    region = resource_arn.split(':')[3]
print(region)us-east-1

Amazon S3 bucket names are globally unique. To create a unique bucket name, the code below will append the string `personalizepocvod` to your AWS account number. Then it creates a bucket with this name in the region discovered in the previous cell._____no_output_____

s3 = boto3.client('s3')
account_id = boto3.client('sts').get_caller_identity().get('Account')
bucket_name = account_id + "personalizepocvod"
print(bucket_name)
s3.create_bucket(Bucket=bucket_name)001723200916personalizepocvod

### Upload data to S3

Now that your Amazon S3 bucket has been created, upload the CSV file of our user-item-interaction data. _____no_output_____

interactions_file_path = data_dir + "/" + interactions_filename
boto3.Session().resource('s3').Bucket(bucket_name).Object(interactions_filename).upload_file(interactions_file_path)
interactions_s3DataPath = "s3://"+bucket_name+"/"+interactions_filename_____no_output_____

### Set the S3 bucket policy
Amazon Personalize needs to be able to read the contents of your S3 bucket. So add a bucket policy which allows that._____no_output_____

policy = {
    "Version": "2012-10-17",
    "Id": "PersonalizeS3BucketAccessPolicy",
    "Statement": [
        {
            "Sid": "PersonalizeS3BucketAccessPolicy",
            "Effect": "Allow",
            "Principal": {
                "Service": "personalize.amazonaws.com"
            },
            "Action": [
                "s3:*Object",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::{}".format(bucket_name),
                "arn:aws:s3:::{}/*".format(bucket_name)
            ]
        }
    ]
}

s3.put_bucket_policy(Bucket=bucket_name, Policy=json.dumps(policy))_____no_output_____

### Create an IAM role

Amazon Personalize needs the ability to assume roles in AWS in order to have the permissions to execute certain tasks. Let's create an IAM role and attach the required policies to it. The code below attaches very permissive policies; please use more restrictive policies for any production application._____no_output_____

iam = boto3.client("iam")

role_name = "PersonalizeRolePOC"
assume_role_policy_document = {
    "Version": "2012-10-17",
    "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Service": "personalize.amazonaws.com"
          },
          "Action": "sts:AssumeRole"
        }
    ]
}

create_role_response = iam.create_role(
    RoleName = role_name,
    AssumeRolePolicyDocument = json.dumps(assume_role_policy_document)
)

# AmazonPersonalizeFullAccess provides access to any S3 bucket with a name that includes "personalize" or "Personalize"
# if you would like to use a bucket with a different name, please consider creating and attaching a new policy
# that provides read access to your bucket or attaching the AmazonS3ReadOnlyAccess policy to the role
policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonPersonalizeFullAccess"
iam.attach_role_policy(
    RoleName = role_name,
    PolicyArn = policy_arn
)

# Now add S3 support
iam.attach_role_policy(
    PolicyArn='arn:aws:iam::aws:policy/AmazonS3FullAccess',
    RoleName=role_name
)
time.sleep(60) # wait for a minute to allow IAM role policy attachment to propagate

role_arn = create_role_response["Role"]["Arn"]
print(role_arn)arn:aws:iam::001723200916:role/PersonalizeRolePOC

## Import the interactions data
[Back to top](#top)

Earlier you created the dataset group and dataset to house your information, so now you will execute an import job that will load the data from the S3 bucket into the Amazon Personalize dataset. _____no_output_____

create_dataset_import_job_response = personalize.create_dataset_import_job(
    jobName = "personalize-poc-import1",
    datasetArn = interactions_dataset_arn,
    dataSource = {
        "dataLocation": "s3://{}/{}".format(bucket_name, interactions_filename)
    },
    roleArn = role_arn
)

dataset_import_job_arn = create_dataset_import_job_response['datasetImportJobArn']
print(json.dumps(create_dataset_import_job_response, indent=2)){
  "datasetImportJobArn": "arn:aws:personalize:us-east-1:001723200916:dataset-import-job/personalize-poc-import1",
  "ResponseMetadata": {
    "RequestId": "90524d37-7595-41ca-8baa-f0d9489370ec",
    "HTTPStatusCode": 200,
    "HTTPHeaders": {
      "content-type": "application/x-amz-json-1.1",
      "date": "Wed, 18 Nov 2020 10:41:04 GMT",
      "x-amzn-requestid": "90524d37-7595-41ca-8baa-f0d9489370ec",
      "content-length": "111",
      "connection": "keep-alive"
    },
    "RetryAttempts": 0
  }
}

Before we can use the dataset, the import job must be active. Execute the cell below and wait for it to show the ACTIVE status. It checks the status of the import job every second, up to a maximum of 6 hours.

Importing the data can take some time, depending on the size of the dataset. In this workshop, the data import job should take around 15 minutes._____no_output_____

%%time

max_time = time.time() + 6*60*60 # 6 hours
while time.time() < max_time:
    describe_dataset_import_job_response = personalize.describe_dataset_import_job(
        datasetImportJobArn = dataset_import_job_arn
    )
    status = describe_dataset_import_job_response["datasetImportJob"]['status']
    print("DatasetImportJob: {}".format(status))

    if status == "ACTIVE" or status == "CREATE FAILED":
        break

    time.sleep(60)DatasetImportJob: CREATE PENDING
DatasetImportJob: CREATE IN_PROGRESS
DatasetImportJob: CREATE IN_PROGRESS
DatasetImportJob: CREATE IN_PROGRESS
DatasetImportJob: CREATE IN_PROGRESS
DatasetImportJob: CREATE IN_PROGRESS
DatasetImportJob: CREATE IN_PROGRESS
DatasetImportJob: CREATE IN_PROGRESS

When the dataset import is active, you are ready to start building models with SIMS, Personalized-Ranking, Popularity-Count, and User Personalization. This process will continue in other notebooks. Run the cell below before moving on to store a few values for usage in the next notebooks._____no_output_____

%store USE_FULL_MOVIELENS
%store dataset_dir
%store interactions_dataset_arn
%store dataset_group_arn
%store bucket_name
%store role_arn
%store role_name
%store data_dir
%store region_____no_output_____

# Release Resources_____no_output_____

%%javascript
Jupyter.notebook.save_checkpoint();
Jupyter.notebook.session.delete();_____no_output_____


# Notebook from hilmireza09/DQLab
Path: python/Project/3_Project Machine Learning with Python Building Recommender System.ipynb

#import library yang dibutuhkan
import pandas as pd
import numpy as np

#lakukan pembacaan dataset
movie_df = pd.read_csv('https://storage.googleapis.com/dqlab-dataset/title.basics.tsv', sep='\t') #untuk menyimpan title_basics.tsv
rating_df = pd.read_csv('https://storage.googleapis.com/dqlab-dataset/title.ratings.tsv', sep='\t') #untuk menyimpan title.ratings.tsv_____no_output_____

## ---------------------------------------------------------------CHAPTER 3_____no_output_____

print(movie_df.head())
movie_df.head()      tconst  titleType                                      primaryTitle  \
0  tt0221078      short                         Circle Dance, Ute Indians
1  tt8862466  tvEpisode  ¡El #TeamOsos va con todo al "Reality del amor"!
2  tt7157720  tvEpisode                                     Episode #3.41
3  tt2974998  tvEpisode                         Episode dated 16 May 1987
4  tt2903620  tvEpisode                  Frances Bavier: Aunt Bee Retires

                                      originalTitle  isAdult startYear  \
0                         Circle Dance, Ute Indians        0      1898
1  ¡El #TeamOsos va con todo al "Reality del amor"!        0      2018
2                                     Episode #3.41        0      2016
3                         Episode dated 16 May 1987        0      1987
4                  Frances Bavier: Aunt Bee Retires        0      1973

  endYear runtimeMinutes             genres
0      \N             \N  Documentary,Short
1      \N             \N       Comedy,Drama
2      \N             29   Comedy,Game-Show
3      \N             \N               News
4      \N             \N        Documentary
print(movie_df.info())<class 'pandas.core.frame.DataFrame'>
RangeIndex: 9025 entries, 0 to 9024
Data columns (total 9 columns):
 #   Column          Non-Null Count  Dtype
---  ------          --------------  -----
 0   tconst          9025 non-null   object
 1   titleType       9025 non-null   object
 2   primaryTitle    9011 non-null   object
 3   originalTitle   9011 non-null   object
 4   isAdult         9025 non-null   int64
 5   startYear       9025 non-null   object
 6   endYear         9025 non-null   object
 7   runtimeMinutes  9025 non-null   object
 8   genres          9014 non-null   object
dtypes: int64(1), object(8)
memory usage: 634.7+ KB
None
print(movie_df.isnull().sum())tconst             0
titleType          0
primaryTitle      14
originalTitle     14
isAdult            0
startYear          0
endYear            0
runtimeMinutes     0
genres            11
dtype: int64
print(movie_df.loc[(movie_df['primaryTitle'].isnull()) |
                   (movie_df['originalTitle'].isnull())])          tconst  titleType primaryTitle originalTitle  isAdult startYear  \
9000  tt10790040  tvEpisode          NaN           NaN        0      2019
9001  tt10891902  tvEpisode          NaN           NaN        0      2020
9002  tt11737860  tvEpisode          NaN           NaN        0      2020
9003  tt11737862  tvEpisode          NaN           NaN        0      2020
9004  tt11737866  tvEpisode          NaN           NaN        0      2020
9005  tt11737872  tvEpisode          NaN           NaN        0      2020
9006  tt11737874  tvEpisode          NaN           NaN        0      2020
9007   tt1971246  tvEpisode          NaN           NaN        0      2011
9008   tt2067043  tvEpisode          NaN           NaN        0      1965
9009   tt4404732  tvEpisode          NaN           NaN        0      2015
9010   tt5773048  tvEpisode          NaN           NaN        0      2015
9011   tt8473688  tvEpisode          NaN           NaN        0      1987
9012   tt8541336  tvEpisode          NaN           NaN        0      2018
9013   tt9824302  tvEpisode          NaN           NaN        0      2016

     endYear runtimeMinutes                genres
9000      \N             \N                    \N
9001      \N             \N                 Crime
9002      \N             \N  Comedy,Drama,Romance
9003      \N             \N  Comedy,Drama,Romance
9004      \N             \N  Comedy,Drama,Romance
9005      \N             \N                    \N
9006      \N             \N  Comedy,Drama,Romance
9007      \N             \N             Biography
9008      \N             \N                 Music
9009      \N             \N                Comedy
9010      \N             \N             Talk-Show
9011      \N             \N                 Drama
9012      \N             \N    Reality-TV,Romance
9013      \N             \N           Documentary
#mengupdate movie_df dengan membuang data-data bernilai NULL
movie_df = movie_df.loc[(movie_df['primaryTitle'].notnull()) &
                       (movie_df['originalTitle'].notnull())]

#menampilkan jumlah data setelah data dengan nilai NULL dibuang
print(len(movie_df))9011
print(movie_df.loc[movie_df['genres'].isnull()])          tconst  titleType  \
9014  tt10233364  tvEpisode
9015  tt10925142  tvEpisode
9016  tt10970874  tvEpisode
9017  tt11670006  tvEpisode
9018  tt11868642  tvEpisode
9019   tt2347742  tvEpisode
9020   tt3984412  tvEpisode
9021   tt8740950  tvEpisode
9022   tt9822816  tvEpisode
9023   tt9900062  tvEpisode
9024   tt9909210  tvEpisode

                                           primaryTitle originalTitle  \
9014  Rolling in the Deep Dish\tRolling in the Deep ...             0
9015  The IMDb Show on Location: Star Wars Galaxy's ...             0
9016  Die Bauhaus-Stadt Tel Aviv - Vorbild für die M...             0
9017  ...ein angenehmer Unbequemer...\t...ein angene...             0
9018  GGN Heavyweight Championship Lungs With Mike T...             0
9019  No sufras por la alergia esta primavera\tNo su...             0
9020  I'm Not Going to Come Last, I'm Just Going to ...             0
9021  Weight Loss Resolution Restart - Ins & Outs of...             0
9022  Zwischen Vertuschung und Aufklärung - Missbrau...             0
9023  The Direction of Yuu's Love: Hings Aren't Goin...             0
9024  Politik und/oder Moral - Wie weit geht das Ver...             0

      isAdult startYear endYear          runtimeMinutes genres
9014     2019        \N      \N              Reality-TV    NaN
9015     2019        \N      \N               Talk-Show    NaN
9016     2019        \N      \N                      \N    NaN
9017     1981        \N      \N             Documentary    NaN
9018     2020        \N      \N               Talk-Show    NaN
9019     2004        \N      \N                      \N    NaN
9020     2014        \N      \N              Reality-TV    NaN
9021     2015        \N      \N              Reality-TV    NaN
9022     2019        \N      \N                      \N    NaN
9023     1994        \N      \N  Animation,Comedy,Drama    NaN
9024     2005        \N      \N                      \N    NaN
#mengupdate movie_df dengan membuang data-data bernilai NULL
movie_df = movie_df.loc[movie_df['genres'].notnull()]

#menampilkan jumlah data setelah data dengan nilai NULL dibuang
print(len(movie_df))9014
#mengubah nilai '\\N' pada startYear menjadi np.nan dan cast kolomnya menjadi float64
movie_df['startYear'] = movie_df['startYear'].replace('\\N',np.NaN)
movie_df['startYear'] = movie_df['startYear'].astype(np.float64)
print(movie_df['startYear'].unique()[:5])

#mengubah nilai '\\N' pada endYear menjadi np.nan dan cast kolomnya menjadi float64
movie_df['endYear'] = movie_df['endYear'].replace('\\N',np.NaN)
movie_df['endYear'] = movie_df['endYear'].astype(np.float64)
print(movie_df['endYear'].unique()[:5])

#mengubah nilai '\\N' pada runtimeMinutes menjadi np.nan dan cast kolomnya menjadi float64
movie_df['runtimeMinutes'] = movie_df['runtimeMinutes'].replace('\\N',np.NaN)
movie_df['runtimeMinutes'] = movie_df['runtimeMinutes'].astype(np.float64)
print(movie_df['runtimeMinutes'].unique()[:5])[1898. 2018. 2016. 1987. 1973.]
[  nan 2005. 1955. 2006. 1999.]
[nan 29.  7. 23. 85.]
#mengubah nilai '\\N' pada startYear menjadi np.nan dan cast kolomnya menjadi float64
movie_df['startYear'] = movie_df['startYear'].replace('\\N',np.NaN)
movie_df['startYear'] = movie_df['startYear'].astype('float64')
print(movie_df['startYear'].unique()[:5])

#mengubah nilai '\\N' pada endYear menjadi np.nan dan cast kolomnya menjadi float64
movie_df['endYear'] = movie_df['endYear'].replace('\\N',np.NaN)
movie_df['endYear'] = movie_df['endYear'].astype('float64')
print(movie_df['endYear'].unique()[:5])

#mengubah nilai '\\N' pada runtimeMinutes menjadi np.nan dan cast kolomnya menjadi float64
movie_df['runtimeMinutes'] = movie_df['runtimeMinutes'].replace('\\N',np.NaN)
movie_df['runtimeMinutes'] = movie_df['runtimeMinutes'].astype('float64')
print(movie_df['runtimeMinutes'].unique()[:5])[1898. 2018. 2016. 1987. 1973.]
[  nan 2005. 1955. 2006. 1999.]
[nan 29.  7. 23. 85.]
def transform_to_list(x):
    if ',' in x:
    #ubah menjadi list apabila ada data pada kolom genre
        return x.split(',')
    else:
    #jika tidak ada data, ubah menjadi list kosong
        return []

movie_df['genres'] = movie_df['genres'].apply(lambda x: transform_to_list(x))_____no_output_____

## ------------------------------------------------------CHAPTER 4_____no_output_____

print(rating_df.head())      tconst  averageRating  numVotes
0  tt0000001            5.6      1608
1  tt0000002            6.0       197
2  tt0000003            6.5      1285
3  tt0000004            6.1       121
4  tt0000005            6.1      2050
print(rating_df.info())<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1030009 entries, 0 to 1030008
Data columns (total 3 columns):
 #   Column         Non-Null Count    Dtype
---  ------         --------------    -----
 0   tconst         1030009 non-null  object
 1   averageRating  1030009 non-null  float64
 2   numVotes       1030009 non-null  int64
dtypes: float64(1), int64(1), object(1)
memory usage: 23.6+ MB
None

## ----------------------------------------------------------CHAPTER 5_____no_output_____

#Lakukan join pada kedua table
movie_rating_df = pd.merge( movie_df, rating_df, on='tconst', how='inner')

#Tampilkan 5 data teratas
print(movie_rating_df.head())

#Tampilkan tipe data dari tiap kolom
print(movie_rating_df.info())      tconst  titleType              primaryTitle             originalTitle  \
0  tt0043745      short                 Lion Down                 Lion Down
1  tt0167491      video         Wicked Covergirls         Wicked Covergirls
2  tt6574096  tvEpisode      Shadow Play - Part 2      Shadow Play - Part 2
3  tt6941700  tvEpisode              RuPaul Roast              RuPaul Roast
4  tt7305674      video  UCLA Track & Field Promo  UCLA Track & Field Promo

   isAdult startYear endYear runtimeMinutes                      genres  \
0        0      1951      \N              7     Animation,Comedy,Family
1        1      1998      \N             85                       Adult
2        0      2017      \N             22  Adventure,Animation,Comedy
3        0      2017      \N             \N                  Reality-TV
4        0      2017      \N             \N                 Short,Sport

   averageRating  numVotes
0            7.1       459
1            5.7         7
2            8.5       240
3            8.0        11
4            9.7         7
<class 'pandas.core.frame.DataFrame'>
Int64Index: 1378 entries, 0 to 1377
Data columns (total 11 columns):
 #   Column          Non-Null Count  Dtype
---  ------          --------------  -----
 0   tconst          1378 non-null   object
 1   titleType       1378 non-null   object
 2   primaryTitle    1378 non-null   object
 3   originalTitle   1378 non-null   object
 4   isAdult         1378 non-null   int64
 5   startYear       1378 non-null   object
 6   endYear         1378 non-null   object
 7   runtimeMinutes  1378 non-null   object
 8   genres          1376 non-null   object
 9   averageRating   1378 non-null   float64
 10  numVotes        1378 non-null   int64
dtypes: float64(1), int64(2), object(8)
memory usage: 129.2+ KB
None
# pd.to_numeric(movie_rating_df['starYear'], errors='coerce').astype('float64')
# movie_rating_df['startYear'].astype('float64')
movie_rating_df[1]_____no_output_____movie_rating_df = movie_rating_df.loc[(movie_rating_df['startYear'].notnull()) &
                                      (movie_rating_df['runtimeMinutes'].notnull())]

#Untuk memastikan bahwa sudah tidak ada lagi nilai NULL
print(movie_rating_df.info())<class 'pandas.core.frame.DataFrame'>
Int64Index: 1376 entries, 0 to 1375
Data columns (total 11 columns):
 #   Column          Non-Null Count  Dtype
---  ------          --------------  -----
 0   tconst          1376 non-null   object
 1   titleType       1376 non-null   object
 2   primaryTitle    1376 non-null   object
 3   originalTitle   1376 non-null   object
 4   isAdult         1376 non-null   int64
 5   startYear       1376 non-null   object
 6   endYear         1376 non-null   object
 7   runtimeMinutes  1376 non-null   object
 8   genres          1376 non-null   object
 9   averageRating   1376 non-null   float64
 10  numVotes        1376 non-null   int64
dtypes: float64(1), int64(2), object(8)
memory usage: 129.0+ KB
None
movie_rating_df = movie_rating_df.dropna(subset=['startYear','runtimeMinutes'])

#Untuk memastikan bahwa sudah tidak ada lagi nilai NULL
print(movie_rating_df.info())<class 'pandas.core.frame.DataFrame'>
Int64Index: 1376 entries, 0 to 1375
Data columns (total 11 columns):
 #   Column          Non-Null Count  Dtype
---  ------          --------------  -----
 0   tconst          1376 non-null   object
 1   titleType       1376 non-null   object
 2   primaryTitle    1376 non-null   object
 3   originalTitle   1376 non-null   object
 4   isAdult         1376 non-null   int64
 5   startYear       1376 non-null   object
 6   endYear         1376 non-null   object
 7   runtimeMinutes  1376 non-null   object
 8   genres          1376 non-null   object
 9   averageRating   1376 non-null   float64
 10  numVotes        1376 non-null   int64
dtypes: float64(1), int64(2), object(8)
memory usage: 129.0+ KB
None

## -------------------------------------------------------------CHAPTER 6_____no_output_____

C = movie_rating_df['averageRating'].mean()
print(C)6.869840116279075
m = movie_rating_df['numVotes'].quantile(0.8)
print(m)136.0
def imdb_weighted_rating(df, var=0.8):
    v = df['numVotes']
    R = df['averageRating']
    C = df['averageRating'].mean()
    m = df['numVotes'].quantile(var)
    df['score'] =  (v/(m+v))*R + (m/(m+v))*C #Rumus IMDb
    return df['score']

imdb_weighted_rating(movie_rating_df)

#melakukan pengecekan dataframe
print(movie_rating_df.head())      tconst  titleType              primaryTitle             originalTitle  \
0  tt0043745      short                 Lion Down                 Lion Down
1  tt0167491      video         Wicked Covergirls         Wicked Covergirls
2  tt6574096  tvEpisode      Shadow Play - Part 2      Shadow Play - Part 2
3  tt6941700  tvEpisode              RuPaul Roast              RuPaul Roast
4  tt7305674      video  UCLA Track & Field Promo  UCLA Track & Field Promo

   isAdult startYear endYear runtimeMinutes                      genres  \
0        0      1951      \N              7     Animation,Comedy,Family
1        1      1998      \N             85                       Adult
2        0      2017      \N             22  Adventure,Animation,Comedy
3        0      2017      \N             \N                  Reality-TV
4        0      2017      \N             \N                 Short,Sport

   averageRating  numVotes     score
0            7.1       459  7.047618
1            5.7         7  6.813514
2            8.5       240  7.910725
3            8.0        11  6.955323
4            9.7         7  7.009318
def simple_recommender(df, top=100):
    df = df.loc[df['numVotes'] >= m]
    df = df.sort_values(by='score', ascending=False) #urutkan dari nilai tertinggi ke terendah

    #Ambil data 100 teratas
    df = df[:top]
    return df

#Ambil data 25 teratas
print(simple_recommender(movie_rating_df, top=25))         tconst  titleType                                   primaryTitle  \
68    tt4110822  tvEpisode                                  S.O.S. Part 2
1150  tt5807780  videoGame                                     Spider-Man
236   tt2200252      video                    Attack of the Clones Review
1181  tt7697962  tvEpisode            Chapter Seventeen: The Missionaries
326   tt7124590  tvEpisode            Chapter Thirty-Four: Judgment Night
71    tt8399426  tvEpisode                                        Savages
1045  tt0533506  tvEpisode                                       The Prom
1234  tt2843830  tvEpisode                                          VIII.
1087  tt4295140   tvSeries                                   Chef's Table
1054  tt2503932  tvEpisode                                Trial and Error
448   tt0337566      video                       AC/DC: Live at Donington
189   tt5661506      video            Florence + the Machine: The Odyssey
624   tt0620159  tvEpisode                                     Strike Out
973   tt0340374  videoGame                                  Splinter Cell
1281  tt3166390  tvEpisode                         Looking for a Plus-One
49    tt9119838  tvEpisode                      Parisian Legend Has It...
151   tt3954426  tvEpisode                                Bleeding Kansas
357   tt4084774  tvEpisode                           Trial and Punishment
1217  tt3642464  tvEpisode                                    Giant Woman
1344  tt6644294  tvEpisode                 The Hostile Hospital: Part Two
314   tt0954759  tvEpisode                                   Ben Franklin
1242  tt3677742  tvSpecial  Saturday Night Live: 40th Anniversary Special
790   tt4279086  tvEpisode                       And Santa's Midnight Run
544   tt0734655  tvEpisode                              The Little People
983   tt5063116  tvEpisode                                          Shell

                                      originalTitle  isAdult startYear  \
68                                    S.O.S. Part 2        0      2015
1150                                     Spider-Man        0      2018
236                     Attack of the Clones Review        0      2010
1181            Chapter Seventeen: The Missionaries        0      2019
326             Chapter Thirty-Four: Judgment Night        0      2018
71                                          Savages        0      2018
1045                                       The Prom        0      1999
1234                                          VIII.        0      2014
1087                                   Chef's Table        0      2015
1054                                Trial and Error        0      2013
448                        AC/DC: Live at Donington        0      1992
189             Florence + the Machine: The Odyssey        0      2016
624                                      Strike Out        0      2000
973                                   Splinter Cell        0      2002
1281                         Looking for a Plus-One        0      2014
49                        Parisian Legend Has It...        0      2019
151                                 Bleeding Kansas        0      2014
357                            Trial and Punishment        0      2015
1217                                    Giant Woman        0      2014
1344                 The Hostile Hospital: Part Two        0      2018
314                                    Ben Franklin        0      2007
1242  Saturday Night Live: 40th Anniversary Special        0      2015
790                        And Santa's Midnight Run        0      2014
544                               The Little People        0      1962
983                                           Shell        0      2016

     endYear runtimeMinutes                      genres  averageRating  \
68        \N             43      Action,Adventure,Drama            9.4
1150      \N             \N    Action,Adventure,Fantasy            9.3
236       \N             86                      Comedy            9.3
1181      \N             54        Drama,Fantasy,Horror            9.2
326       \N             42         Crime,Drama,Mystery            9.1
71        \N             58       Drama,Fantasy,Romance            9.0
1045      \N             60        Action,Drama,Fantasy            8.9
1234      \N             57             Adventure,Drama            8.9
1087      \N             50                 Documentary            8.6
1054      \N             43        Drama,Fantasy,Horror            8.6
448       \N            120           Documentary,Music            8.5
189       \N             49                       Music            8.8
624       \N             30                      Comedy            8.7
973       \N             \N   Action,Adventure,Thriller            8.3
1281      \N             28        Comedy,Drama,Romance            8.7
49        \N             42                       Drama            8.9
151       \N             42               Drama,Western            8.6
357       \N             56             Adventure,Drama            8.8
1217      \N             11  Adventure,Animation,Comedy            8.4
1344      \N             40      Adventure,Comedy,Drama            8.3
314       \N             21                      Comedy            8.1
1242      \N            106                      Comedy            8.1
790       \N             42     Action,Adventure,Comedy            8.2
544       \N             25        Drama,Fantasy,Horror            8.1
983       \N             32                Comedy,Drama            8.5

      numVotes     score
68        3820  9.313018
1150     10104  9.267724
236       1411  9.086360
1181      1536  9.010465
326       1859  8.947969
71        1428  8.814769
1045      2740  8.803998
1234      1753  8.753837
1087     12056  8.580700
1054      2495  8.510566
448       1343  8.350100
189        330  8.236692
624        401  8.236496
973       2791  8.233549
1281       396  8.232140
49         263  8.208016
151        437  8.189351
357        289  8.182349
1217       566  8.103559
1344       812  8.094829
314       2766  8.042350
1242      1931  8.019061
790        823  8.011364
544       1559  8.001297
983        275  7.960580
m = df['numVotes'].quantile(0.8)_____no_output_____df = movie_rating_df.copy()

def user_prefer_recommender(df, ask_adult, ask_start_year, ask_genre, top=100):
    #ask_adult = yes/no
    if ask_adult.lower() == 'yes':
        df = df.loc[df['isAdult'] == 1]
    elif ask_adult.lower() == 'no':
        df = df.loc[df['isAdult'] == 0]

    #ask_start_year = numeric
# ------------------------------------------------------------------------------------------------------    line ini harusnya bukan str, tapi int.
    df = df.loc[df['startYear'] >= str(ask_start_year)]

    #ask_genre = 'all' atau yang lain
    if ask_genre.lower() == 'all':
        df = df
    else:
        def filter_genre(x):
            if ask_genre.lower() in str(x).lower():
                return True
            else:
                return False
        df = df.loc[df['genres'].apply(lambda x: filter_genre(x))]

    df = df.loc[df['numVotes'] >= m] #Mengambil film dengan m yang lebih besar dibanding numVotes
    df = df.sort_values(by='score', ascending=False)

    #jika kamu hanya ingin mengambil 100 teratas
    df = df[:top]
    return df

print(user_prefer_recommender(df,
                       ask_adult = 'no',
                        ask_start_year = 2000,
                       ask_genre = 'drama'
                       ))         tconst  titleType                         primaryTitle  \
68    tt4110822  tvEpisode                        S.O.S. Part 2
1181  tt7697962  tvEpisode  Chapter Seventeen: The Missionaries
326   tt7124590  tvEpisode  Chapter Thirty-Four: Judgment Night
71    tt8399426  tvEpisode                              Savages
1234  tt2843830  tvEpisode                                VIII.
...         ...        ...                                  ...
1138  tt1126516      movie              Money Hai Toh Honey Hai
1197  tt3016748      movie                       Jacob's Ladder
1208  tt3044882      movie                                Nasha
846   tt0488164      movie                          Moscow Zero
90    tt0299981    tvMovie               Highlander: The Source

                            originalTitle  isAdult startYear endYear  \
68                          S.O.S. Part 2        0      2015      \N
1181  Chapter Seventeen: The Missionaries        0      2019      \N
326   Chapter Thirty-Four: Judgment Night        0      2018      \N
71                                Savages        0      2018      \N
1234                                VIII.        0      2014      \N
...                                   ...      ...       ...     ...
1138              Money Hai Toh Honey Hai        0      2008      \N
1197                       Jacob's Ladder        0      2019      \N
1208                                Nasha        0      2013      \N
846                           Moscow Zero        0      2006      \N
90                 Highlander: The Source        0      2007      \N

     runtimeMinutes                  genres  averageRating  numVotes     score
68               43  Action,Adventure,Drama            9.4      3820  9.313052
1181             54    Drama,Fantasy,Horror            9.2      1536  9.010546
326              42     Crime,Drama,Mystery            9.1      1859  8.948036
71               58   Drama,Fantasy,Romance            9.0      1428  8.814855
1234             57         Adventure,Drama            8.9      1753  8.753908
...             ...                     ...            ...       ...       ...
1138            138            Comedy,Drama            2.8       527  3.635041
1197             89    Drama,Horror,Mystery            3.4      2065  3.614463
1208            122          Drama,Thriller            3.2      1132  3.593716
846              82     Action,Drama,Horror            3.0      1620  3.299791
90               86  Action,Adventure,Drama            3.1      8855  3.157038

[94 rows x 12 columns]
movie_rating_df.head()_____no_output_____movie_rating_df.info()<class 'pandas.core.frame.DataFrame'>
Int64Index: 1376 entries, 0 to 1375
Data columns (total 12 columns):
 #   Column          Non-Null Count  Dtype
---  ------          --------------  -----
 0   tconst          1376 non-null   object
 1   titleType       1376 non-null   object
 2   primaryTitle    1376 non-null   object
 3   originalTitle   1376 non-null   object
 4   isAdult         1376 non-null   int64
 5   startYear       1376 non-null   object
 6   endYear         1376 non-null   object
 7   runtimeMinutes  1376 non-null   object
 8   genres          1376 non-null   object
 9   averageRating   1376 non-null   float64
 10  numVotes        1376 non-null   int64
 11  score           1376 non-null   float64
dtypes: float64(2), int64(2), object(8)
memory usage: 172.0+ KB