# Notebook from rizwandel/haystack
Path: tutorials/Tutorial15_TableQA.ipynb

# Open-Domain QA on Tables
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial15_TableQA.ipynb)

This tutorial shows you how to perform question-answering on tables using the `TableTextRetriever` or `ElasticsearchRetriever` as retriever node and the `TableReader` as reader node._____no_output_____### Prepare environment

#### Colab: Enable the GPU runtime
Make sure you enable the GPU runtime to experience decent speed in this tutorial.
**Runtime -> Change Runtime type -> Hardware accelerator -> GPU**

_____no_output_____

# Make sure you have a GPU running
!nvidia-smi_____no_output_____# Install the latest release of Haystack in your own environment
#! pip install farm-haystack

# Install the latest master of Haystack
!pip install grpcio-tools==1.34.1
!pip install git+https://github.com/deepset-ai/haystack.git

# The TaPAs-based TableReader requires the torch-scatter library
!pip install torch-scatter -f https://data.pyg.org/whl/torch-1.9.0+cu111.html

# If you run this notebook on Google Colab, you might need to
# restart the runtime after installing haystack._____no_output_____

### Start an Elasticsearch server
You can start Elasticsearch on your local machine instance using Docker. If Docker is not readily available in your environment (e.g. in Colab notebooks), then you can manually download and execute Elasticsearch from source._____no_output_____

# Recommended: Start Elasticsearch using Docker via the Haystack utility function
from haystack.utils import launch_es

launch_es()_____no_output_____# In Colab / No Docker environments: Start Elasticsearch from source
! wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz -q
! tar -xzf elasticsearch-7.9.2-linux-x86_64.tar.gz
! chown -R daemon:daemon elasticsearch-7.9.2

import os
from subprocess import Popen, PIPE, STDOUT
es_server = Popen(['elasticsearch-7.9.2/bin/elasticsearch'],
                   stdout=PIPE, stderr=STDOUT,
                   preexec_fn=lambda: os.setuid(1)  # as daemon
                  )
# wait until ES has started
! sleep 30_____no_output_____# Connect to Elasticsearch
from haystack.document_stores import ElasticsearchDocumentStore

# We want to use a small model producing 512-dimensional embeddings, so we need to set embedding_dim to 512
document_store = ElasticsearchDocumentStore(host="localhost",
                                            username="",
                                            password="",
                                            index="document",
                                            embedding_dim=512)_____no_output_____

## Add Tables to DocumentStore
To quickly demonstrate the capabilities of the `TableTextRetriever` and the `TableReader` we use a subset of 1000 tables of the [Open Table-and-Text Question Answering (OTT-QA) dataset](https://github.com/wenhuchen/OTT-QA).

Just as text passages, tables are represented as `Document` objects in Haystack. The content field, though, is a pandas DataFrame instead of a string._____no_output_____

# Let's first fetch some tables that we want to query
# Here: 1000 tables from OTT-QA
from haystack.utils import fetch_archive_from_http

doc_dir = "data"
s3_url = "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/ottqa_tables_sample.json.zip"
fetch_archive_from_http(url=s3_url, output_dir=doc_dir)_____no_output_____# Add the tables to the DocumentStore

import json
from haystack import Document
import pandas as pd

def read_ottqa_tables(filename):
    processed_tables = []
    with open(filename) as tables:
        tables = json.load(tables)
        for key, table in tables.items():
            current_columns = table["header"]
            current_rows = table["data"]
            current_df = pd.DataFrame(columns=current_columns, data=current_rows)
            current_doc_title = table["title"]
            current_section_title = table["section_title"]
            document = Document(
                content=current_df,
                content_type="table",
                meta={"title": current_doc_title, "section_title": current_section_title},
                id=key
            )
            processed_tables.append(document)

    return processed_tables

tables = read_ottqa_tables("data/ottqa_tables_sample.json")
document_store.write_documents(tables, index="document")

# Showing content field and meta field of one of the Documents of content_type 'table'
print(tables[0].content)
print(tables[0].meta)      Result  ...            Score
0     Winner  ...        6-1 , 6-1
1     Winner  ...  6-2 , 4-6 , 6-3
2     Winner  ...        6-2 , 6-2
3  Runner-up  ...        3-6 , 2-6
4     Winner  ...  6-7 , 6-3 , 6-0
5     Winner  ...        6-1 , 6-0
6     Winner  ...  6-2 , 2-6 , 6-2
7     Winner  ...        6-0 , 6-4

[8 rows x 8 columns]
{'title': 'Rewa Hudson', 'section_title': 'ITF finals ( 7–3 ) -- Doubles ( 7–1 )'}

## Initalize Retriever, Reader, & Pipeline

### Retriever

Retrievers help narrowing down the scope for the Reader to a subset of tables where a given question could be answered.
They use some simple but fast algorithm.

**Here:** We use the `TableTextRetriever` capable of retrieving relevant content among a database
of texts and tables using dense embeddings. It is an extension of the `DensePassageRetriever` and consists of three encoders (one query encoder, one text passage encoder and one table encoder) that create embeddings in the same vector space. More details on the `TableTextRetriever` and how it is trained can be found in [this paper](https://arxiv.org/abs/2108.04049).

**Alternatives:**

- `ElasticsearchRetriever` that uses BM25 algorithm
_____no_output_____

from haystack.nodes.retriever import TableTextRetriever

retriever = TableTextRetriever(
    document_store=document_store,
    query_embedding_model="deepset/bert-small-mm_retrieval-question_encoder",
    passage_embedding_model="deepset/bert-small-mm_retrieval-passage_encoder",
    table_embedding_model="deepset/bert-small-mm_retrieval-table_encoder",
    embed_meta_fields=["title", "section_title"]
)_____no_output_____# Add table embeddings to the tables in DocumentStore
document_store.update_embeddings(retriever=retriever)_____no_output_____## Alternative: ElasticsearchRetriever
#from haystack.nodes.retriever import ElasticsearchRetriever
#retriever = ElasticsearchRetriever(document_store=document_store)_____no_output_____# Try the Retriever
from haystack.utils import print_documents

retrieved_tables = retriever.retrieve("How many twin buildings are under construction?", top_k=5)
# Get highest scored table
print(retrieved_tables[0].content)                             Name  ...              Status
0                  Twin Towers II  ...         Never built
1              World Trade Center  ...           Destroyed
2                Three Sixty West  ...  Under construction
3                  Gateway Towers  ...  Under construction
4                 Rustomjee Crown  ...  Under construction
5                  Orchid Heights  ...             On-hold
6                Hermitage Towers  ...            Proposed
7             Lokhandwala Minerva  ...  Under construction
8                    Lamar Towers  ...  Under construction
9            Indonesia One Towers  ...  Under construction
10                       Sky link  ...            Approved
11                  Vida Za'abeel  ...            Proposed
12  Broadway Corridor Twin Towers  ...         Never built
13   India Bulls Sky Forest Tower  ...  Under construction
14                 Capital Towers  ...  Under construction
15               One Avighna Park  ...  Under construction
16                     NEB Towers  ...             On hold
17          The Destiny ( Tower )  ...  Under construction
18          Oberoi Esquire Towers  ...  Under construction
19                Bhoomi Celestia  ...  Under construction

[20 rows x 6 columns]

### Reader
The `TableReader` is based on TaPas, a transformer-based language model capable of grasping the two-dimensional structure of a table. It scans the tables returned by the retriever and extracts the anser. The available TableReader models can be found [here](https://huggingface.co/models?pipeline_tag=table-question-answering&sort=downloads).

**Notice**: The `TableReader` will return an answer for each table, even if the query cannot be answered by the table. Furthermore, the confidence scores are not useful as of now, given that they will *always* be very high (i.e. 1 or close to 1)._____no_output_____

from haystack.nodes import TableReader

reader = TableReader(model_name_or_path="google/tapas-base-finetuned-wtq", max_seq_len=512)_____no_output_____# Try the TableReader on one Table (highest-scored retrieved table from previous section)

table_doc = document_store.get_document_by_id("List_of_tallest_twin_buildings_and_structures_in_the_world_1")
print(table_doc.content)                             Name  ...              Status
0                  Twin Towers II  ...         Never built
1              World Trade Center  ...           Destroyed
2                Three Sixty West  ...  Under construction
3                  Gateway Towers  ...  Under construction
4                 Rustomjee Crown  ...  Under construction
5                  Orchid Heights  ...             On-hold
6                Hermitage Towers  ...            Proposed
7             Lokhandwala Minerva  ...  Under construction
8                    Lamar Towers  ...  Under construction
9            Indonesia One Towers  ...  Under construction
10                       Sky link  ...            Approved
11                  Vida Za'abeel  ...            Proposed
12  Broadway Corridor Twin Towers  ...         Never built
13   India Bulls Sky Forest Tower  ...  Under construction
14                 Capital Towers  ...  Under construction
15               One Avighna Park  ...  Under construction
16                     NEB Towers  ...             On hold
17          The Destiny ( Tower )  ...  Under construction
18          Oberoi Esquire Towers  ...  Under construction
19                Bhoomi Celestia  ...  Under construction

[20 rows x 6 columns]
from haystack.utils import print_answers

prediction = reader.predict(query="How many twin buildings are under construction?", documents=[table_doc])
print_answers(prediction, details="all"){   'answers': [   Answer(answer='12', type='extractive', score=1.0, context=                             Name  ...              Status
0                  Twin Towers II  ...         Never built
1              World Trade Center  ...           Destroyed
2                Three Sixty West  ...  Under construction
3                  Gateway Towers  ...  Under construction
4                 Rustomjee Crown  ...  Under construction
5                  Orchid Heights  ...             On-hold
6                Hermitage Towers  ...            Proposed
7             Lokhandwala Minerva  ...  Under construction
8                    Lamar Towers  ...  Under construction
9            Indonesia One Towers  ...  Under construction
10                       Sky link  ...            Approved
11                  Vida Za'abeel  ...            Proposed
12  Broadway Corridor Twin Towers  ...         Never built
13   India Bulls Sky Forest Tower  ...  Under construction
14                 Capital Towers  ...  Under construction
15               One Avighna Park  ...  Under construction
16                     NEB Towers  ...             On hold
17          The Destiny ( Tower )  ...  Under construction
18          Oberoi Esquire Towers  ...  Under construction
19                Bhoomi Celestia  ...  Under construction

[20 rows x 6 columns], offsets_in_document=[Span(start=12, end=13), Span(start=18, end=19), Span(start=24, end=25), Span(start=42, end=43), Span(start=48, end=49), Span(start=54, end=55), Span(start=78, end=79), Span(start=84, end=85), Span(start=90, end=91), Span(start=102, end=103), Span(start=108, end=109), Span(start=114, end=115)], offsets_in_context=[Span(start=12, end=13), Span(start=18, end=19), Span(start=24, end=25), Span(start=42, end=43), Span(start=48, end=49), Span(start=54, end=55), Span(start=78, end=79), Span(start=84, end=85), Span(start=90, end=91), Span(start=102, end=103), Span(start=108, end=109), Span(start=114, end=115)], document_id='List_of_tallest_twin_buildings_and_structures_in_the_world_1', meta={'aggregation_operator': 'COUNT', 'answer_cells': ['Three Sixty West', 'Gateway Towers', 'Rustomjee Crown', 'Lokhandwala Minerva', 'Lamar Towers', 'Indonesia One Towers', 'India Bulls Sky Forest Tower', 'Capital Towers', 'One Avighna Park', 'The Destiny ( Tower )', 'Oberoi Esquire Towers', 'Bhoomi Celestia']})],
    'query': 'How many twin buildings are under construction?'}

The offsets in the `offsets_in_document` and `offsets_in_context` field indicate the table cells that the model predicts to be part of the answer. They need to be interpreted on the linearized table, i.e., a flat list containing all of the table cells.

In the `Answer`'s meta field, you can find the aggreagtion operator used to construct the answer (in this case `COUNT`) and the answer cells as strings._____no_output_____

print(f"Predicted answer: {prediction['answers'][0].answer}")
print(f"Meta field: {prediction['answers'][0].meta}")Predicted answer: 12
Meta field: {'aggregation_operator': 'COUNT', 'answer_cells': ['Three Sixty West', 'Gateway Towers', 'Rustomjee Crown', 'Lokhandwala Minerva', 'Lamar Towers', 'Indonesia One Towers', 'India Bulls Sky Forest Tower', 'Capital Towers', 'One Avighna Park', 'The Destiny ( Tower )', 'Oberoi Esquire Towers', 'Bhoomi Celestia']}

### Pipeline
The retriever and the retriever can be sticked together to a pipeline in order to first retrieve relevant tables and then extract the answer.

**Notice**: Given that the `TableReader` does not provide useful confidence scores and returns an answer for each of the tables, the sorting of the answers might be not helpful._____no_output_____

# Initialize pipeline
from haystack import Pipeline

table_qa_pipeline = Pipeline()
table_qa_pipeline.add_node(component=retriever, name="TableTextRetriever", inputs=["Query"])
table_qa_pipeline.add_node(component=reader, name="TableReader", inputs=["TableTextRetriever"])_____no_output_____prediction = table_qa_pipeline.run("How many twin buildings are under construction?")
print_answers(prediction, details="minimal")[   {   'answer': '12',
        'context':                              Name  ...              Status
0                  Twin Towers II  ...         Never built
1              World Trade Center  ...           Destroyed
2                Three Sixty West  ...  Under construction
3                  Gateway Towers  ...  Under construction
4                 Rustomjee Crown  ...  Under construction
5                  Orchid Heights  ...             On-hold
6                Hermitage Towers  ...            Proposed
7             Lokhandwala Minerva  ...  Under construction
8                    Lamar Towers  ...  Under construction
9            Indonesia One Towers  ...  Under construction
10                       Sky link  ...            Approved
11                  Vida Za'abeel  ...            Proposed
12  Broadway Corridor Twin Towers  ...         Never built
13   India Bulls Sky Forest Tower  ...  Under construction
14                 Capital Towers  ...  Under construction
15               One Avighna Park  ...  Under construction
16                     NEB Towers  ...             On hold
17          The Destiny ( Tower )  ...  Under construction
18          Oberoi Esquire Towers  ...  Under construction
19                Bhoomi Celestia  ...  Under construction

[20 rows x 6 columns]},
    {   'answer': '7',
        'context':                                             Building or structure  ...                Listing
0                 Ford Assembly Plant Building Now Public Storage  ...       Seattle landmark
1                                        Immanuel Lutheran Church  ...  Seattle landmark NRHP
2                                                    Jensen Block  ...       Seattle landmark
3         Lake Union Steam Plant and Hydro House Now Zymogenetics  ...       Seattle landmark
4        New Richmond Laundry Now part of the Alley24 development  ...       Seattle landmark
5                         St. Spiridon Russian Orthodox Cathedral  ...       Seattle landmark
6  Supply Laundry Building Now part of the Stackhouse development  ...  Seattle landmark NRHP

[7 rows x 3 columns]},
    {   'answer': '8',
        'context':      Years                         Venue                      Location
0     1989      Bamm Hollow Country Club         Lincroft , New Jersey
1  1987-88         Navesink Country Club       Middletown , New Jersey
2  1985-86        Fairmount Country Club          Chatham , New Jersey
3  1983-84  Upper Montclair Country Club          Clifton , New Jersey
4     1982          Wykagyl Country Club       New Rochelle , New York
5     1981        Ridgewood Country Club          Paramus , New Jersey
6  1979-80  Upper Montclair Country Club          Clifton , New Jersey
7  1976-78         Forsgate Country Club  Monroe Township , New Jersey},
    {   'answer': '8',
        'context':    Model               Specification  ... Prime mover          Power output
0   RS-1                     E-1641A  ...      6-539T  1,000 hp ( 0.75 MW )
1   RS-2  E-1661 , E-1661A , E-1661B  ...      12-244  1,500 hp ( 1.12 MW )
2   RS-2                     E-1661C  ...      12-244  1,600 hp ( 1.19 MW )
3   RS-3  E-1662 , E-1662A , E-1662B  ...      12-244  1,600 hp ( 1.19 MW )
4  RS-11                      DL-701  ...      12-251  1,800 hp ( 1.34 MW )
5  RS-27                      DL-640  ...      16-251  2,400 hp ( 1.79 MW )
6  RS-32                      DL-721  ...      12-251  2,000 hp ( 1.49 MW )
7  RS-36                   DL-701XAP  ...      12-251  1,800 hp ( 1.34 MW )

[8 rows x 7 columns]},
    {   'answer': '10',
        'context':                                 Name or designation  ...                                                                            Notes
0                                 Aluminum Overcast  ...                                                    One of only ten flyable B-17s
1                              Avro Lancaster PA474  ...                      One of only two Lancasters in flying condition in the world
2   Avro Vulcan XH558 , aka Spirit of Great Britain  ...  The only Cold War / Falklands War -era Vulcan bomber to fly after 1986 . Res...
3                               Douglas DC-7B N836D  ...
4                               Douglas R4D-3 N763A  ...  Used by the US Navy during World War II . Placed on the National Register of...
5                                              FIFI  ...                                                     One of only two B-29s flying
6                                      Glacier Girl  ...  Forced to land in Greenland in 1942 along with five other P-38s and two B-17...
7                            Hawker Hurricane PZ865  ...  Last Hurricane produced . Retained by Hawker Aircraft for trials work . Give...
8                                        My Gal Sal  ...  Forced to land on the Greenland icecap during World War II and abandoned , a...
9                               Piccadilly Lilly II  ...  Last B-17 to serve in the US Air Force , flying her last mission in 1959 . U...
10                                    The Pink Lady  ...      Only flying B-17 survivor to have seen action in Europe during World War II
11                                          Sally B  ...         Only airworthy B-17 left in Europe . Used in the 1990 film Memphis Belle
12                              Sentimental Journey  ...  Based at the Commemorative Air Force Museum in Mesa , Arizona , and regularl...
13                                   Shoo Shoo Baby  ...                      Crash-landed in Sweden in 1944 . Restored from 1978 to 1988
14                                      Swamp Ghost  ...  Ran out of fuel and crash-landed in a swamp in Papua New Guinea . Recovered ...
15                                    Texas Raiders  ...  Maintained and flown by the Commemorative Air Force ( formerly Confederate A...
16                                      Thunderbird  ...                       Housed at the Lone Star Flight Museum in Galveston , Texas
17                                       Worry Bird  ...  Served in World War II and the Korean War before being retired in 1957 and p...
18                                      Yankee Lady  ...                                                                          Flyable

[19 rows x 6 columns]},
    {   'answer': '13',
        'context':        N    Year Country  ... Link                       Remark  K
0   003+    2013   INDIA  ...   LK                           RK  K
1    005    2006     USA  ...   LK                           RK  K
2    010    2014     ZAF  ...   LK                           RK  K
3    020    2010     USA  ...   LK                           RK  K
4    030   201 ?     USA  ...   LK                           RK  K
5    040    2007     USA  ...   LK                           RK  K
6    042    2004     USA  ...   LK  Only G-S With Large Battery  K
7    050   201 ?     USA  ...   LK                           RK  K
8    100  20 ? ?     USA  ...   LK                           RK  K
9    200  20 ? ?     USA  ...   LK                           RK  K
10   300    2013     EUR  ...   LK                           RK  K
11   400  20 ? ?     USA  ...   LK                           RK  K
12   995  20 ? ?     USA  ...   LK                           RK  K

[13 rows x 12 columns]},
    {   'answer': '5',
        'context':                    Team  ... Capacity
0              Barnsley  ...   23,009
1             Blackpool  ...   16,750
2         Bradford City  ...   25,136
3         Burton Albion  ...    6,912
4                  Bury  ...   11,840
5          Chesterfield  ...   10,400
6     Colchester United  ...   10,105
7         Coventry City  ...   32,500
8       Crewe Alexandra  ...   10,066
9      Doncaster Rovers  ...   15,231
10       Fleetwood Town  ...    5,311
11           Gillingham  ...   11,582
12             Millwall  ...   20,146
13      Oldham Athletic  ...   13,512
14  Peterborough United  ...   14,319
15            Port Vale  ...   18,947
16             Rochdale  ...   10,249
17    Scunthorpe United  ...    9,183
18     Sheffield United  ...   32,702
19      Shrewsbury Town  ...    9,875

[20 rows x 4 columns]},
    {   'answer': '7',
        'context':                                           Resource Name  ...                Added
0  Whitfield Estates-Broughton Street Historic District  ...    October 29 , 1993
1                                 John M. Beasley House  ...       March 5 , 1996
2    Whitfield Estates-Lantana Avenue Historic District  ...       March 8 , 1997
3                                          Austin House  ...    February 5 , 1998
4                                      Reid-Woods House  ...     August 31 , 2000
5                               Villa Serena Apartments  ...  September 29 , 2000
6                                  Paul M. Souder House  ...    November 2 , 2000
7                               Stevens-Gilchrist House  ...     August 17 , 2001

[8 rows x 3 columns]},
    {   'answer': '19',
        'context':                                           Name ( Alternative names in parenthesis )  ...                                                           Carries
0                                                           Arboretum Sewer Trestle  ...                                              Sewer and a footpath
1                                             Ballard Bridge ( 15th Avenue Bridge )  ...                                                    15th Avenue NW
2                                                                 Cowen Park Bridge  ...                                                    15th Avenue NE
3                                                         First Avenue South Bridge  ...                                                    State Route 99
4                                          Fremont Bridge ( Fremont Avenue Bridge )  ...                 Road connecting Fremont Avenue N and 4th Avenue N
5                               George Washington Memorial Bridge ( Aurora Bridge )  ...                                                    State Route 99
6                  Homer M. Hadley Memorial Bridge ( Third Lake Washington Bridge )  ...                                                     Interstate 90
7                         Jeanette Williams Memorial Bridge ( West Seattle Bridge )  ...  Road connecting Fauntleroy Way SW and the Spokane Street Viaduct
8                                    Jose Rizal Bridge ( 12th Avenue South Bridge )  ...                                   12th Avenue S and Interstate 90
9                                                   Lacey V. Murrow Memorial Bridge  ...                                                     Interstate 90
10                                                                  Magnolia Bridge  ...                                                 W Garfield Street
11                                                                  Montlake Bridge  ...                                                   State Route 513
12                                                    North Queen Anne Drive Bridge  ...                                                N Queen Anne Drive
13                                                                Salmon Bay Bridge  ...                                                      BNSF Railway
14                                                                Ship Canal Bridge  ...                                                      Interstate 5
15                                                              Schmitz Park Bridge  ...                                                    SW Admiral Way
16                                                            Spokane Street Bridge  ...                                                 SW Spokane Street
17  SR 520 Albert D. Rosellini Evergreen Point Floating Bridge ( Evergreen Point...  ...                                                   State Route 520
18                                    20th Avenue NE Bridge ( Ravenna Park Bridge )  ...                         20th Avenue NE ( pedestrian access only )
19                                                                University Bridge  ...                                                Eastlake Avenue NE

[20 rows x 6 columns]},
    {   'answer': '8',
        'context':              Location  ...                                                                         Comments
0                 Ayr  ...  Known as Wonderwest World 1988-1998 ; operated as Craig Tara by Haven since ...
1             Bahamas  ...  The site is now occupied by a new hotel and marina complex known as Old Baha...
2        Barry Island  ...                Operated independently until closure in 1996 . Demolished in 2005
3        Bognor Regis  ...         Known as Southcoast World 1987-1998 . Still open as Butlins Bognor Regis
4             Clacton  ...             Demolished , now a housing estate . Small area yet to be redeveloped
5  Filey Holiday Camp  ...  Operated independently for six weeks in 1986 , but the venture failed and it...
6            Minehead  ...  Known as Somerwest World 1986-1998 . Still open as Butlins Minehead 30 April...
7              Mosney  ...  Operated independently until closure and conversion into an Irish Government...
8            Pwllheli  ...  Known as Starcoast World 1990-1998 ; operated as Hafan Y Mor by Haven since ...
9            Skegness  ...               Known as Funcoast World 1987-1998 . Still open as Butlins Skegness

[10 rows x 4 columns]}]

## About us

This [Haystack](https://github.com/deepset-ai/haystack/) notebook was made with love by [deepset](https://deepset.ai/) in Berlin, Germany

We bring NLP to the industry via open source!
Our focus: Industry specific language models & large scale QA systems.

Some of our other work:
- [German BERT](https://deepset.ai/german-bert)
- [GermanQuAD and GermanDPR](https://deepset.ai/germanquad)
- [FARM](https://github.com/deepset-ai/FARM)

Get in touch:
[Twitter](https://twitter.com/deepset_ai) | [LinkedIn](https://www.linkedin.com/company/deepset-ai/) | [Slack](https://haystack.deepset.ai/community/join) | [GitHub Discussions](https://github.com/deepset-ai/haystack/discussions) | [Website](https://deepset.ai)

By the way: [we're hiring!](https://www.deepset.ai/jobs)
_____no_output_____


# Notebook from gopala-kr/ds-notebooks
Path: tests/practice/ktf_1.1 Introduction - Deep Learning and ANN.ipynb

# Introduction to Deep Learning_____no_output_____Deep learning allows computational models that are composed of multiple processing **layers** to learn representations of data with multiple levels of abstraction._____no_output_____These methods have dramatically improved the state-of-the-art in speech recognition, visual object recognition, object detection and many other domains such as drug discovery and genomics. _____no_output_____**Deep learning** is one of the leading tools in data analysis these days and one of the most common frameworks for deep learning is **Keras**. _____no_output_____The Tutorial will provide an introduction to deep learning using `keras` with practical code examples._____no_output_____## This Section will cover:

* Getting a conceptual understanding of multi-layer neural networks
* Training neural networks for image classification
* Implementing the powerful backpropagation algorithm
* Debugging neural network implementations_____no_output_____# Building Blocks: Artificial Neural Networks (ANN)_____no_output_____In machine learning and cognitive science, an artificial neural network (ANN) is a network inspired by biological neural networks which are used to estimate or approximate functions that can depend on a large number of inputs that are generally unknown_____no_output_____An ANN is built from nodes (neurons) stacked in layers between the feature vector and the target vector. _____no_output_____A node in a neural network is built from Weights and Activation function_____no_output_____An early version of ANN built from one node was called the **Perceptron**_____no_output__________no_output_____The Perceptron is an algorithm for supervised learning of binary classifiers. functions that can decide whether an input (represented by a vector of numbers) belongs to one class or another.

Much like logistic regression, the weights in a neural net are being multiplied by the input vertor summed up and feeded into the activation function's input._____no_output_____A Perceptron Network can be designed to have *multiple layers*, leading to the **Multi-Layer Perceptron** (aka `MLP`)_____no_output__________no_output_____# Single Layer Neural Network_____no_output_____

_(Source: Python Machine Learning, S. Raschka)______no_output_____### Weights Update Rule_____no_output_____- We use a **gradient descent** optimization algorithm to learn the _Weights Coefficients_ of the model.

- In every **epoch** (pass over the training set), we update the weight vector $w$ using the following update rule:

$$
w = w + \Delta w, \text{where } \Delta w = - \eta \nabla J(w)
$$

In other words, we computed the gradient based on the whole training set and updated the weights of the model by taking a step into the **opposite direction** of the gradient $ \nabla J(w)$.

In order to fin the **optimal weights of the model**, we optimized an objective function (e.g. the Sum of Squared Errors (SSE)) cost function $J(w)$.

Furthermore, we multiply the gradient by a factor, the learning rate $\eta$ , which we choose carefully to balance the **speed of learning** against the risk of overshooting the global minimum of the cost function._____no_output_____### Gradient Descent_____no_output_____In **gradient descent optimization**, we update all the **weights simultaneously** after each epoch, and we define the _partial derivative_ for each weight $w_j$ in the weight vector $w$ as follows:

$$
\frac{\partial}{\partial w_j} J(w) = \sum_{i} ( y^{(i)} - a^{(i)} )  x^{(i)}_j
$$

**Note**: _The superscript $(i)$ refers to the i-th sample. The subscript $j$ refers to the j-th dimension/feature_

Here $y^{(i)}$ is the target class label of a particular sample $x^{(i)}$ , and $a^{(i)}$ is the **activation** of the neuron

(which is a linear function in the special case of _Perceptron_)._____no_output_____We define the **activation function** $\phi(\cdot)$ as follows:

$$
\phi(z) = z = a = \sum_{j} w_j x_j = \mathbf{w}^T \mathbf{x}
$$_____no_output_____### Binary Classification_____no_output_____While we used the **activation** $\phi(z)$ to compute the gradient update, we may use a **threshold function** _(Heaviside function)_ to squash the continuous-valued output into binary class labels for prediction:

$$
\hat{y} =
\begin{cases}
    1 & \text{if } \phi(z) \geq 0 \\
    0 & \text{otherwise}
\end{cases}
$$_____no_output_____## Building Neural Nets from scratch
_____no_output_____### Idea:

We will build the neural networks from first principles.
We will create a very simple model and understand how it works. We will also be implementing backpropagation algorithm.

**Please note that this code is not optimized and not to be used in production**.

This is for instructive purpose - for us to understand how ANN works.

Libraries like `theano` have highly optimized code._____no_output_____### Perceptron and Adaline Models

Take a look at this notebook :  Perceptron and Adaline _____no_output_____If you want a sneak peek of alternate (production ready) implementation of _Perceptron_ for instance try:
```python
from sklearn.linear_model import Perceptron
```_____no_output_____## Introducing the multi-layer neural network architecture_____no_output_____

_(Source: Python Machine Learning, S. Raschka)______no_output_____Now we will see how to connect **multiple single neurons** to a **multi-layer feedforward neural network**; this special type of network is also called a **multi-layer perceptron** (MLP).

The figure shows the concept of an **MLP** consisting of three layers: one _input_ layer, one _hidden_ layer, and one _output_ layer.

The units in the hidden layer are fully connected to the input layer, and the output layer is fully connected to the hidden layer, respectively.

If such a network has **more than one hidden layer**, we also call it a **deep artificial neural network**.
_____no_output_____### Notation_____no_output_____we denote the `ith` activation unit in the `lth` layer as $a_i^{(l)}$ , and the activation units $a_0^{(1)}$ and
$a_0^{(2)}$ are the **bias units**, respectively, which we set equal to $1$.

The _activation_ of the units in the **input layer** is just its input plus the bias unit:

$$
\mathbf{a}^{(1)} = [a_0^{(1)}, a_1^{(1)}, \ldots, a_m^{(1)}]^T = [1, x_1^{(i)}, \ldots, x_m^{(i)}]^T
$$

**Note**: $x_j^{(i)}$ refers to the jth feature/dimension of the ith sample_____no_output_____### Notes on Notation (usually) Adopted

The terminology around the indices (subscripts and superscripts) may look a little bit confusing at first.

You may wonder why we wrote $w_{j,k}^{(l)}$ and not $w_{k,j}^{(l)}$ to refer to
the **weight coefficient** that connects the *kth* unit in layer $l$ to the jth unit in layer $l+1$.

What may seem a little bit quirky at first will make much more sense later when we **vectorize** the neural network representation.

For example, we will summarize the weights that connect the input and hidden layer by a matrix
$$ W^{(1)} \in \mathbb{R}^{h×[m+1]}$$

where $h$ is the number of hidden units and $m + 1$ is the number of hidden units plus bias unit. _____no_output_____

_(Source: Python Machine Learning, S. Raschka)______no_output_____## Forward Propagation_____no_output_____* Starting at the input layer, we forward propagate the patterns of the training data through the network to generate an output.

* Based on the network's output, we calculate the error that we want to minimize using a cost function that we will describe later.

* We backpropagate the error, find its derivative with respect to each weight in the network, and update the model._____no_output_____### Sigmoid Activation_____no_output_____

_(Source: Python Machine Learning, S. Raschka)______no_output_____

_(Source: Python Machine Learning, S. Raschka)______no_output_____

_(Source: Python Machine Learning, S. Raschka)______no_output_____## Backward Propagation_____no_output_____The weights of each neuron are learned by **gradient descent**, where each neuron's error is derived with respect to it's weight._____no_output_____

_(Source: Python Machine Learning, S. Raschka)______no_output_____Optimization is done for each layer with respect to the previous layer in a technique known as **BackPropagation**._____no_output__________no_output_____(*The following code is inspired from [these](https://github.com/dennybritz/nn-from-scratch) terrific notebooks*)_____no_output_____

#load watermark
%load_ext watermark
%watermark -a 'Gopala KR' -u -d -v -p watermark,numpy,pandas,matplotlib,nltk,sklearn,tensorflow,theano,mxnet,chainer,seaborn,keras,tflearn,bokeh,gensimWARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.
Using TensorFlow backend.
/srv/venv/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  if d.decorator_argspec is not None), _inspect.getargspec(target))
/srv/venv/lib/python3.5/site-packages/scipy/sparse/sparsetools.py:20: DeprecationWarning: `scipy.sparse.sparsetools` is deprecated!
scipy.sparse.sparsetools is a private module for scipy.sparse, and should not be used.
  _deprecated()
# Import the required packages
import numpy as np
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
import scipy_____no_output_____# Display plots in notebook
%matplotlib inline
# Define plot's default figure size
matplotlib.rcParams['figure.figsize'] = (10.0, 8.0)_____no_output_____#read the datasets
train = pd.read_csv("../data/intro_to_ann.csv")_____no_output_____X, y = np.array(train.ix[:,0:2]), np.array(train.ix[:,2])/srv/venv/lib/python3.5/site-packages/ipykernel_launcher.py:1: DeprecationWarning:
.ix is deprecated. Please use
.loc for label based indexing or
.iloc for positional indexing

See the documentation here:
http://pandas.pydata.org/pandas-docs/stable/indexing.html#ix-indexer-is-deprecated
  """Entry point for launching an IPython kernel.
X.shape_____no_output_____y.shape_____no_output_____#Let's plot the dataset and see how it is
plt.scatter(X[:,0], X[:,1], s=40, c=y, cmap=plt.cm.BuGn)_____no_output_____

## Start Building our MLP building blocks

Note: This process will eventually result in our own Neural Networks class_____no_output_____### A look at the details_____no_output__________no_output_____

import random
random.seed(123)

# calculate a random number where:  a <= rand < b
def rand(a, b):
    return (b-a)*random.random() + a_____no_output_____

##### Function to generate a random number, given two numbers

**Where will it be used?**: When we initialize the neural networks, the weights have to be randomly assigned._____no_output_____

# Make a matrix
def makeMatrix(I, J, fill=0.0):
    return np.zeros([I,J])_____no_output_____

### Define our activation function. Let's use sigmoid function_____no_output_____

# our sigmoid function
def sigmoid(x):
    #return math.tanh(x)
    return 1/(1+np.exp(-x))_____no_output_____

### Derivative of our activation function.

Note: We need this when we run the backpropagation algorithm
_____no_output_____

# derivative of our sigmoid function, in terms of the output (i.e. y)
def dsigmoid(y):
    return y - y**2_____no_output_____

### Our neural networks class

When we first create a neural networks architecture, we need to know the number of inputs, number of hidden layers and number of outputs.

The weights have to be randomly initialized._____no_output_____```python
class MLP:
    def __init__(self, ni, nh, no):
        # number of input, hidden, and output nodes
        self.ni = ni + 1 # +1 for bias node
        self.nh = nh
        self.no = no

        # activations for nodes
        self.ai = [1.0]*self.ni
        self.ah = [1.0]*self.nh
        self.ao = [1.0]*self.no

        # create weights
        self.wi = makeMatrix(self.ni, self.nh)
        self.wo = makeMatrix(self.nh, self.no)

        # set them to random vaules
        self.wi = rand(-0.2, 0.2, size=self.wi.shape)
        self.wo = rand(-2.0, 2.0, size=self.wo.shape)

        # last change in weights for momentum
        self.ci = makeMatrix(self.ni, self.nh)
        self.co = makeMatrix(self.nh, self.no)
```_____no_output_____### Activation Function_____no_output_____```python
def activate(self, inputs):

    if len(inputs) != self.ni-1:
        print(inputs)
        raise ValueError('wrong number of inputs')

    # input activations
    for i in range(self.ni-1):
        self.ai[i] = inputs[i]

    # hidden activations
    for j in range(self.nh):
        sum_h = 0.0
        for i in range(self.ni):
            sum_h += self.ai[i] * self.wi[i][j]
        self.ah[j] = sigmoid(sum_h)

    # output activations
    for k in range(self.no):
        sum_o = 0.0
        for j in range(self.nh):
            sum_o += self.ah[j] * self.wo[j][k]
        self.ao[k] = sigmoid(sum_o)

    return self.ao[:]
```_____no_output_____### BackPropagation_____no_output_____```python
def backPropagate(self, targets, N, M):

    if len(targets) != self.no:
        print(targets)
        raise ValueError('wrong number of target values')

    # calculate error terms for output
    output_deltas = np.zeros(self.no)
    for k in range(self.no):
        error = targets[k]-self.ao[k]
        output_deltas[k] = dsigmoid(self.ao[k]) * error

    # calculate error terms for hidden
    hidden_deltas = np.zeros(self.nh)
    for j in range(self.nh):
        error = 0.0
        for k in range(self.no):
            error += output_deltas[k]*self.wo[j][k]
        hidden_deltas[j] = dsigmoid(self.ah[j]) * error

    # update output weights
    for j in range(self.nh):
        for k in range(self.no):
            change = output_deltas[k] * self.ah[j]
            self.wo[j][k] += N*change +
                             M*self.co[j][k]
            self.co[j][k] = change

    # update input weights
    for i in range(self.ni):
        for j in range(self.nh):
            change = hidden_deltas[j]*self.ai[i]
            self.wi[i][j] += N*change +
                             M*self.ci[i][j]
            self.ci[i][j] = change

    # calculate error
    error = 0.0
    for k in range(len(targets)):
        error += 0.5*(targets[k]-self.ao[k])**2
    return error
```_____no_output_____

# Putting all together

class MLP:
    def __init__(self, ni, nh, no):
        # number of input, hidden, and output nodes
        self.ni = ni + 1 # +1 for bias node
        self.nh = nh
        self.no = no

        # activations for nodes
        self.ai = [1.0]*self.ni
        self.ah = [1.0]*self.nh
        self.ao = [1.0]*self.no

        # create weights
        self.wi = makeMatrix(self.ni, self.nh)
        self.wo = makeMatrix(self.nh, self.no)

        # set them to random vaules
        for i in range(self.ni):
            for j in range(self.nh):
                self.wi[i][j] = rand(-0.2, 0.2)
        for j in range(self.nh):
            for k in range(self.no):
                self.wo[j][k] = rand(-2.0, 2.0)

        # last change in weights for momentum
        self.ci = makeMatrix(self.ni, self.nh)
        self.co = makeMatrix(self.nh, self.no)

    def backPropagate(self, targets, N, M):

        if len(targets) != self.no:
            print(targets)
            raise ValueError('wrong number of target values')

        # calculate error terms for output
        output_deltas = np.zeros(self.no)
        for k in range(self.no):
            error = targets[k]-self.ao[k]
            output_deltas[k] = dsigmoid(self.ao[k]) * error

        # calculate error terms for hidden
        hidden_deltas = np.zeros(self.nh)
        for j in range(self.nh):
            error = 0.0
            for k in range(self.no):
                error += output_deltas[k]*self.wo[j][k]
            hidden_deltas[j] = dsigmoid(self.ah[j]) * error

        # update output weights
        for j in range(self.nh):
            for k in range(self.no):
                change = output_deltas[k] * self.ah[j]
                self.wo[j][k] += N*change + M*self.co[j][k]
                self.co[j][k] = change

        # update input weights
        for i in range(self.ni):
            for j in range(self.nh):
                change = hidden_deltas[j]*self.ai[i]
                self.wi[i][j] += N*change + M*self.ci[i][j]
                self.ci[i][j] = change

        # calculate error
        error = 0.0
        for k in range(len(targets)):
            error += 0.5*(targets[k]-self.ao[k])**2
        return error

    def test(self, patterns):
        self.predict = np.empty([len(patterns), self.no])
        for i, p in enumerate(patterns):
            self.predict[i] = self.activate(p)
            #self.predict[i] = self.activate(p[0])

    def activate(self, inputs):

        if len(inputs) != self.ni-1:
            print(inputs)
            raise ValueError('wrong number of inputs')

        # input activations
        for i in range(self.ni-1):
            self.ai[i] = inputs[i]

        # hidden activations
        for j in range(self.nh):
            sum_h = 0.0
            for i in range(self.ni):
                sum_h += self.ai[i] * self.wi[i][j]
            self.ah[j] = sigmoid(sum_h)

        # output activations
        for k in range(self.no):
            sum_o = 0.0
            for j in range(self.nh):
                sum_o += self.ah[j] * self.wo[j][k]
            self.ao[k] = sigmoid(sum_o)

        return self.ao[:]

    def train(self, patterns, iterations=1000, N=0.5, M=0.1):
        # N: learning rate
        # M: momentum factor
        patterns = list(patterns)
        for i in range(iterations):
            error = 0.0
            for p in patterns:
                inputs = p[0]
                targets = p[1]
                self.activate(inputs)
                error += self.backPropagate([targets], N, M)
            if i % 5 == 0:
                print('error in interation %d : %-.5f' % (i,error))
            print('Final training error: %-.5f' % error)_____no_output_____

### Running the model on our dataset_____no_output_____

# create a network with two inputs, one hidden, and one output nodes
ann = MLP(2, 1, 1)

%timeit -n 1 -r 1 ann.train(zip(X,y), iterations=2)error in interation 0 : 53.62995
Final training error: 53.62995
Final training error: 47.35136
33.1 ms ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)

### Predicting on training dataset and measuring in-sample accuracy_____no_output_____

%timeit -n 1 -r 1 ann.test(X)7.05 ms ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)
prediction = pd.DataFrame(data=np.array([y, np.ravel(ann.predict)]).T,
                          columns=["actual", "prediction"])
prediction.head()_____no_output_____np.min(prediction.prediction)_____no_output_____

### Let's visualize and observe the results_____no_output_____

# Helper function to plot a decision boundary.
# This generates the contour plot to show the decision boundary visually
def plot_decision_boundary(nn_model):
    # Set min and max values and give it some padding
    x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
    y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
    h = 0.01
    # Generate a grid of points with distance h between them
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                         np.arange(y_min, y_max, h))
    # Predict the function value for the whole gid
    nn_model.test(np.c_[xx.ravel(), yy.ravel()])
    Z = nn_model.predict
    Z[Z>=0.5] = 1
    Z[Z<0.5] = 0
    Z = Z.reshape(xx.shape)
    # Plot the contour and training examples
    plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)
    plt.scatter(X[:, 0], X[:, 1], s=40,  c=y, cmap=plt.cm.BuGn)_____no_output_____plot_decision_boundary(ann)
plt.title("Our initial model")_____no_output_____

**Exercise**:

Create Neural networks with 10 hidden nodes on the above code.

What's the impact on accuracy?_____no_output_____

# Put your code here
#(or load the solution if you wanna cheat :-)_____no_output_____# %load ../solutions/sol_111.py
_____no_output_____

**Exercise:**

Train the neural networks by increasing the epochs.

What's the impact on accuracy?_____no_output_____

#Put your code here_____no_output_____# %load ../solutions/sol_112.py_____no_output_____

# Addendum

There is an additional notebook in the repo, i.e. [MLP and MNIST](1.1.2 MLP and MNIST.ipynb) for a more complete (but still *naive* implementation) of **SGD** and **MLP** applied on **MNIST** dataset.

Another terrific reference to start is the online book http://neuralnetworksanddeeplearning.com/. Highly recommended!  _____no_output_____


# Notebook from hello-deepak/sat-ml-training
Path: _notebooks/2020-02-21-IntroPython.ipynb

# Intro to Python
> A selection of background material and resources for Python.

- toc: true
- badges: true
- author: Alex Mandel
- comments: false
- categories: [python, background]
- hide: false
- sticky_rank: 1_____no_output_____## General python

In this section we review the fundamentals of using the Python programming language with a focus on being able to use python for geospatial data science with machine learning. Based on the list of goals, and your prior experience select from the list of tutorials. Feel free to do as many as you like, and skim lightly if you are already familiar with the topics.

For local practice you can install and user the [Python with Conda](https://docs.anaconda.com/anaconda/user-guide/getting-started/), we recommend coding in a Jupyter Notebook which is similar to the Google Colab online editor that will be used in the other tutorials on this site.

### Goals

-   Data types

    -   Basic Types: Integers (Whole numbers), Floats (Numbers with
        decimals), Strings
    -   Constructed Types: Lists, Dictionaries

        >   Note: Python Dictionaries and JSON files very similar.

-   Looping (Program flow control)

    -   Using looping to iterate over a set of items to repeat a
        process

        -   Example: Loop over a list of items

        -   Example: Loop over list of files

-   Calling functions

    -   Importing libraries (aka modules, packages)
    -   How to call a function
    -   How to write a small function

### Lessons

Pick from the following lessons. We recommend at least 1, and if you choose to watch the videos then we suggest you also do some of the Interactive coding examples from the other options. Trying some code is one of the best ways to learn.

1.  An introduction or refresher for Python 3 Online Videos from [Udemy](https://www.udemy.com/course/pythonforbeginnersintro/)
2.  Interactive lessons with examples and quizzes from [CodeAcademy](https://www.codecademy.com/learn/learn-python-3). All lessons except \#10 on Classes (optional)
3.  [Demonstration Notebooks](https://github.com/jakevdp/WhirlwindTourOfPython) that  Accompany the Whirlwind Tour Of Python Book (Available for free). Lessons 1-9, everything else is optional.

### Additional References

-   Python Documentation
     [https://docs.python.org/3/tutorial/index.html](https://docs.python.org/3/tutorial/index.html)

-   More tutorials
     [https://docs.python-guide.org/intro/learning/](https://docs.python-guide.org/intro/learning/)_____no_output_____## Python for Data Science

In this section we specifically review Python for Data Science, focusing on the main data structures used to manipulate data, numpy arrays and pandas dataframes. Each link is a tutorial or example that you can try.

### Numpy Arrays

> Important: Understanding multidimensional data arrays as a data structure.

> Note: Numpy array data must all be of the same type (e.g. Integer, Float, etc…)

#### Lessons:

-   [Introduction to NumPy](https://jakevdp.github.io/PythonDataScienceHandbook/02.00-introduction-to-numpy.html)

-   [Understanding Data Types in Python](https://jakevdp.github.io/PythonDataScienceHandbook/02.01-understanding-data-types.html)

-   [The Basics of NumPy Arrays](https://jakevdp.github.io/PythonDataScienceHandbook/02.02-the-basics-of-numpy-arrays.html)

-   [Computation on NumPy Arrays: Universal Functions](https://jakevdp.github.io/PythonDataScienceHandbook/02.03-computation-on-arrays-ufuncs.html)

-   [Aggregations: Min, Max, and Everything In Between](https://jakevdp.github.io/PythonDataScienceHandbook/02.04-computation-on-arrays-aggregates.html)

### Pandas DataFrames

> Important: Understanding data frames as a table (sheet), with data
records as rows, and data attributes as columns. Each column
can be a different data type.

#### Lessons:

-  [Introduction](https://pandas.pydata.org/docs/getting_started/intro_tutorials/01_table_oriented.html)

-  Importing and exporting
     [CSV](https://pythonbasics.org/read-csv-with-pandas/) and
     [JSON](https://pythonbasics.org/pandas-json/) data sources

-  [Summarizing (aggregating)
     data](https://jakevdp.github.io/PythonDataScienceHandbook/03.08-aggregation-and-grouping.html)

-  [Selecting relevant data records](https://jakevdp.github.io/PythonDataScienceHandbook/03.02-data-indexing-and-selection.html#Data-Selection-in-DataFrame)

- [All in one](https://datacarpentry.org/python-ecology-lesson/02-starting-with-data/index.html)

-   (Optional) [Further Reading and Materials](https://jakevdp.github.io/PythonDataScienceHandbook/03.01-introducing-pandas-objects.html)

### References

- Primary Source
[PythonDataScienceHandbook](https://jakevdp.github.io/PythonDataScienceHandbook/index.html)
- Secondary Source
[Datacarpentry python-ecology-lesson](https://datacarpentry.org/python-ecology-lesson/)_____no_output_____## Plotting

> Important: Being able to visualize selections of data in a variety of standard plot types. Histogram, Bar, Line, Scatter Plot (XY)

- [Visualization with Matplotlib](https://jakevdp.github.io/PythonDataScienceHandbook/04.00-introduction-to-matplotlib.html)

-   [Simple Line Plots](https://jakevdp.github.io/PythonDataScienceHandbook/04.01-simple-line-plots.html)

-   [Scatter Plots](https://jakevdp.github.io/PythonDataScienceHandbook/04.02-simple-scatter-plots.html)

-   [Density and Contour Plots](https://jakevdp.github.io/PythonDataScienceHandbook/04.04-density-and-contour-plots.html)

-   [Histograms, Binnings, and Density](https://jakevdp.github.io/PythonDataScienceHandbook/04.05-histograms-and-binnings.html)_____no_output_____


# Notebook from yes-G/FussballPrognose
Path: LeaguePrognosis_Public.ipynb

_____no_output_____#Poor start! Relegation ahead?

Abysmal performance on the professional playground has different consequences for different people in different sports. One common consequence in football for ardent fans, employees and wannabe punters is the total and utter loss of peace of mind. I too found myself in such a situation with my *favorite club*, and here i describe what i did to reclaim that much treasured peace of mind.

#Disclaimer
My affiliation with *my favourite* club is a personal matter. The club *analysed* here is a random club lounging in the bottom half of an unrelated(?) league and is NOT *my favourite* club._____no_output_____

import numpy as np
import pandas as pd

import requests
import bs4

import matplotlib.pyplot as plt_____no_output_____#Configuration

#Football league: Spanish 1st division a.k.a. La Liga
League = 'esp-primera-division'

#A proxy for my favourite by abysmally performing team
team = 'Getafe CF'_____no_output_____def split(score):
  """The scores are reported by websites in the following format 7:2 (3:0),
  where 7:2 is the full time score and (3:0) is the score at half time
  this function returns only the full time score as a string"""
  return score.split('(')[0].strip()

def points(score, invert=False, winPoints=3, drawPoints=1, losePoints=0):
  """How many points did this match give my team?
  By default, 3 points for a win,
  1 for a draw, and 0 for a loss

  Remember to use invert=True, it is an away game!"""
  x = score.split(':')
  pts = losePoints
  try:
    if invert:
      a = int(x[1])
      b = int(x[0])
    else:
      a = int(x[0])
      b = int(x[1])
    if a > b:
      pts = winPoints
    if a == b:
      pts = drawPoints
  except:
    pts = None
  return pts
  print(x)_____no_output_____def ExtractMatchResults(season='2020'):
  s1 = season
  s2 = str(int(season)+1)

  #get the HTML page containing the match results for the season
  r = requests.get(f'https://www.weltfussball.com/alle_spiele/{League}-{s1}-{s2}/')
  html = r.text

  # create a BeautifulSoup object
  bs = bs4.BeautifulSoup(html, "html.parser")

  dfScores = pd.read_html(str(bs.find(class_="standard_tabelle")))[0]
  dfScores.columns = ['Date', 'Time', 'Home', 'Q1', 'Away', 'Score', 'Q2']
  dfScores.Date.fillna(method='ffill', inplace=True)
  dfScores.Home = dfScores.Home.apply(lambda x: None if 'Spieltag' in x else x)
  dfScores.drop(['Q1', 'Q2', 'Time'], axis=1, inplace=True)
  dfScores.dropna(inplace=True)
  dfScores['FT_Score'] = dfScores.Score.apply(split)
  dfScores['FT_Score'] = dfScores['FT_Score'].apply(lambda x: '-:-' if 'verl.' in x else x)
  dfScores.drop(['Score'], axis=1, inplace=True)
  dfScores['Season']=season
  return dfScores
  #dfScores.head(10)_____no_output_____def PrepareFinalPoints(dfMatches):
  year = dfMatches.Season.value_counts().index[0]
  finalTable = pd.DataFrame(data={f'Team-{year}':[], 'Score':[]})
  for name in dfMatches.Home.value_counts().index:
    dfH = dfMatches[(dfMatches.Home == name)].copy()
    #dfH['Home']='H'
    dfH['Points']=dfH.FT_Score.apply(points)

    dfA = dfMatches[(dfMatches.Away == name)].copy()
    #dfA['Home']='A'
    dfA['Points']=dfA.FT_Score.apply(points, invert=True)

    df = pd.concat([dfA, dfH]).sort_index()
    score = df.Points.sum()
    finalTable=finalTable.append({f'Team-{year}':name, 'Score':score}, ignore_index=True)
  finalTable.sort_values(by='Score', ascending=False, inplace=True)
  return finalTable.reset_index(drop=True)_____no_output_____#Extract all the match results for the year 2020-21
results2020 = ExtractMatchResults('2020')

#Extract all the match results for the year 2019-20
results2019 = ExtractMatchResults('2019')

#Extract all the match results for the year 2021-22
results2021 = ExtractMatchResults('2021')_____no_output_____#Prepare the final points table for the season 2020-21
table2020 = PrepareFinalPoints(results2020)

#Prepare the final points table for the season 2020-21
table2019 = PrepareFinalPoints(results2019)

#display(table2020)

#display(table2019)_____no_output_____def ExtractTeamPeformance(team, results):
  df1 = results[(results.Home ==team) | (results.Away == team)].copy().reset_index()
  df1['Opponent'] = df1.apply(lambda x: x['Away'] if x['Home'] == team else x['Home'], axis=1)
  df1['Home'] = df1.apply(lambda x: 'H' if x['Home'] == team else 'A', axis=1)
  df1['Points'] = df1.apply(lambda x: points(x['FT_Score']) if x['Home']=='H' else points(x['FT_Score'], True), axis=1)
  df1.drop(['Away', 'index'], inplace=True, axis=1)
  return df1_____no_output_____try:
  df2019 = ExtractTeamPeformance(team, results2019)
except:
  df2019 = pd.DataFrame()

df2020 = ExtractTeamPeformance(team, results2020)
df2021 = ExtractTeamPeformance(team, results2021)_____no_output_____

#Background info.

*Getafe has lost every single match thusfar,* thereby accumulating a grand total of 0 points from 7 matches, i.e. in 2021-22 the team has displayed a very strong attraction to the very bottom of the abyss.

How did the team perform in the two preceeding seasons?  Or where were we now (after 7 matches) last season and in the one before it?_____no_output_____

plt.figure(figsize=(15,7));
NumberOfMatches = len(df2021.Opponent.values)
x = np.arange(1, NumberOfMatches+1)
if df2019.empty == False:
  plt.plot(x, df2019.Points.cumsum(), 'r', marker='o', label='2019-20');
plt.plot(x, df2020.Points.cumsum(), 'g', marker='o', label='2020-21');
plt.plot(x, df2021.Points.cumsum(), 'b', marker='o', label='2021-22');
plt.ylabel('Points')
plt.title(f'Points accumulated by {team} over the course of a season')
plt.xlabel('Match days')
plt.grid(color='gainsboro', linestyle='--', linewidth=0.7)
plt.xticks(ticks=x)
plt.legend();_____no_output_____

The above the plot answers exactly that question. The match days are on the X-axis and the Y-axis reflects the number of points accumulated until that match day. It is so very clear that on the 7th match the team had boarded the points train and bid its final good bye to the start line fading away in the horizon. In both the past seasons. In this season, it seems to be unclear if the train is scheduled to depart or has terminated at the start line. This chart is the root cause of that aforementioned loss of *peace of ze mind*. _____no_output_____#What does statistics say? - The independent 2-sample t-test to the rescue

The independent 2-sample t-test can be applied on the points scored in the current current season and the points scored in the "baseline" season. The null hypothesis of the test is that *the means of the two independent populations are equal*. i.e. the performance of the team in the two seasons is "similar".

If the p-value of the test is greater than the threshold (an arbitraty 5%) then the null hypothesis is not rejected. Food for thought - one can't simply replace *not rejected* with *accepted**. _____no_output_____

from scipy.stats import ttest_ind
a = df2021.Points.copy()
b = df2020.Points.copy()
n = len(a.dropna())
t, p = ttest_ind(a[:n], b[:n])

print(f"p-value is {p*100:.2}%")p-value is 2.9%

Since, the p-value is less than 5%, the alternative hypothesis (the population means are not equal) may not be rejected.

In english - there is a difference in the perfomance of the team between the two seasons!

The more i mulled and ruminated about it, the more i asked myself - *Are we where we are now, because we are facing the big **powerful teams back-to-back**? Is the result no more than an illusion?*_____no_output_____#The Schedule factor

To qualify the statement, i.e. we performing badly this season, because we are facing the challenging teams first, i tried to answer the question - **if the results are independent of the order of matches, then how would the points graph look like if last season's schedule were like this season's schedule?**

That Q doesn't have a straight forward answer because, 3 new contenders make an appearance in this season's schedule compared to last season. So, a little assumptions was made leading to two different results - and the reality would've been somewhere in between the two results.

The assumption -
1. the team loses all the matches played against all the new contenders; or
2. the team wins every single match against the new teams. _____no_output_____

def CompareWith(row, withDF, ignore=False):
  ret = row['Points']
  if (row['FT_Score']=='-:-') | (ignore == True):
    data = withDF[(withDF.Home==row['Home']) & (withDF.Opponent == row['Opponent'])]['Points']
    if len(data)>0:
      ret = data.values[0]
    else:
      ret = None
  return ret_____no_output_____if df2019.empty == False:
  df2021_19 = df2021.copy()
  df2021_19['Projection'] = df2021_19.apply(CompareWith, withDF=df2019, axis=1)
  df2021_19['ProjectionFull'] = df2021_19.apply(CompareWith, withDF=df2019, axis=1, ignore=True)

df2021_20 = df2021.copy()
df2021_20['Projection'] = df2021_20.apply(CompareWith, withDF=df2020, axis=1)
df2021_20['ProjectionFull'] = df2021_20.apply(CompareWith, withDF=df2020, axis=1, ignore=True)_____no_output_____def PlotProjection(dfSeasonA, yearA, dfSeasonB, yearB, relegationScoreSeasonB=0):
  """Plot the points projections for the schedule of yearA based on the scores for the sames matches from yearB"""

  NumberOfMatches = len(dfSeasonB.Opponent.values)
  x = np.arange(1, NumberOfMatches+1)

  plt.figure(figsize=(15,10));

  #plot the relegation line for Season B on the chart
  if relegationScoreSeasonB > 0:
    plt.hlines(relegationScoreSeasonB,0,NumberOfMatches,
               color='goldenrod', linestyle='--',
               label=f'{yearB} relegation level' )

  #plot the cumulative points for season B.
  plt.plot(x, dfSeasonB.Points.cumsum(),
           color='black', linewidth=3,
           label=f'{yearB} schedule and scores');

  #Plot the projetion since the begining of the season
  ProjFullX1 = dfSeasonA.ProjectionFull.fillna(0).cumsum()
  ProjFullX2 = dfSeasonA.ProjectionFull.fillna(3).cumsum()
  plt.plot(x, ProjFullX1.values,
           'salmon', linestyle='--', linewidth=2,
           label=f'{yearA} schedule, {yearB} scores since start of season');
  plt.plot(x, ProjFullX2.values, 'salmon', linestyle=':', linewidth=2);
  plt.fill_between(x, ProjFullX1.values, ProjFullX2.values, alpha=0.1, color='salmon')

  #Plot the projetion since the next match of the season
  ProjX1 = dfSeasonA.Projection.fillna(0).cumsum()
  ProjX2 = dfSeasonA.Projection.fillna(3).cumsum()
  plt.plot(x, ProjX1.values, 'b', linestyle='--', linewidth=2,
           label=f'{yearA} schedule, {yearB} scores from next match');
  plt.plot(x, ProjX2.values, 'b', linestyle=':', linewidth=2);
  plt.fill_between(x, ProjX1.values, ProjX2.values, alpha=0.1, color='blue')

  plt.plot(x, dfSeasonA.Points.cumsum(), 'green', label=f'{yearA} scores', marker='^', linewidth=3);
  plt.xlabel('Match day (Team played against)')
  plt.xticks(ticks=x, labels=dfSeasonA.Opponent.values, rotation=90)
  plt.ylabel('Cumulative points')
  #plt.vlines(len(dfSeasonA.Points.values) - dfSeasonA.Points.isna().sum() - 1 , 0, max(dfSeasonB.Points.cumsum()),color='green', linewidth=3)
  plt.legend();
  plt.title(f'{team}\'s {yearB} Performance in {yearA}\'s schedule ');
  plt.tight_layout()
  plt.grid(color='gainsboro', linestyle='--', linewidth=0.7)_____no_output_____PlotProjection(df2021_20,'2021',df2020,'2020',34)_____no_output_____

The answer to that question is found in the above chart. The dashed orange line would've been the worst case performance of the team in the last season if last season's schedule were like this season. And the dotted orange line in the best case perfomance. The orange area between them would've been the area of truth.

The actual points progression of the team in last year's season is the solid black line, and reminds us that not all promising starts are sustained through to the end. The key-take away for me is - last season's start would've been **dismal but not abysmal**, if the schedule were like this season._____no_output_____#Statistically there is hope!

Applying the same t-test as before to compare the 2021 season and the projection on the scores based on the 2020 season yields the following p-value._____no_output_____

b = df2021_20.ProjectionFull.fillna(3).copy()
a = df2021.Points.copy()
n = len(a.dropna())
t, p = ttest_ind(a[:n], b[:n])

print(f"p-value is {p*100:.3}%")p-value is 12.8%

Since it is greater than 5%. i.e. The null hypothesis can not be rejected, that the means of the points scored in each match of two seasons are the same. _____no_output_____If team's psychology is like that of an aspiring Engineer staring down the barrel of a final exam, then the burning question would be - I failed so far, but if i were to be as good as i can be, from **now** - then what?

*Win all the matches henceforth*? Nah, that is way too optimistic for an engineer.

*So, last year's performance from now?* Yeah. That sounds more realistic. Improbable, but realistic.

This performace is the light blue area between the dotted blue line representing the best case wins against the promoted teams the dashed blue line of the worst case against the same teams.

If this season's relegation level is going to be just like last season, then there is a very realistic chance that the team will relegate. Unless...

Unless -
- they pull off wins against every promoted team
- and win as many games hence forth as they did last season._____no_output_____

#if df2019.empty != True:
#  PlotProjection(df2021_19,'2021',df2019,'2019',36)_____no_output_____finalTable=pd.DataFrame(data={'Team':[], 'Points-Min':[], 'Points-Max':[]})

for team in results2021.Home.value_counts().index:
  try:
    df2020 = ExtractTeamPeformance(team, results2020)
    df2021 = ExtractTeamPeformance(team, results2021)

    df2021_20 = df2021.copy()
    df2021_20['Projection'] = df2021_20.apply(CompareWith, withDF=df2020, axis=1)

    #print(team, df2021_20.Projection.fillna(1).sum())

    finalTable=finalTable.append({
        'Team':team,
        'Points-Min':int(df2021_20.Projection.fillna(0).sum()),
        'Points-Max':int(df2021_20.Projection.fillna(3).sum())
    }, ignore_index=True)
  except:
        finalTable=finalTable.append({
        'Team':team,
        'Points-Min':int(0),
        'Points-Max':int(0)
    }, ignore_index=True)

finalTable=finalTable.sort_values(by='Points-Min', ascending=False).reset_index(drop=True)
finalTable.index = np.arange(1, len(finalTable)+1)
finalTable['Points-Max']=finalTable['Points-Max'].astype(int)
finalTable['Points-Min']=finalTable['Points-Min'].astype(int)
display(finalTable.iloc[:-3])_____no_output_____

In order to understand where the 2021-22 season's relegation cut off could be, a similar projection was performed for all the other teams in the league. The projected final scores are shown in the above table. The "promoted" teams have been given 0 points, although this is not accurate, because it is not possible to predict where they will land based on last season's performance. The relegation score will lie between 28 and 43 points. And as of now, the team is on track to score between 29 and 44 points, and hence definately **at a slight risk of relegation**, unless they improve **or others perform poorly**._____no_output_____


# Notebook from datalayer-contrib/ipywidgets-tutorial
Path: notebooks/07.More-libraries/07.03-ipydatagrid.ipynb

# `ipydatagrid`: fast, performant data grid

## https://github.com/bloomberg/ipydatagrid

`ipydatagrid` is a complete data grid solution providing full integration with pandas DataFrames without compromising on performance:

- built-in sorting and filtering
- full integration with the jupyter-widgets ecosystem
- highly customisable renderers
- complete two-way data binding between Python and the front-end
- supports Vega expressions for conditional formatting and styling

**Installation**:
```bash
conda install -c conda-forge ipydatagrid
```_____no_output_____

from ipydatagrid import DataGrid, TextRenderer, BarRenderer, Expr, VegaExpr
import pandas as pd
import numpy as np
import requests_____no_output_____

#### Use any dataset_____no_output_____

# Numpy/pandas
np.random.seed(0)
p_t, n = 100, 260
stock_df = pd.DataFrame(
    {f'Stock {i}': p_t + np.round(np.random.standard_normal(n).cumsum(), 2) for i in range(10)}
)

# Requests/JSON
req = requests.get("https://raw.githubusercontent.com/bloomberg/ipydatagrid/main/examples/cars.json")
data = req.json()
cars_data = data['data']

# Random matrix
rand_df = pd.DataFrame(
    {f'Column {col}': np.round(np.random.random(100), 2) for col in [chr(n) for n in range(65, 91)]}
)

# Small grid
small_df = pd.DataFrame(
    np.eye(10),
    columns=[f'Col {i}' for i in range(10)],
    index=[f'Row {i}' for i in range(10)]
)

# Multi-index
top_level = ['Value Factors', 'Value Factors', 'Momentum Factors', 'Momentum Factors']
bottom_level = ['Factor A', 'Factor B', 'Factor C', 'Factor D']

nested_df = pd.DataFrame(np.random.randn(4,4).round(2),
                         columns=pd.MultiIndex.from_arrays([top_level, bottom_level]),
                         index=pd.Index(['Security {}'.format(x) for x in ['A', 'B', 'C', 'D']], name='Ticker'))_____no_output_____

#### Convert your pandas DataFrame to a datagrid - it 'just works'_____no_output_____

df = pd.DataFrame(cars_data).set_index('index')
df.head()_____no_output_____grid = DataGrid(df)
grid_____no_output_____

Additional constructor options_____no_output_____

grid = DataGrid(
    dataframe=df,
    base_row_size=30,
    base_column_size=92,
    base_row_header_size=128,
    base_column_header_size=40
)
grid_____no_output_____grid.header_visibility = 'row'_____no_output_____grid.header_visibility = 'column'_____no_output_____grid.header_visibility = 'none'_____no_output_____grid.header_visibility = 'all'_____no_output_____

#### Style your data grid with custom renderers based on Vega expressions

- Custom cell renderers can be defined for the entire grid or column-wise.

- Two types of cell renderers are currently available: `TextRenderer` and `BarRenderer`.

- Most of the TextRenderer/BarRenderer attributes (`background_color`, `text_color` etc.) can either be a `value`, a `bqplot` scale or a `VegaExpr` or `Expr` instance.

The `VegaExpr` class allows you to define an attribute value as a result of a Vega-expression (see https://vega.github.io/vega/docs/expressions/):
```python
background_color = VegaExpr("value < 150 ? 'red' : 'green'").
```
You can look at the vega-expression documentation for more information about available constants and functions. In the scope of the expression are also available: value: cell value, x and y: cell position in pixel, width and height of the cell, row and column: cell position._____no_output_____

from bqplot import LinearScale, ColorScale, OrdinalColorScale, OrdinalScale
from py2vega.functions.color import rgb

def horsepower_coloring(cell):
    if cell.value < 100:
        return "red"
    elif cell.value < 150:
        return "orange"
    else:
        return "green"

def weight_coloring(cell):
    scaled_value = 1 if cell.value > 4500 else cell.value / 4500
    color_value = scaled_value * 255

    return rgb(color_value, 0, 0)

renderers = {
    "Acceleration": BarRenderer(
        horizontal_alignment="center",
        bar_color=ColorScale(min=0, max=20, scheme="viridis"),
        bar_value=LinearScale(min=0, max=20),
    ),
    "Cylinders": TextRenderer(
        background_color=Expr('"grey" if cell.row % 2 else default_value')
    ),
    "Displacement": TextRenderer(
        text_color=ColorScale(min=97, max=455),
        font=Expr(
            "'16px sans-serif' if cell.value > 400 else '12px sans-serif'"
        ),
    ),
    "Horsepower": TextRenderer(
        text_color="black", background_color=Expr(horsepower_coloring)
    ),
    "Miles_per_Gallon": TextRenderer(
        background_color=Expr('"grey" if cell.value is None else default_value')
    ),
    "Name": TextRenderer(
        background_color=Expr(
            'rgb(0, 100, 255) if "chevrolet" in cell.value or "ford" in cell.value else default_value'
        )
    ),
    "Origin": TextRenderer(
        text_color="black",
        background_color=OrdinalColorScale(domain=["USA", "Japan", "Europe"]),
        horizontal_alignment=Expr(
            "'right' if cell.value in ['USA', 'Japan'] else 'left'"
        ),
    ),
    "Weight_in_lbs": TextRenderer(
        text_color="black", background_color=Expr(weight_coloring)
    ),
    "Year": TextRenderer(text_color="black", background_color="green"),
}

grid.renderers = renderers_____no_output_____renderers["Name"] \
    .background_color.value = '"green" if "pontiac" in cell.value or "citroen" in cell.value else default_value'_____no_output_____renderers["Year"].background_color = "yellow"_____no_output_____

#### Apply styling to column and row headers_____no_output_____

grid.header_renderer = TextRenderer(
    background_color=Expr('"salmon" if cell.value == "Horsepower" else "skyblue"'),
    font='italic small-caps bold 12px/30px Georgia, serif'
)

renderers['index'] = TextRenderer(background_color='slateblue')
grid.renderers = renderers_____no_output_____

#### Built-in sorting and filtering functionality which you can trigger from both Python and directly via the GUI._____no_output_____

# Filtering based on car origin and sorting based on displacement
grid.transform(
    [
        {
            "type": "filter",
            "operator": "=",
            "columnIndex": 2,
            "value": "Europe",
        },
        {"type": "sort", "columnIndex": 9, "desc": True},
    ]
)_____no_output_____

#### Conditional formatting based on another cell_____no_output_____

def format_based_on_other_column(cell):
    return ("green" if cell.column == 2 and cell.metadata.data["Return"] > 0 else "red")

signal_column_formatting = TextRenderer(
    text_color="white",
    background_color=Expr(format_based_on_other_column),
)

renderers = {
    "Signal": signal_column_formatting,
    "Return": TextRenderer(background_color='seashell',
                           text_color=VegaExpr('cell.value > 0 ? "green" : "firebrick"')
    )
}

conditional_grid = DataGrid(
    pd.DataFrame(
        {"Stock": "A B C D".split(),
         "Return": [0.11, -0.05, 0.08, -0.20],
         "Signal": ["Buy", "Sell", "Buy", "Sell"]}
    ),
    column_widths={"Stock": 64, "Return": 64,  "Signal": 300},
    base_row_size=30,
    renderers=renderers,
    layout={"height": "150px"},
)

conditional_grid_____no_output_____

#### Multi-index and nested columns DataFrames are also supported_____no_output_____

nested_df_____no_output_____columns_renderer = TextRenderer(
    background_color='dimgray',
    horizontal_alignment='center')

renderers= {
    "('Ticker', '')": TextRenderer(background_color='dimgray')
}

default_renderer = TextRenderer(
    background_color=VegaExpr('cell.value > 0 ? "steelblue" : "seagreen"')
)

nested_grid = DataGrid(nested_df,
                       base_column_size=90,
                       column_widths={"('Ticker', '')": 80},
                       layout={'height':'140px'},
                       renderers=renderers,
                       default_renderer=default_renderer,
                       header_renderer=columns_renderer)

nested_grid_____no_output_____

#### Two-way selections model at your disposal

DataGrid cells can be selected using mouse by simply clicking and dragging over the cells. Pressing Cmd / Ctrl key during selection will add to existing selections. Pressing Shift key allows selection of regions between two clicks.

DataGrid supports three modes of selection `cell`, `row`, `column`. In order to disable selections, selection mode can be set to `none` which is the default setting.

Selection Modes:

- `cell`: Clicking on grid will select only the cell under mouse cursor
- `row`: Clicking on grid will select all the cells on the row under mouse cursor
- `column`: Clicking on grid will select all the cells on the column under mouse cursor

You can clear all selections by hitting the `esc` key._____no_output_____

sel_grid = DataGrid(stock_df, selection_mode='cell')
sel_grid_____no_output_____

Select from the UI and retrieve on the Python side_____no_output_____

sel_grid.selections_____no_output_____

We can also access the selected cell values_____no_output_____

sel_grid.selected_cell_values_____no_output_____

Select from the Python side and see selections highlighted on the UI.

Parameters:

- `row1`: start row (starts from 0).
- `row2`: end row (starts from 0).
- `column1`: start column.
- `column2`: end column.

We can automatically clear any existing selections by passing a value for `clear_mode`:

- `current`: clear last selection
- `all`: clear all existing selections
- `none`: do not clear selections (default)_____no_output_____

# Select top left corner of grid
sel_grid.select(row1=0, column1=0, row2=1, column2=1, clear_mode='current')_____no_output_____# Clear selection
sel_grid.clear_selection()_____no_output_____

We can select individual sells by omitting `row2` and `column2`_____no_output_____

sel_grid.select(10, 5)_____no_output_____

When working with large grids, we can opt to use the `selected_cell_iterator`. It will yield values for each loop iteration, avoiding the need to store all selections in a list, in advance._____no_output_____

for cell in sel_grid.selected_cell_iterator:
    print(f'Cell value: {cell}')_____no_output_____

We can modify selections in place by passing a list of selections_____no_output_____

sel_grid.selections = [
    {"r1": 22, "r2": 20, "c1": 0, "c2": 2},
    {"r1": 6, "r2": 6, "c1": 2, "c2": 2},
    {"r1": 10, "r2": 10, "c1": 3, "c2": 3},
    {"r1": 13, "r2": 13, "c1": 2, "c2": 2},
]_____no_output_____

Row selection mode_____no_output_____

sel_grid.selection_mode='row'_____no_output_____for i in range(260):
    sel_grid.select(i, i) if i % 2 == 0 else None_____no_output_____

#### Two-way cell editing is possible

Just pass `editable=True` to the grid's constructorand you're good to go (grids are not editable by default)._____no_output_____

small_grid = DataGrid(small_df,
                      editable=True,
                      default_renderer=TextRenderer(
                          background_color=VegaExpr("cell.value === 1 ? 'limegreen' : 'hotpink'")
                      ),
                      layout={'height': '250px'})
small_grid_____no_output_____

You can change values directly in the UI by double clicking a cell and changing the value. You can navigate the grid using the keyboard. You can use the arrow keys or the grid's __cursor__:

- __Down__: Enter
- __Up__: Shift + Enter
- __Right__: Tab
- __Left__: Shift + Tab_____no_output_____..or you can change it directly from the Python side_____no_output_____

# Returns a boolean to indicate whether operations was successful
small_grid.set_cell_value('Col 0', 'Row 9', 1) # Sets value based on row name_____no_output_____small_grid.set_cell_value_by_index('Col 9', 0, 1) # Sets value based on row index_____no_output_____

#### Events and integration with `ipywidgets`_____no_output_____Listen to cell change events_____no_output_____

def cell_changed(e):
    row, column, col_index, value = e['row'], e['column'], e['column_index'], e['value']
    print(f'The cell at row {row}, column "{column}" (index {col_index}), changed to {value}')

small_grid.on_cell_change(cell_changed)
small_grid_____no_output_____

#### An example with the BarRenderer

Renders cell values as horizontal bars based on a scale. `ipydatagrid` has two renderers - `TextRenderer`, which is the default one we've seen, and `BarRenderer`, which we will use now._____no_output_____

from bqplot import LinearScale, ColorScale
from ipydatagrid import DataGrid, BarRenderer

linear_scale = LinearScale(min=0, max=1)
color_scale = ColorScale(min=0, max=1)
bar_renderer = BarRenderer(
    bar_color=color_scale,
    bar_value=linear_scale,
    bar_horizontal_alignment="center",
)

rand_grid = DataGrid(rand_df, default_renderer=bar_renderer, base_column_size=76)
rand_grid_____no_output_____bar_renderer.show_text = False_____no_output_____from ipywidgets import FloatSlider, link

slider = FloatSlider(
    description="Scale: ", value=linear_scale.max, min=0, max=0.99, step=0.01
)
link((color_scale, "min"), (slider, "value"))
link((linear_scale, "min"), (slider, "value"))

slider_____no_output_____color_scale.min_____no_output_____

#### Integration with `bqplot`

We have a DataGrid with time series of 10 different stock prices. Each time we click on a stock price column, we want to plot the time series of that stock price in a line chart._____no_output_____

stock_df.head()_____no_output_____from bqplot import LinearScale, Axis, Figure, Lines, CATEGORY10
from ipywidgets import HBox, Layout

# Setting up the data grid
stock_grid = DataGrid(stock_df, selection_mode='column')

# Creating the bqplot chart objects
sc_x = LinearScale()
sc_y = LinearScale()
line = Lines(x=[], y=[], labels=['Fake stock price'], display_legend=True,
                 scales={'x': sc_x, 'y': sc_y})
ax_x = Axis(scale=sc_x, label='Index')
ax_y = Axis(scale=sc_y, orientation='vertical', label='y-value')
fig = Figure(marks=[line], axes=[ax_x, ax_y], title='Line Chart', layout=Layout(flex='1 1 auto', width='100%'))_____no_output_____def plot_stock(*args):
    line.y = stock_grid.selected_cell_values
    line.x = range(len(line.y))
    column_index = stock_grid.selections[0]['c1']
    line.labels = [stock_df.columns[column_index]]
    line.colors = [CATEGORY10[np.random.randint(0, len(CATEGORY10)) % len(CATEGORY10)]]

# Event listener for cell click
stock_grid.observe(plot_stock, names='selections')_____no_output_____HBox(
    [stock_grid, fig]
)_____no_output_____