# Notebook from PavanK8/DataQuest-Guided-Projects
Path: 10_Answering_Business_Questions_Using_SQL.ipynb

# Project: Using SQL to analyse data about sales and answer business questions
In this project, we will be using SQL to answer business questions. We will work with [Chinook database](https://github.com/lerocha/chinook-database) which is provided as a SQLite database `chinook.db`

`Chinook` database contains many tables. All the tables and the relations between them of the `Chinook` schema is in the below picture
![Chinook_schema](https://s3.amazonaws.com/dq-content/189/chinook-schema.svg)
## 1. Connecting Jupyter Notebook to the database file_____no_output_____

%%capture
%load_ext sql
%sql sqlite:///chinook.db_____no_output_____

## 2. Overview of the data_____no_output_____

%%sql
SELECT
    name,
    type
FROM sqlite_master
WHERE type IN ("table", "view")Done.

As pointed out in the schema summary figure, the database contains 11 tables._____no_output_____## 3. Making a business decision on which albums to purchase

**Background**:
The Chinook record store has just signed a deal with a new record label, and you've been tasked with selecting the first three albums that will be added to the store, from a list of four. All four albums are by artists that don't have any tracks in the store right now - we have the artist names, and the genre of music they produce:

| Artist Name | Genre |
| ----------- | ------ |
| Regal | Hip-Hop |
| Red Tone | Punk |
| Meteor and the Girls | Pop |
| Slim Jim Bites | Blues |

The record label specializes in artists from the USA, and they have given Chinook some money to advertise the new albums in the USA, so we're interested in finding out which genres sell the best in the USA.

** We have to make a recommendation for the three artists whose albums we should purchase for the store **

We will start looking at which genres sell the most in USA. For this we have to answer two questions. Among the users from USA,
- How many tracks of each genre has been sold ?
- What are the top genres which made more money ?_____no_output_____

%%sql

DROP VIEW IF EXISTS tr_gn_info;
DROP VIEW IF EXISTS lk_cst_tr;

/* view for getting track and genre info */
CREATE VIEW tr_gn_info AS
    SELECT tr.track_id, tr.genre_id, gn.name, tr.unit_price
    FROM track tr
    INNER JOIN genre gn ON tr.genre_id = gn.genre_id;

/* view for linking customer and track tables */
CREATE VIEW lk_cst_tr AS
    SELECT cst.customer_id, tr.track_id
    FROM customer cst
    INNER JOIN invoice inv ON cst.customer_id = inv.customer_id
    INNER JOIN invoice_line il ON inv.invoice_id = il.invoice_id
    INNER JOIN track tr ON il.track_id = tr.track_id;

WITH
    /* selecting customers from USA */
    cst_usa AS
    (
        SELECT customer_id, country
        FROM customer
        WHERE country = 'USA'
    )

SELECT
    tr_gn_info.name Genre,
    COUNT(tr_gn_info.track_id) no_tracks_purchased,
    ROUND(SUM(tr_gn_info.unit_price), 2) total_money_sales
FROM cst_usa
INNER JOIN lk_cst_tr ON cst_usa.customer_id = lk_cst_tr.customer_id
INNER JOIN tr_gn_info ON lk_cst_tr.track_id = tr_gn_info.track_id
GROUP BY tr_gn_info.name
ORDER BY 2 DESC;Done.
Done.
Done.
Done.
Done.

### Observations:
The above table suggests total money generated is proportional to the number of tracks sold to the customers from `USA`.

Hence for the options available to our company, following can be tabulated.

| Artist Name | `Genre` | Tracks sold in `Genre` |
| ----------- | ------ | ----------------------------- |
| Regal | Hip-Hop | 20 |
| Red Tone | Punk | 130 |
| Meteor and the Girls | Pop | 22 |
| Slim Jim Bites | Blues | 36 |
### Recommendation:
Based on the available data, it is advisable to go with the following 3 artists
- Red Tone (Punk)
- Slim Jim Bites (Blues)
- Meteor and the Girls (Pop)_____no_output_____## 4. Identifying star employees and their characteristics
Each customer for the `Chinook` store gets assigned to a sales support agent within the company when they first make a purchase. We will analyze the purchases of customers belonging to each employee to see if any sales support agent is performing either better or worse than the others._____no_output_____

%%sql

/* selecting employees-customer relationship */
WITH
    emp_cst AS
    (
        SELECT
            emp.first_name || ' ' || emp.last_name employee_name,
            emp.employee_id,
            emp.title,
            emp.reports_to,
            emp.birthdate,
            emp.hire_date,
            emp.country,
            cst.customer_id
        FROM employee emp
        LEFT JOIN customer cst ON emp.employee_id = cst.support_rep_id
    )

SELECT
    ROUND(SUM(tr_gn_info.unit_price), 2) total_money_sales,
    emp_cst.*
FROM emp_cst
    LEFT JOIN lk_cst_tr ON emp_cst.customer_id = lk_cst_tr.customer_id
    LEFT JOIN tr_gn_info ON lk_cst_tr.track_id = tr_gn_info.track_id
GROUP BY emp_cst.employee_id
ORDER BY 1 DESC;Done.
%%sql
SELECT invoice_date
FROM invoice
ORDER BY 1 DESC
LIMIT 10;Done.

### Observations:
- The data shows only 8 employees in total for the company.
- Of those 8, only 3 are Sales Agents and they all 3 report to Sales Manager.
- The last sale is in December 2020 according to the data and all the three Sales agents have been hired in 2017, albeit at various times.
- So, even if we take average amount of sales carried out per month, both the ladies score higher than `Steve`, although not by much.
- Ms. `Jane Peacock` is clearly the star employee of the sales department.

| Employee | Months in service | Total amt sales | Avg amt sales/month |
| ----- | ---- | ----| --- |
| Jane Peacock | 44 | 1733 | 39.38 |
| Margaret Park | 43 | 1585 | 36.86 |
| Steve Johnson | 38 | 1393 | 36.65 |_____no_output_____## 5. Analysing sales by country
Our next task is to analyze the sales data for customers from each different country. We will use the `country` value from the customers table, and ignore the `country` from the billing address in the invoice table.

For each country, in the database, we will calculate
- total number of customers
- total value of sales
- average value of sales per customer
- average order value_____no_output_____

%%sql

/* linking customer and invoice tables */
WITH cst_inv AS
(
    SELECT
        CASE
            WHEN
                (
                    SELECT COUNT(*)
                    FROM customer
                    WHERE country = cst.country
                ) = 1 THEN 'Others'
            ELSE cst.country
        END AS country_col,
        cst.customer_id,
        SUM(inv.total) total,
        COUNT(inv.invoice_id) no_orders
    FROM customer cst
    INNER JOIN invoice inv ON cst.customer_id = inv.customer_id
    GROUP BY cst.customer_id
)

SELECT
    country_col,
    COUNT(customer_id) total_no_customers,
    ROUND(SUM(CAST(total AS FLOAT)), 2) total_sales_USD,
    ROUND(SUM(CAST(total AS FLOAT)) / COUNT(customer_id), 2) avg_sales_customer,
    ROUND(SUM(CAST(total AS FLOAT)) / SUM(no_orders), 2) avg_order_val/* average order value = total sales / no of orders */
FROM
    (
        SELECT *,
            CASE
                WHEN country_col = 'Others' THEN 1
                ELSE 0
            END AS sort
        FROM cst_inv
    )
GROUP BY country_col
ORDER BY sort, 2 DESC;
    Done.

Everything looks good, except that there are many countries with only a single customer. It would be be good to
- Club all the countries with only a single customer into `Others` category
- This `Others` category should be listed at the last of the table_____no_output_____### Observations:
- By far, USA generates the bulk of sales followed by Canada and Brazil.
- It is interesting to know that countries of UK, Czech Republic and India have higher average sales / customer and average order values. However, it has to be noted that there are very few samples for these countries to formulate any kind of hypotheses.
- If we have some money, small surveys could be carried out in these countries to generate more data to make an informed decision_____no_output_____## 6. Business decision to change purchase strategy : Albums vs Individual Tracks
The Chinook store is setup in a way that allows customer to make purchases in one of the two ways:
- purchase a whole album
- purchase a collection of one or more individual tracks.

The store does not let customers purchase a whole album, and then add individual tracks to that same purchase (unless they do that by choosing each track manually). When customers purchase albums they are charged the same price as if they had purchased each of those tracks separately.

**Possible business strategy**:

Management are currently considering changing their purchasing strategy to save money. The strategy they are considering is to purchase only the most popular tracks from each album from record companies, instead of purchasing every track from an album.

We are asked to give our recommendation.

**Assumption**:

It is very common when we are performing an analysis to have 'edge cases' which prevent us from getting a 100% accurate answer to the question. In this instance, we have two edge cases to consider:
- Albums that have only one or two tracks are likely to be purchased by customers as part of a collection of individual tracks.
- Customers may decide to manually select every track from an album, and then add a few individual tracks from other albums to their purchase.

In the first case, since our analysis is concerned with maximizing revenue we can safely ignore albums consisting of only a few tracks. The company has previously done analysis to confirm that the second case does not happen often, so we can ignore this case also.
_____no_output_____

%%sql
WITH invoice_track_info AS
(
SELECT invoice_id, MIN(track_id) first_track_id
FROM invoice_line
GROUP BY 1
),
inv_album_or_not_table AS
(
    SELECT
    in_tr.invoice_id,
        CASE
            WHEN
                (
                    SELECT
                        track_id
                    FROM (SELECT tr.track_id, tr.album_id
                            FROM track tr
                            WHERE tr.album_id = (
                                                    SELECT album_id
                                                    FROM track
                                                    WHERE track_id = in_tr.first_track_id
                                                )
                         )
                    EXCEPT
                    SELECT track_id
                    FROM (
                            SELECT il.invoice_id, il.track_id
                            FROM invoice_line il
                            WHERE il.invoice_id = in_tr.invoice_id
                          )
                ) IS NULL
                AND
                (
                    SELECT track_id
                    FROM (
                            SELECT il.invoice_id, il.track_id
                            FROM invoice_line il
                            WHERE il.invoice_id = in_tr.invoice_id
                        )
                    EXCEPT
                    SELECT
                        track_id
                    FROM (SELECT tr.track_id, tr.album_id
                            FROM track tr
                            WHERE tr.album_id = (
                                                    SELECT album_id
                                                    FROM track
                                                    WHERE track_id = in_tr.first_track_id
                                                )
                         )
                ) IS NULL
                THEN "Yes"
            ELSE "No"
        END AS invoice_album_or_not

    FROM invoice_track_info AS in_tr
)

SELECT
    invoice_album_or_not,
    count(invoice_album_or_not) tot_number,
    ROUND(CAST(count(invoice_album_or_not) AS FLOAT)*100/(SELECT COUNT(*) FROM inv_album_or_not_table),2) AS percentage
FROM inv_album_or_not_table
GROUP BY invoice_album_or_not;Done.

### Recommendation:
- It can be observed that 18.57% of total sales are album purchases.
- If the management decides to purchase only the hit tracks, instead of entire albums, bulk of this 18.57% sales will be effected.

**Hence it is recommended not to go ahead with that decision**_____no_output_____## 7. Miscellaneous
Few more questions asked at the end are
- Which artist is used in the most playlists?
- How many tracks have been purchased vs not purchased?
- Is the range of tracks in the store reflective of their sales popularity?
- Do protected vs non-protected media types have an effect on popularity?_____no_output_____

%%sql
WITH playlist_info AS
(
    SELECT ar.artist_id artist_id, ar.name name
    FROM playlist_track pl_tr
    INNER JOIN track tr ON pl_tr.track_id = tr.track_id
    INNER JOIN album alb ON tr.album_id = alb.album_id
    INNER JOIN artist ar ON alb.artist_id = ar.artist_id
)

SELECT COUNT(artist_id) no_times, name
FROM playlist_info
GROUP BY artist_id
ORDER BY 1 DESC
LIMIT 10;
Done.

### a) Which artist is used in the most playlists?
Answer: It can be seen that Iron Maiden is in most of the playlists

### b) How many tracks have been purchased vs not purchased?_____no_output_____

%%sql
WITH invoice_info AS
(
SELECT track_id AS track_invoice FROM invoice_line
)

SELECT
    COUNT(track_id) no_times,
    CASE
        WHEN track_id in invoice_info
        THEN "Yes"
        ELSE "No"
    END AS purchased_or_not,
    ROUND(CAST(COUNT(track_id) * 100 AS FLOAT)/(SELECT COUNT(track_id) FROM track), 2) percentage
FROM track
GROUP BY purchased_or_not;Done.

**Answer**: Almost half of the tracks are never purchased. Now this becomes interesting. How many of the tracks were purchased more than 10 times, 100 times, 1000 times etc.
### c) Is the range of tracks in the store reflective of their sales popularity?_____no_output_____

%%sql
WITH track_purchases_info AS
(
SELECT
    COUNT(inv_li.track_id) AS track_purchased_times,
    inv_li.track_id,
    tr.name,
    tr.composer
FROM invoice_line inv_li
    INNER JOIN track tr ON inv_li.track_id = tr.track_id
GROUP BY inv_li.track_id
)

SELECT * FROM track_purchases_info ORDER BY 1 DESC LIMIT 50;Done.

**Observations**:
- Looks like HeavyMetal/Rock tracks are top sellers.
- Top two songs are on War theme.
- Overall there are only 59 users in the database, yet `War Pigs` was purchased by more than half of them. Interesting, may be the purchasers are Vietnam war veterans from USA ??? Lets, find out._____no_output_____

%%sql
/* Users who purchased War Pigs song */
SELECT
    /*cst.first_name || ' ' || cst.last_name AS user_name,*/
    COUNT(cst.customer_id),
    cst.country
FROM customer cst
    INNER JOIN invoice inv ON cst.customer_id = inv.customer_id
    INNER JOIN invoice_line inv_li ON inv.invoice_id = inv_li.invoice_id
    INNER JOIN track tr ON inv_li.track_id = tr.track_id
WHERE tr.track_id = 3336
GROUP BY country
ORDER BY 1 DESC;
Done.

Lol, our assumption is wrong. Only 6 purchases of 31 are from USA.
### d) Do protected vs non-protected media types have an effect on popularity?_____no_output_____

%%sql
WITH track_purchases_info AS
(
SELECT
    COUNT(inv_li.track_id) no_purchases,
    inv_li.track_id,
    inv_li.invoice_line_id inv_line_id,
    tr.media_type_id media_type_id,
    med_typ.name media_type_desc
FROM invoice_line inv_li
    INNER JOIN track tr ON inv_li.track_id = tr.track_id
    INNER JOIN media_type med_typ ON tr.media_type_id = med_typ.media_type_id
GROUP BY inv_li.track_id
)

SELECT SUM(no_purchases) total_purchases, media_type_id, media_type_desc
FROM track_purchases_info
GROUP BY media_type_desc
ORDER BY 1 DESC;Done.

**Answer is No. The most popular media type by far is `MPEG audio file` and not the latest `MPEG-4 Video`.**_____no_output_____


# Notebook from nikbearbrown/ARTG_5430
Path: Google_Trends_API/Google_Trends_API.ipynb

# Google Trends API for Python_____no_output_____## How to use the Google Trends API

![alt text](Google_Trends.png)_____no_output_____# Introduction

[Google trends](https://trends.google.com/trends/?geo=US) is a website that analyzes and lists the popular search results on Google search based on various regions and languages. The Python API called `pytrends` makes it super easy to grab and analyze these data._____no_output_____[pytrends API](https://pypi.org/project/pytrends/)

---_____no_output_____# Installation

The first step is to install the library manually. So, open your favorite IDE or notebook start typing the following code. I will use [Google Colab](https://colab.research.google.com/) because it's my favorite notebook.

> If you are using jupyter notebook, just type the code as it is (make sure you have '!' at the beginning)
_____no_output_____

# !pip install pytrends_____no_output_____

Or, if you are using an IDE, just type the following code

_____no_output_____```python
pip install pytrends
```_____no_output_____After executing the above code you should get a successful message as shown above

---

_____no_output_____# Implementation

## Connecting to Google

You must connect to Google first because after all, we are requesting the Google trending topics from Google Trends. For this, we need to import the method called `TrendReq` from `pytrends.request` library. Also, I will import the pandas library to store and visualize the data which you see in the later tutorial._____no_output_____

import pandas as pd
from pytrends.request import TrendReq
pytrend = TrendReq(hl='en-US', tz=360, timeout=(10,25))_____no_output_____

---

_____no_output_____```python
pytrend = TrendReq()
```

or if you want to use proxies as you are blocked due to Google rate limit:

```python
from pytrends.request import TrendReq

pytrends = TrendReq(hl='en-US', tz=360, timeout=(10,25), proxies=['https://34.203.233.13:80',], retries=2, backoff_factor=0.1, requests_args={'verify':False})
```

-   `timeout(connect, read)`

    -   See explantation on this on [requests docs]

-   tz

    -   Timezone Offset
    -   For example US CST is `'360'` (note **NOT** -360, Google uses
        timezone this way...)

-   `proxies`

    -   https proxies Google passed ONLY
    -   list
        `['https://34.203.233.13:80','https://35.201.123.31:880', ..., ...]`

-   `retries`

    -   number of retries total/connect/read all represented by one
        scalar

-   `backoff_factor`

    -   A backoff factor to apply between attempts after the second try
        (most errors are resolved immediately by a second try without a
        delay). urllib3 will sleep for:
        `{backoff factor} * (2 ^ ({number of total retries} - 1))`
        seconds. If the backoff\_factor is 0.1, then sleep() will sleep
        for \[0.0s, 0.2s, 0.4s, …\] between retries. It will never be
        longer than Retry.BACKOFF\_MAX. By default, backoff is disabled
        (set to 0).

-   `requests_args`

    -   A dict with additional parameters to pass along to the
        underlying requests library, for example verify=False to ignore
        SSL errors

Note: the parameter `hl` specifies host language for accessing Google
Trends. Note: only https proxies will work, and you need to add the port
number after the proxy ip address

### Build Payload

    kw_list = ["Blockchain"]
    pytrends.build_payload(kw_list, cat=0, timeframe='today 5-y', geo='', gprop='')

Parameters

-   `kw_list`

    -   *Required*
    -   Keywords to get data for

  [requests docs]: https://requests.readthedocs.io/en/master/user/advanced/#timeouts
pandoc 2.1

_____no_output_____### Keyword starter set

We will start with a keyword starter set then use related keywords and topics to expand the search.  _____no_output_____

keywords=['banana bread','cookie','muffin','brownies','beef stew','chili','sour doug','pretzel','meatloaf','salmon','smoothie','oatmeal','cake','casserole','steak','rice bean','pizza','chicken','hiit','cardio','yoga','meditation','arms workout','legs workout','core workout','workout music','full body workout','abs workout','back workout','booty workout','gluten workout','workout supplement','protein powder','belly fat workout','intense workout','beginner workout','weight loss workout','strength training','flexibility','cool down', 'stretching','skincare','haircare','body care','tanning', 'self tan','fake tan','nails tutorial','glow up','make up routine','brows','lash extension','dye hair','lounge wear','shaving/shave','waxing','lip care','teeth whitening','kept diet','paleo','vegan','vegetarian','carnivore diet','low carb','gluten-free','dairy-free','AIP diet','low FODMAP','atkins','raw diet','mediterranean diet','whole30','alkaline diet','zone diet','mind diet','sugar free','intermittent fasting','weight watchers','juice diet']

keywords_____no_output_____

## Building a payload

Up to five keywords can get trends at a time.  To do this we build a paylod._____no_output_____

five=['cookie','hiit','cardio','yoga','meditation']
five_____no_output_____pytrend.build_payload(kw_list=five,timeframe='today 5-y', geo='US')_____no_output_____

### Let's Find Interest Over Time

Let's find interest over time from today for the past five years._____no_output_____

# Interest Over Time
df_over_time = pytrend.interest_over_time()
print(df_over_time.shape)
df_over_time.tail(33)(260, 6)

**Now you might be thinking what are the values, what do they denote?**

The values are calculated on a scale from 0 to 100, where 100 is the location with the most popularity as a fraction of total searches in that location, a value of 50 indicates a location which is half as popular. A value of 0 indicates a location where there was not enough data for this term. Source → [Google Trends](https://support.google.com/trends/answer/4355212)..

Notice that the data comes back weekly and in the Google Trend 0 to 100 values. _____no_output_____### Saving data

The data comes back as a Pandas dataframe so we can

```python

df_over_time.to_csv(r'self_help_over_time.csv')

```_____no_output_____

df_over_time.reset_index().plot(x='date', y='yoga', figsize=(120, 10), kind ='bar')_____no_output_____

### Getting region specifc data

We can also get region specifc data from the payload.

_____no_output_____

# Interest by Region
df_interest_by_region = pytrend.interest_by_region()
print(df_interest_by_region.shape)
df_interest_by_region.tail(22)(51, 5)
df_interest_by_region.reset_index().plot(x='geoName', y='yoga', figsize=(120, 10), kind ='bar')_____no_output_____

### Related Queries

It's a common thing that when a user searches for a topic, they would also search for something related. These are called related queries._____no_output_____

# Related Queries, returns a dictionary of dataframes
related_queries_dict = pytrend.related_queries()
print(related_queries_dict){'cookie': {'top':                            query  value
0                  cookie recipe    100
1                   cookie dough     54
2                        cookies     52
3                   sugar cookie     44
4                 cookie clicker     43
5          chocolate chip cookie     40
6                 cookie recipes     28
7        chocolate cookie recipe     27
8            sugar cookie recipe     25
9   chocolate chip cookie recipe     24
10                 cookie butter     24
11                cookie monster     22
12                   cookie cake     21
13                 cookie cutter     15
14                    cookie jar     14
15                fortune cookie     12
16                cookie cutters     11
17          peanut butter cookie     11
18               american cookie     11
19                     cookie do     10
20          butter cookie recipe     10
21                  cookie swirl     10
22                cookie company      9
23                 sugar cookies      9
24         great american cookie      9, 'rising':                           query  value
0                        crumbl  34600
1                 crumbl cookie  34250
2            word cookie cheats  10450
3             word cookie cheat   9100
4         cookie swirl c roblox   7850
5   edible cookie dough near me   6100
6              cookie jam blast   5050
7           word cookie answers   4250
8               cookie munchers   3250
9        detroit cookie company   2500
10         digital cookie login   2400
11          cookie cake near me   2150
12      cookie delivery near me   1500
13                cookie kahuna   1400
14             cookie dough nyc   1400
15                  cookie plug   1350
16    cookie monster commercial   1300
17           cookie dough bliss   1200
18        cookie bakery near me   1150
19              the cookie plug   1100
20                cookie do nyc   1100
21          cookie monster siri   1000
22               cookie too big    800
23                  word cookie    700
24              cookie countess    650}, 'hiit': {'top':                      query  value
0             hiit workout    100
1            hiit workouts     57
2              cardio hiit     32
3            hiit training     22
4             what is hiit     12
5           treadmill hiit     11
6             hiit fitness     10
7           hiit exercises      9
8        best hiit workout      8
9                hiit burn      8
10            hiit at home      8
11     hiit cardio workout      8
12            hiit trainer      7
13       home hiit workout      6
14        hiit weight loss      6
15           hiit exercise      6
16  treadmill hiit workout      5
17            proform hiit      5
18    at home hiit workout      5
19          hiit for women      5
20                hiit gym      5
21       hiit workout plan      5
22    proform hiit trainer      5
23            hiit youtube      4
24         hiit elliptical      4, 'rising':                                      query   value
0                     proform hiit trainer  154150
1                      proform cardio hiit   71300
2              proform cardio hiit trainer   58000
3                       proform elliptical   31900
4                     hiit classes near me   28950
5                 proform hiit trainer pro   23350
6                               altra hiit   19100
7                          hiit with britt   15000
8                                  v shred   13000
9                            altra hiit xt   12250
10            proform hiit trainer reviews   12200
11  proform cardio hiit elliptical trainer   11750
12                       hiit gyms near me   10650
13                         proform hiit h7    9150
14                           down dog hiit    9100
15           proform hiit trainer lite 5.9    8400
16                            proform hiit    7750
17                             hiit by laf    7200
18                        hiit gym near me    6950
19                    30 min hiit workouts    6600
20                           hiit house la    5100
21       what does hiit training stand for    5050
22                         altra hiit xt 2    2500
23                   hiit workouts near me     800
24                  15 minute hiit workout     700}, 'cardio': {'top':                      query  value
0           cardio workout    100
1              best cardio     62
2          cardio workouts     46
3         cardio exercises     33
4           what is cardio     31
5          cardio exercise     30
6                 exercise     30
7              cardio hiit     30
8       cardio weight loss     20
9           at home cardio     20
10   cardio to lose weight     18
11          cardio machine     17
12           fasted cardio     16
13                insanity     15
14         insanity cardio     15
15  cardio for weight loss     14
16       cardio heart rate     13
17       cardio or weights     13
18     best cardio workout     12
19  cardio before or after     11
20             cardio bike     11
21              cardio abs     10
22        cardio equipment     10
23               treadmill      9
24      how to lose weight      9, 'rising':                                  query  value
0          proform cardio hiit trainer  20650
1                   coffee over cardio  14000
2                         peloton bike   6650
3              cardio drumming near me   5200
4                 cardio fitness score   3700
5            running late is my cardio   1350
6               cardio classes near me    850
7                      cardio drumming    650
8            cardio kickboxing near me    500
9               is cycling good cardio    450
10    insanity max 30 cardio challenge    300
11                      cardio miracle    300
12                 is jump rope cardio    250
13   cardio kickboxing classes near me    200
14        best cardio machine for home    200
15                 steady state cardio    200
16  can you lose weight without cardio    200
17            best home cardio machine    190
18                         liss cardio    190
19                      cardio meaning    170
20                   cardio conversion    170
21                cardio after leg day    160
22             is fasted cardio better    160
23           cardio para principiantes    160
24             does fasted cardio work    160}, 'yoga': {'top':                 query  value
0          yoga pants    100
1            hot yoga     80
2        yoga near me     56
3          yoga poses     55
4              lenovo     49
5         lenovo yoga     47
6            yoga mat     42
7          power yoga     34
8        yoga classes     31
9         bikram yoga     30
10          yoga pose     29
11        yoga studio     29
12     corepower yoga     23
13         yoga class     23
14       youtube yoga     22
15       what is yoga     22
16       adriene yoga     21
17          kids yoga     21
18          free yoga     20
19          core yoga     20
20  yoga with adriene     17
21    core power yoga     15
22       yoga workout     15
23     beginners yoga     14
24           alo yoga     13, 'rising':                         query  value
0             lenovo yoga 720  15400
1             lenovo yoga 710  14100
2             lenovo yoga 730   9650
3           goat yoga near me   8400
4             lenovo yoga 920   6350
5                   goat yoga   4700
6            lenovo yoga c740   4600
7         3 week yoga retreat   4400
8                    crz yoga   4200
9                    yoga 920   3200
10     yoga with adriene home   2600
11           hot yoga near me   1250
12               yoga near me   1150
13  yoga with adriene youtube   1150
14          yoga burn reviews   1050
15              sky ting yoga    950
16               adriene yoga    950
17                   alo yoga    950
18              alo yoga sale    950
19            yoga by adriene    900
20                  noli yoga    900
21           yoga hosers cast    900
22          yoga with adriene    850
23               yoga trapeze    800
24             lenovo x1 yoga    750}, 'meditation': {'top':                         query  value
0           meditation guided    100
1            meditation music     97
2            sleep meditation     82
3          meditation youtube     49
4             yoga meditation     36
5                        yoga     34
6                 mindfulness     33
7      mindfulness meditation     32
8   transcendental meditation     30
9        meditation for sleep     25
10         what is meditation     23
11            free meditation     23
12             meditation app     22
13         anxiety meditation     22
14    sleep guided meditation     22
15         morning meditation     19
16         healing meditation     19
17           daily meditation     16
18            kids meditation     16
19         meditation near me     15
20          chakra meditation     14
21       music for meditation     14
22            meditation apps     14
23         meditation retreat     13
24     meditation for anxiety     12, 'rising':                                                 query  value
0                   jason stephenson sleep meditation  16700
1                     meditation for fidgety skeptics   4750
2   yellow brick cinema reiki zen meditation music...   3800
3                           sam harris meditation app   2550
4                          meditation retreat near me   2400
5                               sound bath meditation   2250
6   the honest guys relaxing music with water soun...   2250
7                                    abide meditation   2000
8                                      den meditation   1500
9                                  meditation near me   1300
10                                 the den meditation   1200
11                               meditation apps free   1150
12                            heartfulness meditation   1150
13                         meditation classes near me   1000
14                        jason stephenson meditation    950
15                          sleep meditation for kids    800
16                            joe dispenza meditation    750
17                              kids sleep meditation    700
18                          best free meditation apps    700
19                            yoga meditation near me    600
20                                               muse    600
21                                   meditation emoji    600
22                         gabby bernstein meditation    550
23                                2 minute meditation    550
24                              kids meditation music    500}}

## Daily Search Trends

Now let us get the top daily search trends worldwide. To do this we have to use the `trending_searches()` method. If you want to search worldwide just don't pass any parameter._____no_output_____

# Get Google Trends data
trending_searches_df = pytrend.trending_searches()
print(trending_searches_df.head())                  0
0             Oncor
1   Vincent Jackson
2           Chelsea
3  Champions League
4         Generator

### Countries other than the US

If you want a country other than the US, make sure you enter the country name in lowercase `pn = "canada"` . Also, you can compare the above results with the [google trend's result](https://trends.google.com/trends/trendingsearches/daily?geo=US). To get today's trending topics just use:_____no_output_____

# Get Google canada Trends data
df = pytrend.trending_searches(pn='canada')
print(df.shape)
df.head(11)(20, 1)

---

_____no_output_____## Top Charts

Let us see what was trending in 2020. With the help of `top_charts` method we can get the top trending searches yearly._____no_output_____

# Get Google Top Charts
top_charts_df = pytrend.top_charts(2020, hl='en-US', tz=365, geo='GLOBAL')
print(top_charts_df.shape)
print(top_charts_df.head())(10, 2)
              title exploreQuery
0       Coronavirus
1  Election results
2       Kobe Bryant
3              Zoom
4               IPL

To compare the results just visit [Google Trends](https://trends.google.com/trends/yis/2019/GLOBAL/). We can specify the year and the country that we want to see the trending searches.

---

_____no_output_____### Google Keyword Suggestions

Let us see how can we obtain google's keyword suggestion. This is our main tool to expand our search.

![alt text](explore_yoga.png)_____no_output_____

# Get Google Keyword Suggestions
# Get Google Keyword Suggestions
keywords_dict = pytrend.suggestions(keyword='yoga')
keywords_dict_____no_output_____

### Create a Pandas dataframe from a python dictionary

Creating a Pandas dataframe from a python dictionary will make saving a visualizing easier._____no_output_____

df = pd.DataFrame(keywords_dict)
df.drop(columns= 'mid')   # This column makes no sense
print(df.shape)
df(5, 3)

### Related Topics and Queries

It's a common thing that when a user searches for a topic or keyword, they would also search for something related.

We need to rebuild the payload for just one keyword for related topics and queries. Always remember when you want to change the topic name you need to rebuild the payload. _____no_output_____

pytrend.build_payload(kw_list=['yoga'],timeframe='today 5-y', geo='US')_____no_output_____

### Related Topics

Finding related topics will also help expand the keyword search.

_____no_output_____

# Related Topics, returns a dictionary of dataframes
related_topics = pytrend.related_topics()
related_topics.values()_____no_output_____# Related Queries, returns a dictionary of dataframes
related_queries = pytrend.related_queries()
related_queries.values()_____no_output_____

Reach out if you have questions.  Nik Bear Brown [https://www.linkedin.com/in/nikbearbrown/](https://www.linkedin.com/in/nikbearbrown/)_____no_output_____


# Notebook from EiriniNBI/eirinipy
Path: AppStat2022/Week7/original/Calibration/Calibration_original.ipynb

# Calibration Exercice

This exercise is about calibration, that is to make a certain type of method or measurement more precise by taking into account effects that otherwise blurs the precision of the method by correcting for these effects. The below example is constructed/invented, but outlines the considerations.

## Description
  You're developping a new method for measuring the distance to stars, and want to
  calibrate and thus improve this method, such that the precision obtained is unbiased
  and has a minimal variance. You know that the method depends on several factors, such as:
   * Amount of signal light from the star `lsig`
   * Amount of background light in the surrounding sky `lbkg`
   * Temperature of star `temp`
   * Transparency of sky `tsky`

In order to determine the influence of these factors, and how much you need to correct for each of them, you consider 10.000 stars with known distances (measured by another method, e.g. triangulation). From these, you can find how well your own method works, make corrections to biases as needed, and finally find out how precise your calibrated method is. Happy calibration.

## Your Task

* As always look at the data and get a feel for each of the variables. A good idea might be to plot them all to know what range to expect them in.

* First, consider the raw distribution of "relative differences" ($R_D$) between the observed and actual distance: $R_{D} = \frac{(D_{obs} - D_{known})}{D_{known}}$. You'll notice that this distribution is far from an ideal (and narrow) gaussian, which is why you'll need to calibrate the underlying data. Your calibration technique should be able to do the following:

    - Reduce the RMS of the distribution
    - Correct the offset from zero
    - Mitigate the tail at high values

* Secondly, look at the distribution of the bias and relative precision as a function of the data variables. Try to identify variables that are correlated with the relative differences, and apply a correction factor to eliminate these biases.

Note that if you are on average say 50% too high compared to the true values, then you need to correct by 50%, i.e. divide by (1 + 0.5), and in general, if your measurement is $f(x)$ off, where $f(x)$ describes the offset, then you need to divide by $(1 + f(T))$:

$R_{D} = \frac{d - d_{true}}{d_{true}}  \rightarrow  d_{calib} = \frac{d}{(1 + f(x))}$

Thus, define `d_calib = d / (1 + f(x))`, and continue using `d_calib` when considering other effects.

##  Author:
- Troels Petersen ([email](mailto:petersen@nbi.dk))

##  Date:
6th of January 2022_____no_output_____

import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
from iminuit import Minuit
from scipy import stats
import os, sys                                         # Modules to see files and folders in directories
from os.path import dirname as parent_folder_____no_output_____sys.path.append('../../../External_Functions')
from ExternalFunctions import Chi2Regression, BinnedLH, UnbinnedLH
from ExternalFunctions import nice_string_output, add_text_to_ax    # Useful functions to print fit results on figure_____no_output_____# Set som plotting standards:
font = {'family' : 'serif',
        'weight' : 'normal',
        'size'   : 18}
mpl.rc('font', **font)_____no_output_____verbose = True
Nverbose = 10
SaveFigures = False_____no_output_____

### Define profile function (2D histogram to 1D plot):

Profile x of the two arrays x and y with defined number of bins and range returns the x-values of the profile, the means and the standard deviations.

This enables you to fit a trend in a 2D histogram (which shows any variation of your key variable of interest with other variables) with a 1D function, and thus currect of the possible effect of this other variable._____no_output_____

def profile_x(x, y, bins=(50, 50), xyrange=[(0, 50), (-1,1)]):

    H, xedges, yedges = np.histogram2d(x, y, bins=bins, range=xyrange)
    x_center = 0.5*(xedges[1:] + xedges[:-1])
    y_center = 0.5*(yedges[1:] + yedges[:-1])

    wsums = H.sum(1)

    mask = wsums > 0

    mean = (H*y_center).sum(1)[mask] / wsums[mask]
    mean_squared = (H*y_center**2).sum(1)[mask] / wsums[mask]
    std = np.sqrt( mean_squared - mean**2 ) / np.sqrt(wsums[mask])

    return x_center[mask], mean, std_____no_output_____

### Reading the data (text) file:

There are multiple ways we can retrieve the information from a text file, but here we use the `loadtxt` function of the `numpy` package. This is a very powerful function that allows you to parse an entire textfile in a single line, _provided that the text file is formatted properly_. For more advanced parsing option, you can also check out `numpy.genfromtxt`._____no_output_____

filename = "data_calib.txt"
dknown, dmeas, lsig, lbkg, temp, tsky = np.loadtxt(filename, unpack=True)_____no_output_____# Check that this worked:
if (verbose) :
    for i in range(Nverbose) :
        print("  Distance (known) = {:8.2f}    Distance (measured) = {:6.3f} ".format(dknown[i], dmeas[i]))
print("The TOTAL number of entries read is: ", len(dmeas))  Distance (known) =  9209.51    Distance (measured) = 5820.208
  Distance (known) =   149.38    Distance (measured) = 174.093
  Distance (known) =   412.18    Distance (measured) = 423.608
  Distance (known) =   119.81    Distance (measured) = 126.137
  Distance (known) =   175.10    Distance (measured) = 250.720
  Distance (known) =   146.20    Distance (measured) = 227.391
  Distance (known) =   324.05    Distance (measured) = 422.474
  Distance (known) =   256.97    Distance (measured) = 229.279
  Distance (known) =   175.40    Distance (measured) = 156.558
  Distance (known) =   353.27    Distance (measured) = 353.121
The TOTAL number of entries read is:  10000

### Make Histograms and vectors:_____no_output_____

plt.ioff() # We use to hide the empty plot on this cell
fig_rel, ax_rel = plt.subplots(figsize=(15, 8));
ax_rel.set_title('Hist relative resolution');
Nbins = 200
xmin, xmax = -2.0, 2.0
binwidth = (xmax-xmin) / Nbins_____no_output_____fig_lsig2D, ax_lsig2D = plt.subplots(figsize=(12,8));
ax_lsig2D.set_title('Hist lsig 2D')_____no_output_____

### Loop over data and make plots: (TO DO YOURSELF)
----
----_____no_output_____

# The initial relative resolution:
distrel = (dmeas - dknown) / dknown_____no_output_____# Apply calibration as found from the 2D plot below:
dmeas_calib = dmeas      # So here is so far no calibration - that is what you have to change!!!_____no_output_____# Once you have made the calibration, the (improved) relative resolution becomes:
distrel_calib = (dmeas_calib - dknown) / dknown_____no_output_____# Determine the resolution (quantified as the RMS) before and after the calibration:
mask_distrel       = (xmin < distrel) & (distrel < xmax)
mask_distrel_calib = (xmin < distrel_calib) & (distrel_calib < xmax)
print("  The initial and final resolutions are:  {:6.3f}  and  {:6.3f} \n".format(
        distrel[mask_distrel].std(ddof=1), distrel_calib[mask_distrel_calib].std(ddof=1)))  The initial and final resolutions are:   0.269  and   0.269

## Compare the raw and calibrated data on a plot_____no_output_____

ax_rel.hist(distrel, bins=Nbins, range=(xmin, xmax), histtype='step', label='Raw')
hist_rel_calib = ax_rel.hist(distrel_calib, bins=Nbins, range=(xmin, xmax), histtype='step', label='Calibration')_____no_output_____# Plotting attributes
ax_rel.set_xlim(xmin, xmax)
ax_rel.set_xlabel('Realitive precision (dmeas - dknown) / dknown',fontsize=18)
ax_rel.set_ylabel('Frequency / 0.02', fontsize=18)
ax_rel.legend(loc='best', fontsize=18)

d = {'Entries':"{:d}".format(len(distrel[mask_distrel])),
     'Mean':"{:.3f}".format(distrel[mask_distrel].mean()),
     'STD Dev.':"{:.3f}".format(distrel[mask_distrel].std(ddof=1))}

ax_rel.text(0.04, 0.95, nice_string_output(d), family='serif', fontsize=18,
        transform=ax_rel.transAxes, verticalalignment='top')
fig_rel.tight_layout()
plt.ion()
fig_rel_____no_output_____if (SaveFigures):
    fig_rel.savefig('UncalibratedCalibrated.pdf', dpi=600)_____no_output_____

##  Calibration using 2D histograms:

The way to investigate, if there is a (cor)relation between the target (relative distance) and any of the factors/variables/auxiliary observables, is to plot them against each other. If there are any visible relation, then this is an effect of the ourside factors (lsig, lbkg, temp, and tsky) on the wanted observable (relative distance), which we want to correct for.

In the following, we produce this 2D histogram along with a 1D graph on top showing the mean and uncertainty for each bin in x. This graph shows any possible (average) relation in a way, which can be easily fitted._____no_output_____

# Histograms for making calibration:
ax_lsig2D.hist2d(lsig, distrel, bins=50, range=[(0, 50), (-1,1)], cmin=1, alpha=0.5)
ax_lsig2D.set_xlabel('Signal light from star (lsig)');
ax_lsig2D.set_ylabel('Realitive precision (dmeas - dknown) / dknown');_____no_output_____# This calculates the 1D "profile" of the 2D histogram:
x_center_lsig2D, mean_lsig2D, std_lsig2D = profile_x(lsig, distrel, bins=(50, 50), xyrange=[(0, 50), (-1,1)])
x_binwidth_lsig2D = x_center_lsig2D[1] - x_center_lsig2D[0]_____no_output_____ax_lsig2D.errorbar(x_center_lsig2D, mean_lsig2D, xerr=x_binwidth_lsig2D/2, yerr=std_lsig2D,  fmt='r.',
                   ecolor='r', elinewidth=1, capsize=1, capthick=1);_____no_output_____# Fit of the effect from "lsig" in the 1D graph:
def simple_fit(x, p0, p1):
    return p0 + p1*x_____no_output_____chi2_object_simple = Chi2Regression(simple_fit, x_center_lsig2D, mean_lsig2D, std_lsig2D)
minuit_simple = Minuit(chi2_object_simple, p0=0.0, p1=0.0)
minuit_simple.errordef = 1.0
minuit_simple.migrad()            # fit_____no_output_____p0, p1 = minuit_simple.values[:]
print("Simple fit result:")
for name in minuit_simple.parameters:
    print("Fit value: {0} = {1:.5f} +/- {2:.5f}".format(name, minuit_simple.values[name], minuit_simple.errors[name]))Simple fit result:
Fit value: p0 = 0.11513 +/- 0.00604
Fit value: p1 = 0.00001 +/- 0.00022
x_fit = np.linspace(0, 50, 1000)
y_fit_simple = simple_fit(x_fit, *minuit_simple.values[:])
ax_lsig2D.plot(x_fit, y_fit_simple, '--k')

d = {'Entries':"{:d}".format(len(lsig)),
     'Mean x' :"{:.3f}".format(lbkg.mean()),
     'Mean y' :"{:.5f}".format(distrel.mean()),
     'RMS x'  :"{:.3f}".format(lbkg.std(ddof=1)),
     'RMS y'  :"{:.5f}".format(distrel.std(ddof=1))}

ax_lsig2D.text(0.03, 0.2, nice_string_output(d), family='monospace',
        transform=ax_lsig2D.transAxes, fontsize=16, verticalalignment='top')
fig_lsig2D.tight_layout()
fig_lsig2D_____no_output_____

## Your Task (reminder)

 As always look at the data and get a feel for each of the variables. A good idea might
 be to plot them all to know what range to expect them in.

 Next, consider the distribution of "relative differences" (RD) between the observed
 and actual distance: RD = (dist_obs - dist_known) / dist_known

 The Standard Deviation is 0.27, i.e. a 27% precision, and it is neither centered at zero
 (as it should, not to be biased), nor very Gaussian. There is also a long tail towards very
 high values. This is what you want to improve upon!

 Finally, there is the distribution of the bias and relative precision as a function of
 the signal luminosity (lsig). As you can see, the response does not depend on lsig,
 and so there seems to be no (varying) bias from this variable. Check the other three
 variables, and if you find some bias, try to correct for it, such that you get the
 relative difference to be the most narrow Gaussian centered around 0.

 Note that if you are on average say 50% too high compared to the true values, then
 you need to correct by 50%, i.e. divide by (1 + 0.5), and in general, if your
 measurement is f(x) off, where f(x) describes the offset, then you need to divide
 by (1 + f(T)). Why:

   RD = (d - d_true) / d_true  =>  d_true = d / (1 + f(x))

 Thus define d_calib = d / (1 + f(x)), and continue using d_calib when considering
 other effects. In the end, your calibrated measurement can be the result of several
 consecutive corrections (note that the order might matter!).

Questions:
----------
1. What corrections do you apply for each of the variables, and how much do each of them improve on the result?

2. What is the final resolution you obtain?

3. Do you think that further improvements are possible in this example (constructed by me)? How about in real and more complicated data?

Advanced Questions:
-------------------
4. Try at the end to figure out, which variables the final resolution depends on.
     Can you give an estimate of the uncertainty for each single star?_____no_output_____