# Notebook from meganmason/website
Path: book/tutorials/microstructure/microstructure-tutorial.ipynb

# Microstructure Tutorial

by Mike Durand, School of Earth Sciences and Byrd Polar & Climate Research Center durand.8@osu.edu

## Learning Objectives

At the end of this tutorial you should be able to...

* Explain why microstructure is important for remote sensing
* Define measures of microstructure, especially specific surface area
* Access and visualize tree different microstructure measurements from SnowEx Grand Mesa 2020

## Acknowledgments

Contributions from: Micah Johnson, Mike Durand, HP Marshall, Tate Meehan, Megan Mason, Scott Henderson. This relies heavily on the snowexsqul database and example scripts created by Micah Johnson.

## Caveats

The integrating sphere and the SMP data are published at NSIDC; you can read the pages there for documentation etc. However the microCT data are not yet published; please contact Lauren Farnsworth (lauren.b.farnsworth@usace.army.mil) for questions on the CT data.

## Fun Facts About Snow Microstructure

Snow microstructure plays a super important role in snow physics and snow remote sensing, so a lot of effort went towards measuring it in SnowEx 2020!

There are several different quantities that are used to measure snow microstructure, including "grain size". Grain size measurements are challenging to make in a repeatable way, and are also challenging to relate to the physical quantities that control remote sensing measurements. In the last ~15 years or so, a lot of effort has gone into more objective ways to measure microstructure.

Snow microstructure  governs response of remote sensing to snow cover for visible, near-infrared and high-frequency microwave wavelengths. See Figure 1, below, and read {cite:p}`Dozier2009`, for more information.

Snow microstructure governs visible and near-infrared reflectance. This is figure 2 from {cite:p}`Dozier2009`

Radar measurements such as those made by the Ku-band SWEARR instrument are also very sensitive to snow microstructure.

Modeled response of radar backscatter to SWE and single-scatter albedo (which in turn is a function of snow microstructure), based on a simple model suggested by {cite:p}`Ulaby2014`

Snow microstructure is super important to efforts to launch a Ku-band SAR to measure global snow water equivalent (SWE). An important area of research right now is exploring how to use estimates of microstructure (e.g. from snowpack evolution models) to improve SWE retrievals.

Snow microstructure evolves through the season, and varies a lot with depth. Snow microstructure evolution is controlled by other snow properties, such as snow temperature, snow height and snow liquid water content. A really great resource on snow microstructure is Kelly Elder's recent talks:
* [Snow Metamorphism](https://boisestate.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=9a051fe4-db09-4f1d-92db-acd1003cc5cc)
* [Snow Grain Identification](https://boisestate.hosted.panopto.com/Panopto/Pages/Embed.aspx?id=136f7580-c1e2-4c55-ab33-acca003c23d6)_____no_output_____## SnowEx Microstructure Measurement Background

### Basic Microstructure Definitions

Microstructure definitions take a bit of getting used to. It's very easy to get confused. Specific surface area (SSA) is one of the most important quantity used to measure snow microstructure, so that's the focus here. Note that SSA is not the be-all and end-all, so there's a short table describing how to relate SSA to other quantities just below.  A couple of good reads on all of this is {cite:p}`Matzler2002` and {cite:p}`Matzl2010`.

Coarse and fine snow microstructure revealed by microCT. The microCT snow renderings on the left are Figure 2 from {cite:p}`Lowe2011`. The colorbars indicate that fine-grained snow (a) has high SSA and low Deq, whereas coarse-grained snow (b) has low SSA and high Deq.

Use Figure 3 above to ground these definitions: SSA is the surface area of the ice-air interface, normalized in some way. Confusingly, SSA is defined in a couple of different ways in the literature: sometimes, surface area within a particular volume of interest (VOI) is normalized by the mass of the ice in the VOI. As defined in this way, SSA has units of length squared per mass, usually expressed as m2/kg. Instead of normalizing by mass, SSA is sometimes defined by normalizing by the volume of the VOI (this is SSAv in {cite:p}`Matzl2010`), and sometimes by normalizing by the volume of the ice in the VOI (this is SSAi in {cite:p}`Matzl2010`, and q in {cite:p}`Matzler2002`). Here let's just go with the first definition I mentioned:

$$
SSA = \frac{\text{Surface area of ice-air interface}}{\text{Mass of ice}} \quad
$$

SSA tends to take values between 5 and 150 m2/kg: fresh, fine-grained snow has high SSA, and coarse snow has low SSA. Because it takes a little while for SSA values to become intuitive, a useful derived metric is the equivalent grain diameter (Deq; note that this is identical to Dq in {cite:p}`Matzler2002`), which by definition is the diameter that a sphere would have if it had a particular value of SSA. This is a one-to-one relationship, so there are no assumptions involved.

$$
D_{eq} = \frac{6}{SSA \rho_i} \quad
$$ _____no_output_____Relationships of specific surface area to other metrics are given in this list if you're curious but otherwise just skip past this bit
* Sometimes people refer to the "optical grain diameter", which is the same as Deq. The "optical" refers to {cite:p}`Grenfell1999`, who showed that any snow with a particular SSA had similar (not identical) radiative transfer properties regardless of particle shape in the visible and near-infrared parts of the spectrum. But note the same is not true in the microwave spectrum.
* Autocorrelation length is usually one of two metrics that summarize the two-point microstructure autocorelation function of the three-dimensional ice-air matrix. Think of the probability that you change media (from ice to air or vice versa) as you move a certain distance within the snow microstructure. The length that defines the likelihood that you'll change media is (an approximation of the correlation length). SSA is by definition (with almost no assumptions) equal to the slope of the autocorrelation function at the origin. But microwave scattering is controlled by correlations at longer lags. For more check out {cite:p}`Matzler2002`. The lack of closing the loop between SSA and correlation length is a significant issue when we have measurements of SSA and microwaves as we do in SnowEx.
* Geometric grain size is what we usually measure when we measure with a hand lens. You can try to relate it to SSA or corelation length, but it is not always possible, and will change with different observers.

Time to stop this list but there are many other metrics as well._____no_output_____### Microstructure Instruments

Now that we know what we're trying to measure (SSA, or correlation length) how do we actually measure? let's talk just about three techniques used in SnowEx 2020 Grand Mesa.

Left: Lauren Farnsworth transports microCT samples from field sites back to Grand Mesa Lodge in a cold storage container. Right: the microCT machine in the lab at CRREL.

Micro-computed tomography (microCT) is the only laboratory-based method used here, and it is the gold standard, although it does still come with caveats. The idea of microCT is to remove a sample of snow from a snow pit face, and either cast it with a compound such as diethyl pthalate that is still a liquid at 0° C, or preserve the same at a very cold temperature. Then the sample is sent back to the laboratory, and bombared with x-rays, similar to how you get x-rays to see if a bone is broken at the doctor. For much more on microCT, check out {cite:p}`Heggli2011`. microCT can be used to extract a ton of information about snow microstructure, including SSA, correlation length and many others.

Left: Kehan Yang operates an IceCube unit at the Grand Mesa Lodge intercomparison snowpit. Top right: schematic showing the integrating sphere measurement principle, from {cite:p}`Gallet2009`. Bottom right: snow in the IceCube sampling container, from {cite:p}`Leppanen2018`.

Integrating spheres are field-based and you make the measurements on samples extracted from the snowpit face. The principle of the measurement is based on firing a laser at the snow sample, within a special reflective hollow sphere, where one side is filled by the snow sample, and measuring how much of the laser is reflected at a sensor at a known geometry. For more information, check out {cite:p}`Gallet2009`.  Most integrating sphere measurements are either made by a commercial firm (A2 Photonics) known as the [IceCube](https://a2photonicsensors.com/icecube-ssa/) or a version constructed at the University of Sherbrooke known as the IRIS {cite:p}`Montpetit2012`. These approaches are set up to measure SSA only. There were three of these at Grand Mesa - one of the Sherbrooke IRIS units, and two IceCubes, one from Finnish Meteorological Institute, and one from Ohio State University.

Left: Megan Mason operates the SMP. Right: Closeup of the SMP sensor tip.

Snow micropenetrometers are also a field-based approach, but they do not require a snowpit, enabling far more observations to be made. Instead, an automated motor pushes a probe vertically downwards into the snowpack. The probe measures the force required to break snow microstructure, yielding a wealth of information. Snow density, specific surface area and correlation length can be retrieved; for background see {cite:p}`Lowe2012` and {cite:p}`Proksch2015`. The micropen effort at Grand Mesa was led by Boise State University. A key thing to be aware of is that differences in the various SMP instruments mean that the empirical relationship of {cite:p}`Proksch2015` will give quite poor results for the particular instrument used in SnowEx, as fully explained in {cite:p}`Calonne2020`.

These methods are not the only ways to measure microstructure! There are several others not mentioned here, but not used at Grand Mesa 2020. Ask if interested.

## SnowEx Microstructure Measurement Data Overview

Of the three methods described above, microCT is by far the most expensive and most time consuming. Samples have to be transported back to the laboratory and the processing time requires a microCT machine. Thus the fewest CT sapmles are taken.

The integrating spheres require a snowpit to be dug, so we have an intermediate number of them: ~100.

The micropen measurements are by far the fastest to make, so a cross pattern of SMP measurements was made on orthogonal directions intersecting at the snowpit. There are thousands of SMP profiles from Grand Mesa 2020._____no_output_____## Working with the data

We're going to do two things! First, we'll intercompare the three different integrating sphere instruments at four different pits where we had multiple instruments operating. We'd expect these data to be fairly self-consistent. Second, we'll compare all three methods (integrating sphere, SMP and microCT) at a single pit where we had all of these measurements present. Here especially with the SMP we would expect to need to intercalibrate the data to match local conditions; so far SSA has only been fit to SMP force measurements in one study, and we should assume we'll need a local calibration to get a tight fit._____no_output_____### 0. Load needed modules_____no_output_____

# Modules needed to access snowexsql: SnowEx field data database
from snowexsql.db import get_db
from snowexsql.data import LayerData, PointData
from snowexsql.conversions import points_to_geopandas, query_to_geopandas

# Modules needed to work with data
import geoalchemy2.functions as gfunc
import geopandas as gpd
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np

import warnings
warnings.filterwarnings('ignore')

#note - this cell does not return any output_____no_output_____

### 1. Intercompare Integrating Sphere Datasets_____no_output_____There were three integrating spheres. The IRIS unit from University of Sherbrooke was operated by Celine Vargel. The IceCube unit from the Finnish Meteorological Institute was operated by Juha Lemmetyinen. And the IceCube unit from Ohio State was operated by Kehan Yang and Kate Hale. Carefully read the [documentation page](https://nsidc.org/data/SNEX20_SSA/versions/1) at NSIDC if you are interested in the data. If you are using the data for a project, please contact the authors and mention what you're doing - they'll appreciate it! Contact for SSA is Mike Durand (durand.8@osu.edu).

See Micah's tutorial on datasets for more on this! Won't explain too much here_____no_output_____

db_name = 'snow:hackweek@db.snowexdata.org/snowex'
engine, session = get_db(db_name)

# Grab all the equivalent diameter profiles
q = session.query(LayerData).filter(LayerData.type == 'specific_surface_area')
df = query_to_geopandas(q, engine)

# End our database session to avoid hanging transactions
session.close()

df.head() #check out the results of the query_____no_output_____

Since we want to intercompare integrating spheres, we need to isolate only the sites that actually had multiple integrating spheres measuring the same snow._____no_output_____

# Grab all the sites with equivalent diameter data (set reduces a list to only its unique entries)
sites = df['site_id'].unique()

# Store all site names that have multiple SSA instruments
multi_instr_sites = []
instruments = []

for site in sites:

    # Grab all the layers associated to this site
    site_data = df.loc[df['site_id'] == site]

    # Do a set on all the instruments used here
    instruments_used = site_data['instrument'].unique()

    if len(instruments_used) > 1:
        multi_instr_sites.append(site)

# Get a unqique list of SSA instruments that were colocated
instruments = df['instrument'].unique()

instruments #check out the list of instruments. note that the IceCube values are displayed as serial numbers_____no_output_____

Finally, plot all Integrating Sphere SSA profiles at all Multi-Integrating Sphere Sites_____no_output_____

# Setup the subplot for each site for each instrument
fig, axes = plt.subplots(1, len(multi_instr_sites), figsize=(4*len(multi_instr_sites), 8))

# Establish plot colors unique to the instrument
c = ['k', 'm', 'c']
colors = {inst:c[i] for i,inst in enumerate(instruments)}

# Loop over all the multi-instrument sites
for i, site in enumerate(multi_instr_sites):

    # Grab the plot for this site
    ax = axes[i]

    # Loop over all the instruments at this site
    for instr in instruments:

        # Grab our profile by site and instrument
        ind = df['site_id'] == site
        ind2 = df['instrument'] == instr
        profile = df.loc[ind & ind2].copy()

        # Don't plot it unless there is data
        if len(profile.index) > 0:

            # Sort by depth so samples that are take out of order won't mess up the plot
            profile = profile.sort_values(by='depth')

            # Layer profiles are always stored as strings.
            profile['value'] = profile['value'].astype(float)

            # Plot our profile
            ax.plot(profile['value'], profile['depth'], colors[instr], label=instr)

    # Labeling and plot style choices
    ax.legend()
    ax.set_xlabel('SSA [m^2/kg]')
    ax.set_ylabel('Height above snow-soil interface [cm]')
    ax.set_title('Site {}'.format(site.upper()))

    # Set the x limits to show more detail
    ax.set_xlim((8, 75))

plt.tight_layout()
plt.show()
_____no_output_____

### 2. Pull the snowmicropenetrometer data and compute SSA

The next step is to grab some SMP data to compare to. We're going to get the SMP at site 2N13, where we have a copule of SSA profiles from integrating spheres (as well as microCT data, to be looked at in the next step!).

The SMP measurements for SnowEx 2020 GrandMesa were all made by Megan Mason. If you're interested in working with the SMP data, please carefully read the NSIDC [documentation page](https://nsidc.org/data/SNEX20_SMP/versions/1). If you're planning to work with the data, please reach out to the author; the contact is (Megan Mason meganmason491@u.boisestate.edu). If you use a profile, consider checking out the comments which are described in the [Excel sheet linked from the Technical References part of the NSIDC documentation](https://nsidc.org/sites/nsidc.org/files/technical-references/SNEX20_SMP_FieldNotes.xlsx), where there are some really useful comments.

There are a few steps here, and one reason for that is just that the SMP data is quite large, and so the full-resolution SMP could not be included in Micah's database. The full resolution profile from SMP is resolved ever 1.25 mm! Instead, the SMP data in Micah's database is sampled to only every 100th datapoint, so it's every 12.5 cm. But the database is still very useful! What we'll do is use the database to find the right profile, then go and download that full resolution dataset from the NSIDC. Easy-peasey!

As mentioned above, {cite:p}`Calonne2020` tested applying the relationship of {cite:p}`Proksch2015` and got quite poor results, explained them by the difference in hardware between generations of SMP instruments. We were unaware of that when designing the tutorial, and so set up use of the so-called official SMP processing repository, linked below, which has not yet been updated with the latest relationship. This would make a great project, as mentioned later!

First up, we'll visualize the location of the SMP profiles, along with the snowpit location._____no_output_____

site = '2N13'
engine_smp, session_smp = get_db(db_name)
q_smp = session_smp.query(LayerData).filter(LayerData.type == 'force').filter(LayerData.site_id.contains(site) )
df_smp = query_to_geopandas(q_smp, engine_smp)

q_pit=session_smp.query(LayerData).filter(LayerData.type == 'hand_hardness').filter(LayerData.site_id.contains(site) )
df_pit = query_to_geopandas(q_pit, engine_smp)

session_smp.close()

# Plot SMP profile locations with colored by the time they were taken using upside down triangles
ax = df_smp.plot(column='time', cmap='jet', marker='v', label='SMP', figsize=(5,5), markersize=100, edgecolor='black')

ax.plot(df_pit.easting, df_pit.northing, color='black', marker='s', markersize=15, label='Pit ({})'.format(site))

# Add important labels
ax.set_xlabel('Easting [m]')
ax.set_ylabel('Northing [m]')
plt.suptitle('SMP Locations at Site {} Showing Acquisition Order'.format(site), fontsize=16)

# Avoid using Scientific notation for coords.
ax.ticklabel_format(style='plain', useOffset=False)
ax.legend()
# plt.tight_layout()
plt.show()_____no_output_____

Next up, let's find the closest SMP profile to the snowpit, and then find the profile ID of that profile, which is in the comments in the database._____no_output_____

# find closest SMP profile to the pit

# No profile is taken at the same time, so we grab all the unique times and sort them
times = sorted(df_smp['time'].unique())

nprofiles=len(times)

ids=np.empty(nprofiles)

p=0
for t in times:
    ind = df_smp['time'] == t
    data = df_smp.loc[ind].copy()
    ids[p]=data.iloc[0].id
    p+=1

i_dists=df_smp['id'].isin(ids)

df_smp_dists=df_smp.loc[i_dists]
df_smp_dists=df_smp_dists.assign(dists=-1)
df_smp_dists['dists']=np.sqrt((df_smp_dists['easting']-df_pit.iloc[0].easting)**2+(df_smp_dists['northing']-df_pit.iloc[0].northing)**2)

df_smp_dists.sort_values(by='dists')[['comments','dists']].head() #check out the list of profiles sorted by distance to pit_____no_output_____

So the id of the closest SMP profile is S19M1174.  I went to the [SMP page on NSIDC](https://nsidc.org/data/SNEX20_SMP/versions/1), and went to "Download" and searched for this ID, downloaded the profile, and then re-uploaded to my home directory here in the Jupyter hub.

Ok next up, we have to compute SSA from the SMP data. For this, we'll use the "snowmicropyn" modules created by the Swiss SLF. You can read more about them [at this site](https://snowmicropyn.readthedocs.io/en/latest/). The software is a little out of date on Python versions; just ignore any warnings that pop up below! Also, the use of the  {cite:p}`Proksch2015` relationship is also out-of-date, as mentioned above. Getting it updated for use with this tutorial would make a perfect project!

The next cell pulls in the needed modules, and then plots the profile of force measurements needed to break through the snow microstructure._____no_output_____

from snowmicropyn import Profile
from snowmicropyn import proksch2015_____no_output_____# Pull in some tutorial datasets
!aws s3 sync --quiet s3://snowex-data/tutorial-data/microstructure/ /tmp/microstructure_____no_output_____p = Profile.load('/tmp/microstructure/SMP/SNEX20_SMP_S19M1174_2N13_20200206.PNT',)
plt.plot(p.samples.distance, p.samples.force)
# Prettify our plot a bit
plt.title(p.name)
plt.ylabel('Force [N]')
plt.xlabel('Depth [mm]')
plt.show()_____no_output_____

The above shows the entire SMP profile; this includes the part of the force profile that is above the snow surface, and thus needs to be removed, in order to apply calculations only to the snow (not the air!). _____no_output_____

#extract the part of the profile that is in the snow (i.e. remove air)
depth_surf=p.detect_surface()
depth_ground=p.detect_ground()
samples_snow=p.samples_within_distance(begin=depth_surf, end=depth_ground, relativize=False)
samples_snow.distance-=depth_surf
plt.plot(samples_snow.distance, samples_snow.force)
plt.title(p.name)
plt.ylabel('Force [N]')
plt.xlabel('Depth [mm]')
plt.show()_____no_output_____

The above part of the profile is just the part that is in the snow. PLEASE NOTE that the automated functions are not infallible, and need to be used with care. For now, these need to be compared back to the notes and interpreted manually. _____no_output_____The next step is the actual calculation of SSA from the force data. It then displays the data and lets you see that there is now a column called SSA! Note that this function is "proksch2015". You can read about how it works in Martin Proksch's paper {cite:p}`Proksch2015`._____no_output_____

# call using the snowmicropyn library proksch2015
p2015 = proksch2015.calc(p.samples)
p2015.head() #check out the first few values of SSA_____no_output_____

### 3. Read microCT data, and compare integrating sphere, SMP and CT data_____no_output_____The microCT samples were extracted in the field and processed at CRREL by Lauren Farnsworth, and is not yet published at NSIDC. Please contact her with questions (lauren.b.farnsworth@usace.army.mil)!

This module reads in microCT datafiles which are stored as text. Some additional data are available, showing the computer generated slices through the ice-air interface: contact Mike (durand.8@osu.edu) if you want to look at a subset of these data that Lauren has shared.

Equivalent grain size is a useful quantity to compare: because it's proportional to 1/SSA, and because after a point as you increase SSA more and more, all fine-grained snow acts more-or-less the same (converging to e.g. the "fine-grained" curve in Figure 1, above), we'll look at equivalent diameter instead of SSA in this comparison._____no_output_____

from read_CT_txt_files import read_CT_txt_files_____no_output_____# read micro CT for 2N13
data_dir='/tmp/microstructure/microCT/txt/'
[SSA_CT,height_min,height_max]=read_CT_txt_files(data_dir)

SSA_CT #check out the SSA values read in from MicroCT_____no_output_____# get data integrating sphere data for 2N13 and plot it
site='2N13'
engine_is, session_is = get_db(db_name)
q_is = session_is.query(LayerData).filter(LayerData.type == 'specific_surface_area').filter(LayerData.site_id.contains(site) )
df_is = query_to_geopandas(q_is, engine_is)
instruments_site = df_is['instrument'].unique()

# Loop over all the integrating sphere instruments at this site. plot equivalent diameter
fig,ax = plt.subplots()
for instr in instruments_site:

    # Grab our profile by site and instrument
    ind = df['site_id'] == site
    ind2 = df['instrument'] == instr
    profile = df.loc[ind & ind2].copy()

    # Don't plot it unless there is data
    if len(profile.index) > 0:

        # Sort by depth so samples that are take out of order won't mess up the plot
        profile = profile.sort_values(by='depth')

        # Layer profiles are always stored as strings.
        profile['value'] = 6/917/profile['value'].astype(float)*1000

        # Plot our profile
        ax.plot(profile['value'], profile['depth'], colors[instr], label=instr)

#All that's left to do is plot the CT and the SMP and label the plot!
ax.plot(6/917/SSA_CT*1000,height_min,label='microCT')        #CT data

ax.plot(6/917/p2015.P2015_ssa*1000,(max(p2015.distance)-p2015.distance)/10,label='SMP') #SMP data

# Labeling and plot style choices
ax.legend()
ax.set_xlabel('Equivalent diameter, mm')
ax.set_ylabel('Height above snow-soil interface [cm]')
ax.set_title('Site {}'.format(site.upper()))

plt.tight_layout()
plt.show()_____no_output_____

Wow, so the datasets are so very different, with the SMP being by far the most different. Comparing with {cite:p}`Calonne2020` shows that the SMP is off in the same direction as diagnosed in that paper. Thus, the difference is most likely due to the difference in SMP instruments. Dr. Mel Sandells of Northumbria University has a github branch of the SMP software SnowMicropyn that has the newer fit relationship integrated in the software. It would be a nice project to loop in Mel's branch with this notebook and see how well things compare to SnowEx data. I'd be happy to help anyone interested get rolling on that!

There's also significant differences between the microCT and the two integrating spheres. This is science - sometimes when we start intercomparing these quantities, we do not get a perfect match. This would also be a fascinating thing to explore in a Hackweek project._____no_output_____Some of the ways that you could imagine connecting microstructure measurements to other quantities would be with the SWESARR radar data. Although the radar data does seem to have some orthorectification issues that haven't been fully worked out, I can imagine these being worked around by careful choice of places you match up the microstructure to the radar. Note that places that are shallower tend to have larger Deq and vice versa, and the spatial variability in SSA was fairly low in general in Grand Mesa 2020, so looking at multiple SSA vs radar samples might not yield a great correlation. But you never know, could be fun to try! Generally speaking, we don't expect a ton of impact of the microstructure on L-band (UAVSAR), but it would be interesting to explore that.

One thing that could be of great value is to calibrate the SMP estimates of SSA to the integrating spheres. If you're interested in doing that, do reach out first. This could be a really interesting thing to explore!

It might also be interesting to compare the data to hand hardness measured in the snowpit, and to traditional hand lens measurements. _____no_output_____


# Notebook from ikhloud/Capstone-Project-Medical-Appointment-No-Shows
Path: Capstone_Project_Medical_Appointment_No_Shows.ipynb

# Capstone Project : Medical Appointment No Shows
## by (Kholoud )

## Table of Contents

Introduction

Data Wrangling

Exploratory Data Analysis

Conclusions

_____no_output_____
## Introduction

## Project Overview
This project is the capestone for the Data Scientist Nandegree Program from Udacity. I choose to work on a data set that contains Medical Appointment No Shows. This dataset collects information from 100k medical appointments in Brazil and is focused on the question of whether or not patients show up for their appointment . We will analyse the Dataset to find what factors are important for us to know in order to predict if a patient will show up for their scheduled appointment?

## Problem Statement
A person makes a doctor appointment, receives all the instructions and no-show appointment.
No-shows have a direct impact on individuals' health. When patients miss appointments, continuity of care is interrupted. Medication efficacy can't be monitored regularly. Preventive services and screenings can't be delivered in a timely manner. Acute illnesses are more likely to go untreated and become chronic conditions with complications. In short, missing an appointment can be severely detrimental to one's health.
So We will analyse the Dataset to find what factors are important for us to know in order to predict if a patient will show up for their scheduled appointment. And I will do descriptive statistics and visuals to answer questions and see hidden insights.

We are interested to answer the following eight questions:
- How many patients for female and male?
- What is the proportion of appointment show-up versus no shows?
- What is the distribution of the Age for the patient show-up and no-show the appointment?
- How many patients attended and did not attended the appointment with the Gender?
- Compare between the patient who show and didn't show the appointment and whether or not the patient is enrolled in Brasilian welfare program?
- How many patients attended and did not attended the appointment by received SMS?
- What are hospitals that have the highest number for patients who attended and didn't attend the appointment?
- Which disease Diabetes, Hypertension, Alcoholism, Handicap patients most showup in at appointment compared with Age ?
### Dataset Description:

The original dataset can be downloaded from Kaggle: https://www.kaggle.com/joniarroba/noshowappointments#KaggleV2-May-2016.csv

110.527 medical appointments its 14 associated variables (characteristics). The most important one if the patient show-up or no-show to the appointment.

The data is contained on 14 variable.

Here is the explanation of each variable :

- 01 - PatientId - Identification of a patient
- 02 - AppointmentID - Identification of each appointment
- 03 - Gender - Male or Female
- 04 - AppointmentDay - The day of the actuall appointment, when they have to visit the doctor.
- 05 - ScheduledDay - The day someone called or registered the appointment, this is before appointment Day.
- 06 - Age - How old is the patient.
- 07 - Neighbourhood - indicates the location of the hospital.
- 08 - Scholarship - indicates whether or not the patient is enrolled in Brasilian welfare program Bolsa Família.
- 09 - Hipertension - True or False
- 10 - Diabetes True or False
- 11 - Alcoholism True or False
- 12 - Handcap True or False
- 13 -SMS_received - 1 if messages sent to the patient else 0.
- 14 - No-show - Be careful about the encoding of "No-show": it says
**No** if the patient showed up to their appointment,**Yes** if they did not show up.
_____no_output_____

# import statements for all of the packages
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline_____no_output_____

## Data Wrangling
In this section, I will load the csv file, look into the data set, clean the data base on the 8 questions I am going to answer , and finall get a clean data set to be analyzed.

### General Properties

I will load the date set using pandas_____no_output_____

# Load your data and print out a few lines. Perform operations to inspect data
#   types and look for instances of missing or possibly errant data.
# load dataset
df_appoint = pd.read_csv('noshow_appointments.csv')
_____no_output_____#view dataset
# although, you can specify however many you'd like returned
df_appoint.head(10)_____no_output_____#this returns a tuple of the dimensions of the dataframe
df_appoint.shape_____no_output_____#this display a concise summary of the dataframe,
# inclouding the number of non-null values in each column
df_appoint.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 110527 entries, 0 to 110526
Data columns (total 14 columns):
PatientId         110527 non-null float64
AppointmentID     110527 non-null int64
Gender            110527 non-null object
ScheduledDay      110527 non-null object
AppointmentDay    110527 non-null object
Age               110527 non-null int64
Neighbourhood     110527 non-null object
Scholarship       110527 non-null int64
Hipertension      110527 non-null int64
Diabetes          110527 non-null int64
Alcoholism        110527 non-null int64
Handcap           110527 non-null int64
SMS_received      110527 non-null int64
No-show           110527 non-null object
dtypes: float64(1), int64(8), object(5)
memory usage: 11.8+ MB
#check if there is missing values
df_appoint.isnull().sum()_____no_output_____

#### we don't have missing values_____no_output_____

#check for duplictes row in the data
sum(df_appoint.duplicated())_____no_output_____

#### we don't have duplictes row _____no_output_____

#this returns the number of unique values in each column
df_appoint.describe()_____no_output_____

There is mines vlaue in age columns.It doesn't make sense._____no_output_____

#this returns the datatypes of the columns
df_appoint.dtypes_____no_output_____

#### we have two columns( ScheduledDay and AppointmentDay),
#### it's need to change the data types form object to datetime._____no_output_____
### Data Cleaning _____no_output_____#### Rename Columns_____no_output_____

# Rename No-show Column and correction spelling mistakes for some Columns
df_appoint.rename(columns={'No-show':'No_show', 'Hipertension':'Hypertension', 'Handcap':'Handicap'}, inplace=True )

# confirm changes
df_appoint.head(1)_____no_output_____

#### Fix ScheduledDay , AppointmentDay datatypes_____no_output_____

#Use the to_datetime function
#I change ScheduledDay cloumns from object to datetime
df_appoint['ScheduledDay'] =  pd.to_datetime(df_appoint['ScheduledDay'])_____no_output_____#Use the to_datetime function
#I change ScheduledDay cloumns from object to datetime
df_appoint['AppointmentDay'] =  pd.to_datetime(df_appoint['AppointmentDay'])_____no_output_____#confirm changes
df_appoint.dtypes_____no_output_____

#### Fix Age cloumn_____no_output_____

df_appoint[df_appoint['Age'] < 0]_____no_output_____# drop row
df_appoint.drop(df_appoint.index[99832], inplace=True)_____no_output_____#confirm changes
df_appoint.describe()_____no_output_____df_appoint.shape_____no_output_____

#### Convert data in No_show column from yes &No to 1,0 to easy handling with it
Here, the 'Yes' and 'No' categorical object data has been replaced with integers for use in statistical
computations._____no_output_____

# ‘No’ if the patient showed up to their appointment, and
# ‘Yes’ if they did not show up.
# ‘No’ = 0 and ‘Yes’ = 1
df_appoint.No_show[df_appoint['No_show'] == 'Yes'] = '1'
df_appoint.No_show[df_appoint['No_show'] == 'No'] = '0'C:\Users\khloud\Anaconda3\lib\site-packages\ipykernel_launcher.py:4: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  after removing the cwd from sys.path.
C:\Users\khloud\Anaconda3\lib\site-packages\ipykernel_launcher.py:5: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  """
#convert datatype from object to numrical
df_appoint['No_show'] = df_appoint['No_show'].astype(int)_____no_output_____df_appoint.head(3)_____no_output_____

## Exploratory Data Analysis
_____no_output_____### Univariate Exploration_____no_output_____
#### How many pationts for female and male ?_____no_output_____

# number gender patients
df_appoint.Gender.value_counts()  _____no_output_____#The number of female and male

sns.countplot(data = df_appoint, x = 'Gender', palette="Set1",  hue = df_appoint.Gender)
plt.title("The number of female and male")
plt.xlabel(" Gender of the pationt")
plt.ylabel("Number of Gender")
plt.show()_____no_output_____

From the gender graph, it can be seen that number of female patients (71839) and number of male patients (38687).

We can said the number of females are greater than of males patients_____no_output_____#### Age of the pationt

The most Distribution of Age from 0-5, 25-40 and 45 - 65_____no_output_____

df_appoint.Age.hist(alpha=0.5, bins=20, label='Age')
plt.title("Distribution of Age")
plt.xlabel(" Age of the pationt")
plt.ylabel("Number of Age")
plt.legend();_____no_output_____

#### What is the proportion of appointment show-up versus no shows?_____no_output_____

No_show_prop = df_appoint.No_show.value_counts('No_show')*100
No_show_prop_____no_output_____#plot(kind='pie')
No_show_proportion = df_appoint.No_show.value_counts()
labels = ['show-up', 'No Show']
explode = (0.1, 0)
colors = ['lightskyblue', 'crimson']

plt.pie(No_show_proportion , labels = labels, explode = explode, autopct= '%1.1f%%', counterclock=True,
        textprops={'color':'black'}, colors = colors , shadow=True);
plt.title(' The proportion of appointment show-up vs. no shows');_____no_output_____

The proportion of patients show-up appointment is 79.8%  and no showe appointment is 20.2% ._____no_output_____Total attended appointments is far greater than the number of appointments that patients did not show-up for. _____no_output_____## Bivariate Exploration_____no_output_____
### what is the distribution of the Age for the patient show-up and no-show the appointment?_____no_output_____Here, a filter has been created for use in statistical computations and graph creation. It filters based on
whether a patient showed up to their appointment (showup) or did not (noshow)._____no_output_____

#create mask
showup = df_appoint.No_show == False
noshow = df_appoint.No_show == True
_____no_output_____df_appoint.Age[showup].mean()_____no_output_____df_appoint.Age[noshow].mean()_____no_output_____#plot Distribution
df_appoint.Age[showup].hist(alpha=0.5, bins=20, label='showup', color='blue')
df_appoint.Age[noshow].hist(alpha=0.5, bins=20, label='noshow', color='red')
plt.title("Distribution of Age")
plt.xlabel(" Age of the pationt")
plt.ylabel("Number of showup and noshow")
plt.legend();_____no_output_____

> #### So, based on this plot, it does look like the really young children have a higher attend appointments then the Age from (45-65).

> #### the Age form (23 to 40) and (0-5) have a higher number didn't attend appointments._____no_output_____### How many patients attended and did not attended the appointment with the Gender?_____no_output_____

df_appoint.groupby('Gender').No_show.value_counts()_____no_output_____#The number of patients attended and did not attended the appointment
sns.countplot(data = df_appoint, x = 'Gender', palette="Set2",  hue = df_appoint.No_show)
plt.title("Distribution of Gender")
plt.xlabel(" Gender of the pationt")
plt.ylabel("Number of showup and noshow")
plt.show()_____no_output_____

> #### 0 means show-up ,1 means no show_____no_output_____#### The number of female who attended the appointment greater than male.
#### The number of female who didn't attend the appointment greater than male. _____no_output_____### compare between the patient who show and didn't show the appointment and whether or not the patient is enrolled in Brasilian welfare program ? _____no_output_____

#the patient show up the appointment
df_appoint.Scholarship[showup].value_counts().plot(kind='bar', alpha=0.8, color=['salmon','maroon'])
plt.title("the patient is enrolled in Brasilian welfare program ")
plt.xlabel("Scholarship")
plt.ylabel("Number of show up")
plt.xticks([0, 1],['unenrolled','enrolled'])
plt.legend();_____no_output_____Scholarship_show = df_appoint.Scholarship[showup].value_counts()
Scholarship_show_____no_output_____#the patient didn't show the appointment
df_appoint.Scholarship[noshow].value_counts().plot(kind='bar',alpha=0.8, color=['salmon','maroon'])
plt.title("the patient is enrolled in Brasilian welfare program ")
plt.xlabel("Scholarship")
plt.ylabel("Number of  no show")
plt.xticks([0, 1],['unenrolled','enrolled'])
plt.legend();_____no_output_____

> #### The most of the patient unenrolled in Brasilian welfare program are  attended the appointment._____no_output_____### How many patients attended and did not attended the appointment by received SMS?_____no_output_____

#patients attended the appointment by received SMS
df_appoint.SMS_received[showup].value_counts().plot(kind='bar', alpha=0.8, color=['crimson','plum'])
plt.title("the patient is received SMS ")
plt.xlabel("Scholarship")
plt.ylabel("Number of show up")
plt.xticks([0, 1],["Didn't received","received SMS"])
plt.legend();_____no_output_____#patients did not attend the appointment by received SMS
df_appoint.SMS_received[noshow].value_counts().plot(kind='bar', alpha=0.8, color=['crimson','plum'])
plt.title("the patient is received SMS ")
plt.xlabel("SMS_received")
plt.ylabel("Number of no show")
plt.xticks([0, 1],["Didn't received","received SMS"])
plt.legend();_____no_output_____

> #### SMS text message reminders are may not be effective in reducing the nonattendance rate _____no_output_____### What are hospitals that have the highest number for patients who attended and didn't attend the appointment ?_____no_output_____

# the names of hospitals that have the highest number for patients who attended the appointmen
df_appoint.Neighbourhood[showup].value_counts().plot(kind='bar', alpha=0.8, figsize=(17,17))
plt.title("location of the hospital ")
plt.xlabel("Neighbourhood")
plt.ylabel("Number of show up")
plt.legend();_____no_output_____

> #### The top 5 the names of hospitals that have the highest number for patients who attended the appointment
JARDIM CAMBURI (6200 patients)
MARIA ORTIZ(4700 patients)
RESISTENCIA(3500 patients)
JARDIM DA PENHA(3200 patients)
SANTA MARTHA(2700 patients)_____no_output_____

#The names of hospitals that have the highest number for patients who didn't attend the appointment
df_appoint.Neighbourhood[noshow].value_counts().plot(kind='bar', alpha=0.8, figsize=(17,17))
plt.title("location of the hospital ")
plt.xlabel("Neighbourhood")
plt.ylabel("Number of no show")
plt.legend();_____no_output_____

> #### The top 5 the names of hospitals that have the highest number for patients who didn't attend the appointment
JARDIM CAMBURI(1475 patients)
MARIA ORTIZ(1220 patients)
ITARARE(930 patients)
RESISTENCIA(700 patients )
CENTRO(690 patients)_____no_output_____## Multivariate Exploration_____no_output_____#### Which disease Diabetes, Hypertension, Alcoholism, Handicap patients most showup in at appointment compared with Age ?_____no_output_____

plt.figure(1)
plt.figure(figsize = (15,10))
disease_cols = ['Hypertension', 'Diabetes', 'Alcoholism', 'Handicap']
for col in disease_cols:
    plt.subplot(int(str(22)+str((disease_cols.index(col)+1))))
    df = pd.DataFrame()
    df['Age'] = range(115)
    Male = df.Age.apply(lambda x:len(df_appoint[(df_appoint.Age == x) & (df_appoint.Gender == 'M') & (df_appoint[col] == 1) & (df_appoint.No_show == 0 )]))
    Female = df.Age.apply(lambda x:len(df_appoint[(df_appoint.Age == x) & (df_appoint.Gender == 'F') & (df_appoint[col] == 1)& (df_appoint.No_show == 0)]))

    # plot
    plt.plot( df, Male , linestyle='solid', color='blue', linewidth=2)
    plt.plot( df, Female , linestyle='solid', color='lightcoral', linewidth=2)

    plt.legend(['Male','Female'])
    plt.xlabel('Age')
    plt.ylabel('Number of attended')
    plt.title('{} patients'.format(col));_____no_output_____

- Hypertension and Diabetes patients are attending the appointment when compared to other disease . Age ranges from 40 to 80 - with females conversion is better than Males.
- Alcoholism patients from Male have attending for appointments  are more compared to female and they are in range from 45 to 65.
-  Male and Female visits for Handicap patient are in the similar range. There isn't much of difference._____no_output_____
## Conclusions
_____no_output_____### Univariate Exploration
#### How many pationts for female and male ?

From the gender graph, it can be seen that number of female patients (71839) and number of male patients (38687).

We can said the number of females are greater than of males patients

#### What is the proportion of appointment show-up versus no shows?

The proportion of patients show-up appointment is 79.8%  and no showe appointment is 20.2% .

Total attended appointments is far greater than the number of appointments that patients did not show-up for. _____no_output_____## Bivariate Exploration

#### what is the distribution of the Age for the patient show-up and no-show the appointment?
>So, based on this plot, it does look like the really young children have a higher attend appointments then the Age from (45-65).

>the Age form (23 to 40) and (0-5) have a higher number didn't attend appointments.

#### How many patients attended and did not attended the appointment with the Gender?)
> The number of female who attended the appointment greater than male.
> The number of female who didn't attend the appointment greater than male.
> female takes way more care of they healthe in comparison to male.

#### compare between the patient who show and didn't show the appointment and whether or not the patient is enrolled in Brasilian welfare program ?
> The most of the patient unenrolled in Brasilian welfare program are  attended the appointment.

#### How many patients attended and did not attended the appointment by received SMS?
> SMS text message reminders are may not be effective in reducing the nonattendance rate.

#### What are hospitals that have the highest number for patients who attended and didn't attend the appointment ?
> The top 5 the names of hospitals that have the highest number for patients who attended the appointment

JARDIM CAMBURI (6200 patients)
MARIA ORTIZ(4700 patients)
RESISTENCIA(3500 patients)
JARDIM DA PENHA(3200 patients)
SANTA MARTHA(2700 patients)

> The top 5 the names of hospitals that have the highest number for patients who didn't attend the appointment

JARDIM CAMBURI(1475 patients)
MARIA ORTIZ(1220 patients)
ITARARE(930 patients)
RESISTENCIA(700 patients )
CENTRO(690 patients)

_____no_output_____## Multivariate Exploration

#### Which disease Diabetes, Hypertension, Alcoholism, Handicap patients most showup in at appointment compared with Age ?

- Hypertension and Diabetes patients are attending the appointment when compared to other disease . Age ranges from 40 to 80 - with females conversion is better than Males.
- Alcoholism patients from Male have attending for appointments  are more compared to female and they are in range from 45 to 65.
-  Male and Female visits for Handicap patient are in the similar range. There isn't much of difference.

_____no_output_____## Conclusions_____no_output_____Finally we can said the most factors are important for us to know in order to predict if a patient will show up for their scheduled appointments are  the patient unenrolled in Brasilian welfare program as we see most of people who has Scholarship are most likely to miss their appointments showing and patients who don't have a scholarship are show up for their appointments,gender is female, the Age from (45–65) and young children because older caretakers are responsible for them and type of disease Hypertension and Diabetes patients are attending the appointment when compared to other disease.
Surprisingly, the patients who did not receive sms regarding their appointment have higher number of patients showing up for their appointment.

The distribution of Show-up Rate v.s. SMS_received does not make sense. People who do receive a reminder SMS usually tend to visit the hospital more than those who haven't received an SMS. This discrepancy may cause by human error when collecting data. Or this could be an SMS that was recived while booking an appointment and could possibly have no correlation with the visit probability.

for improvement to this problem can build a model that predict if a patient will show up for their scheduled appointment._____no_output_____


# Notebook from hemanth-alva/coursera_MachineLearningWithPython
Path: ML0101EN-Clas-K-Nearest-neighbors-CustCat.ipynb

# K-Nearest Neighbors

Estimated time needed: **25** minutes

## Objectives

After completing this lab you will be able to:

-   Use K Nearest neighbors to classify data
_____no_output_____In this Lab you will load a customer dataset, fit the data, and use K-Nearest Neighbors to predict a data point. But what is **K-Nearest Neighbors**?
_____no_output_____**K-Nearest Neighbors** is an algorithm for supervised learning. Where the data is 'trained' with data points corresponding to their classification. Once a point is to be predicted, it takes into account the 'K' nearest points to it to determine it's classification.
_____no_output_____### Here's an visualization of the K-Nearest Neighbors algorithm.

_____no_output_____In this case, we have data points of Class A and B. We want to predict what the star (test data point) is. If we consider a k value of 3 (3 nearest data points) we will obtain a prediction of Class B. Yet if we consider a k value of 6, we will obtain a prediction of Class A.
_____no_output_____In this sense, it is important to consider the value of k. But hopefully from this diagram, you should get a sense of what the K-Nearest Neighbors algorithm is. It considers the 'K' Nearest Neighbors (points) when it predicts the classification of the test point.
_____no_output_____
Table of contents

About the dataset

Data Visualization and Analysis

Classification

_____no_output_____Lets load required libraries
_____no_output_____

import itertools
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.ticker import NullFormatter
import pandas as pd
import numpy as np
import matplotlib.ticker as ticker
from sklearn import preprocessing
%matplotlib inline_____no_output_____

About the dataset

_____no_output_____Imagine a telecommunications provider has segmented its customer base by service usage patterns, categorizing the customers into four groups. If demographic data can be used to predict group membership, the company can customize offers for individual prospective customers. It is a classification problem. That is, given the dataset,  with predefined labels, we need to build a model to be used to predict class of a new or unknown case.

The example focuses on using demographic data, such as region, age, and marital, to predict usage patterns.

The target field, called **custcat**, has four possible values that correspond to the four customer groups, as follows:
  1- Basic Service
  2- E-Service
  3- Plus Service
  4- Total Service

Our objective is to build a classifier, to predict the class of unknown cases. We will use a specific type of classification called K nearest neighbour.
_____no_output_____Lets download the dataset. To download the data, we will use !wget to download it from IBM Object Storage.
_____no_output_____

!wget -O teleCust1000t.csv https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/teleCust1000t.csv_____no_output_____

**Did you know?** When it comes to Machine Learning, you will likely be working with large datasets. As a business, where can you host your data? IBM is offering a unique opportunity for businesses, with 10 Tb of IBM Cloud Object Storage: [Sign up now for free](http://cocl.us/ML0101EN-IBM-Offer-CC)
_____no_output_____### Load Data From CSV File
_____no_output_____

df = pd.read_csv('teleCust1000t.csv')
df.head()_____no_output_____

Data Visualization and Analysis

_____no_output_____#### Let’s see how many of each class is in our data set
_____no_output_____

df['custcat'].value_counts()_____no_output_____

#### 281 Plus Service, 266 Basic-service, 236 Total Service, and 217 E-Service customers
_____no_output_____You can easily explore your data using visualization techniques:
_____no_output_____

df.hist(column='income', bins=50)_____no_output_____

### Feature set
_____no_output_____Lets define feature sets, X:
_____no_output_____

df.columns_____no_output_____

To use scikit-learn library, we have to convert the Pandas data frame to a Numpy array:
_____no_output_____

X = df[['region', 'tenure','age', 'marital', 'address', 'income', 'ed', 'employ','retire', 'gender', 'reside']] .values  #.astype(float)
X[0:5]
_____no_output_____

What are our labels?
_____no_output_____

y = df['custcat'].values
y[0:5]_____no_output_____

## Normalize Data
_____no_output_____Data Standardization give data zero mean and unit variance, it is good practice, especially for algorithms such as KNN which is based on distance of cases:
_____no_output_____

X = preprocessing.StandardScaler().fit(X).transform(X.astype(float))
X[0:5]_____no_output_____

### Train Test Split

Out of Sample Accuracy is the percentage of correct predictions that the model makes on data that that the model has NOT been trained on. Doing a train and test on the same dataset will most likely have low out-of-sample accuracy, due to the likelihood of being over-fit.

It is important that our models have a high, out-of-sample accuracy, because the purpose of any model, of course, is to make correct predictions on unknown data. So how can we improve out-of-sample accuracy? One way is to use an evaluation approach called Train/Test Split.
Train/Test Split involves splitting the dataset into training and testing sets respectively, which are mutually exclusive. After which, you train with the training set and test with the testing set.

This will provide a more accurate evaluation on out-of-sample accuracy because the testing dataset is not part of the dataset that have been used to train the data. It is more realistic for real world problems.
_____no_output_____

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=4)
print ('Train set:', X_train.shape,  y_train.shape)
print ('Test set:', X_test.shape,  y_test.shape)_____no_output_____

Classification

_____no_output_____
K nearest neighbor (KNN)

_____no_output_____#### Import library
_____no_output_____Classifier implementing the k-nearest neighbors vote.
_____no_output_____

from sklearn.neighbors import KNeighborsClassifier_____no_output_____

### Training

Lets start the algorithm with k=4 for now:
_____no_output_____

k = 4
#Train Model and Predict
neigh = KNeighborsClassifier(n_neighbors = k).fit(X_train,y_train)
neigh_____no_output_____

### Predicting

we can use the model to predict the test set:
_____no_output_____

yhat = neigh.predict(X_test)
yhat[0:5]_____no_output_____

### Accuracy evaluation

In multilabel classification, **accuracy classification score** is a function that computes subset accuracy. This function is equal to the jaccard_similarity_score function. Essentially, it calculates how closely the actual labels and predicted labels are matched in the test set.
_____no_output_____

from sklearn import metrics
print("Train set Accuracy: ", metrics.accuracy_score(y_train, neigh.predict(X_train)))
print("Test set Accuracy: ", metrics.accuracy_score(y_test, yhat))_____no_output_____

## Practice

Can you build the model again, but this time with k=6?
_____no_output_____

# write your code here

_____no_output_____

Double-click **here** for the solution.

_____no_output_____#### What about other K?

K in KNN, is the number of nearest neighbors to examine. It is supposed to be specified by the User. So, how can we choose right value for K?
The general solution is to reserve a part of your data for testing the accuracy of the model. Then chose k =1, use the training part for modeling, and calculate the accuracy of prediction using all samples in your test set. Repeat this process, increasing the k, and see which k is the best for your model.

We can calculate the accuracy of KNN for different Ks.
_____no_output_____

Ks = 10
mean_acc = np.zeros((Ks-1))
std_acc = np.zeros((Ks-1))
ConfustionMx = [];
for n in range(1,Ks):

    #Train Model and Predict
    neigh = KNeighborsClassifier(n_neighbors = n).fit(X_train,y_train)
    yhat=neigh.predict(X_test)
    mean_acc[n-1] = metrics.accuracy_score(y_test, yhat)

    std_acc[n-1]=np.std(yhat==y_test)/np.sqrt(yhat.shape[0])

mean_acc_____no_output_____

#### Plot  model accuracy  for Different number of Neighbors
_____no_output_____

plt.plot(range(1,Ks),mean_acc,'g')
plt.fill_between(range(1,Ks),mean_acc - 1 * std_acc,mean_acc + 1 * std_acc, alpha=0.10)
plt.legend(('Accuracy ', '+/- 3xstd'))
plt.ylabel('Accuracy ')
plt.xlabel('Number of Nabors (K)')
plt.tight_layout()
plt.show()_____no_output_____print( "The best accuracy was with", mean_acc.max(), "with k=", mean_acc.argmax()+1) _____no_output_____

Want to learn more?

IBM SPSS Modeler is a comprehensive analytics platform that has many machine learning algorithms. It has been designed to bring predictive intelligence to decisions made by individuals, by groups, by systems – by your enterprise as a whole. A free trial is available through this course, available here: SPSS Modeler

Also, you can use Watson Studio to run these notebooks faster with bigger datasets. Watson Studio is IBM's leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, Watson Studio enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of Watson Studio users today with a free account at Watson Studio
_____no_output_____### Thank you for completing this lab!

## Author

Saeed Aghabozorgi

### Other Contributors

Joseph Santarcangelo

## Change Log

| Date (YYYY-MM-DD) | Version | Changed By | Change Description                 |
| ----------------- | ------- | ---------- | ---------------------------------- |
| 2020-08-27        | 0.1     | Lavanya    | Moved lab to course repo in GitLab |
|                   |         |            |                                    |
|                   |         |            |                                    |

##
 © IBM Corporation 2020. All rights reserved.

_____no_output_____


# Notebook from M-Sender/cmps3160
Path: _projects/project0/Project0.ipynb

# CMPS3660 - Project0

This assignment is intended to ensure that you are able to edit and submit Jupyter Notebooks.

The goal is to write a function to count the frequency of different characters in a given string. For example: character_frequency('google') should return a `dict`: {'g': 2, 'o': 2, 'l': 1, 'e': 1} and then you should add some code to display some summary statistics about some data that you bring in!

There are many tutorials on the web that can show you how to use the Python data structures. Quite a few examples at: https://docs.python.org/3/tutorial/datastructures.html

If you have any trouble with this at all (and we are sure at least some of you will), don't worry and stop by our office hours.

You should edit the function definition below, make sure to press 'Shift + Enter' to execute it, and then execute the two function calls to check that the output is correct._____no_output_____

def character_frequency(s):
    ret = dict()
    for i in range(len(s)):
        if s[i] in ret:
            ret[s[i]] +=1
        else:
            ret[s[i]] = 1
    return ret_____no_output_____character_frequency('google')_____no_output_____character_frequency('This assignment is intended to ensure that you are able to edit and submit Jupyter Notebooks.')_____no_output_____

Now in the cell below you should create two string variables and inside them put a different quotes (at least 10 words long) from a favorite book, movie, tv show, or any other media._____no_output_____

movie1 = "You were the chosen one! It was said that you would destroy the Sith, not join them! Bring balance to the force, not leave it in darkness! You were my brother Anakin! I loved you!"
movie2 = "Señores, Les voy a decir quién soy Yo soy Pablo Emilio Escobar Gaviria Mis ojos están en todos lados o sea ustedes no pueden hacer una puta sola mierda en el departamento de Antioquia que yo me entere. Sí, señores. No pueden mover un dedo. Un día, yo voy a ser presidente de la República de Colombia. Y bien, me gano la vida haciendo negocios. Así que pues, fresco. Tranquilo. Uds pueden aceptar mi negocio o aceptar las consecuencias. Plata o pomo Ustedes eligen"_____no_output_____

For each of the above quotes add code below to print out the character frequency and then give a set of summary statistics for each one, specifically the letter(s) that occur the fewest number of times, the letter(s) that occur the most times, and the average count for all the letters. You should display this data neatly._____no_output_____

def getPercents(data):
    total = sum(data.values())
    print("There were a total of",total,"characters comprising of",len(data),"unique characters.")
    for key in data:
        percent = str(100*data[key]/total)[:4]
        print("\tCharacter '",key,"' had ",data[key]," appeareances and comprised ",percent,"% of the dataset.",sep="")
def analysis(data):
    print("The three most frequent characters were:")
    for i in range(3):
        key = max(data,key = data.get)
        freq = data[key]
        del data[key]
        print("\tThe character '",key,"' appeared ",freq," times.",sep="")
    print("The three least frequent characters were:")
    for i in range(3):
        key = min(data,key = data.get)
        freq = data[key]
        del data[key]
        print("\tThe character '",key,"' only appeared ",freq," times.",sep="")
def overall_analysis(dataArr):
    #merge all data to one dict
    data = dict()
    for dictionary in dataArr:
        for key in dictionary:
            if key in data:
                data[key] += dictionary[key]
            else:
                data[key] = dictionary[key]
    analysis(data.copy())
    getPercents(data.copy())
data1 = character_frequency(movie1)
print("Star Wars' Quote:",data1)
analysis(data1.copy())
data2 = character_frequency(movie2)
print("Narcos' Quote:", data2)
analysis(data2.copy())
print("_______________________________________________\nCombining all of the data\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _")
overall_analysis([data1.copy(),data2.copy()])

Star Wars' Quote: {'Y': 2, 'o': 15, 'u': 5, ' ': 34, 'w': 4, 'e': 18, 'r': 8, 't': 14, 'h': 8, 'c': 3, 's': 6, 'n': 11, '!': 5, 'I': 2, 'a': 8, 'i': 7, 'd': 5, 'y': 4, 'l': 4, 'S': 1, ',': 2, 'j': 1, 'm': 2, 'B': 1, 'g': 1, 'b': 2, 'f': 1, 'v': 2, 'k': 2, 'A': 1}
The three most frequent characters were:
	The character ' ' appeared 34 times.
	The character 'e' appeared 18 times.
	The character 'o' appeared 15 times.
The three least frequent characters were:
	The character 'S' only appeared 1 times.
	The character 'j' only appeared 1 times.
	The character 'B' only appeared 1 times.
Narcos' Quote: {'S': 2, 'e': 54, 'ñ': 2, 'o': 40, 'r': 16, 's': 28, ',': 5, ' ': 83, 'L': 1, 'v': 5, 'y': 6, 'a': 33, 'd': 20, 'c': 13, 'i': 22, 'q': 5, 'u': 14, 'é': 1, 'n': 24, 'Y': 2, 'P': 2, 'b': 5, 'l': 13, 'E': 2, 'm': 9, 'G': 1, 'M': 1, 'j': 1, 't': 13, 'á': 1, 'p': 11, 'h': 2, 'A': 2, '.': 8, 'í': 3, 'N': 1, 'U': 3, 'R': 1, 'ú': 1, 'C': 1, 'g': 4, 'f': 1, 'T': 1}
The three most frequent characters were:
	The character ' ' appeared 83 times.
	The character 'e' appeared 54 times.
	The character 'o' appeared 40 times.
The three least frequent characters were:
	The character 'L' only appeared 1 times.
	The character 'é' only appeared 1 times.
	The character 'G' only appeared 1 times.
_______________________________________________
Combining all of the data
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
The three most frequent characters were:
	The character ' ' appeared 117 times.
	The character 'e' appeared 72 times.
	The character 'o' appeared 55 times.
The three least frequent characters were:
	The character 'B' only appeared 1 times.
	The character 'L' only appeared 1 times.
	The character 'é' only appeared 1 times.
There were a total of 642 characters comprising of 48 unique characters.
	Character 'Y' had 4 appeareances and comprised 0.62% of the dataset.
	Character 'o' had 55 appeareances and comprised 8.56% of the dataset.
	Character 'u' had 19 appeareances and comprised 2.95% of the dataset.
	Character ' ' had 117 appeareances and comprised 18.2% of the dataset.
	Character 'w' had 4 appeareances and comprised 0.62% of the dataset.
	Character 'e' had 72 appeareances and comprised 11.2% of the dataset.
	Character 'r' had 24 appeareances and comprised 3.73% of the dataset.
	Character 't' had 27 appeareances and comprised 4.20% of the dataset.
	Character 'h' had 10 appeareances and comprised 1.55% of the dataset.
	Character 'c' had 16 appeareances and comprised 2.49% of the dataset.
	Character 's' had 34 appeareances and comprised 5.29% of the dataset.
	Character 'n' had 35 appeareances and comprised 5.45% of the dataset.
	Character '!' had 5 appeareances and comprised 0.77% of the dataset.
	Character 'I' had 2 appeareances and comprised 0.31% of the dataset.
	Character 'a' had 41 appeareances and comprised 6.38% of the dataset.
	Character 'i' had 29 appeareances and comprised 4.51% of the dataset.
	Character 'd' had 25 appeareances and comprised 3.89% of the dataset.
	Character 'y' had 10 appeareances and comprised 1.55% of the dataset.
	Character 'l' had 17 appeareances and comprised 2.64% of the dataset.
	Character 'S' had 3 appeareances and comprised 0.46% of the dataset.
	Character ',' had 7 appeareances and comprised 1.09% of the dataset.
	Character 'j' had 2 appeareances and comprised 0.31% of the dataset.
	Character 'm' had 11 appeareances and comprised 1.71% of the dataset.
	Character 'B' had 1 appeareances and comprised 0.15% of the dataset.
	Character 'g' had 5 appeareances and comprised 0.77% of the dataset.
	Character 'b' had 7 appeareances and comprised 1.09% of the dataset.
	Character 'f' had 2 appeareances and comprised 0.31% of the dataset.
	Character 'v' had 7 appeareances and comprised 1.09% of the dataset.
	Character 'k' had 2 appeareances and comprised 0.31% of the dataset.
	Character 'A' had 3 appeareances and comprised 0.46% of the dataset.
	Character 'ñ' had 2 appeareances and comprised 0.31% of the dataset.
	Character 'L' had 1 appeareances and comprised 0.15% of the dataset.
	Character 'q' had 5 appeareances and comprised 0.77% of the dataset.
	Character 'é' had 1 appeareances and comprised 0.15% of the dataset.
	Character 'P' had 2 appeareances and comprised 0.31% of the dataset.
	Character 'E' had 2 appeareances and comprised 0.31% of the dataset.
	Character 'G' had 1 appeareances and comprised 0.15% of the dataset.
	Character 'M' had 1 appeareances and comprised 0.15% of the dataset.
	Character 'á' had 1 appeareances and comprised 0.15% of the dataset.
	Character 'p' had 11 appeareances and comprised 1.71% of the dataset.
	Character '.' had 8 appeareances and comprised 1.24% of the dataset.
	Character 'í' had 3 appeareances and comprised 0.46% of the dataset.
	Character 'N' had 1 appeareances and comprised 0.15% of the dataset.
	Character 'U' had 3 appeareances and comprised 0.46% of the dataset.
	Character 'R' had 1 appeareances and comprised 0.15% of the dataset.
	Character 'ú' had 1 appeareances and comprised 0.15% of the dataset.
	Character 'C' had 1 appeareances and comprised 0.15% of the dataset.
	Character 'T' had 1 appeareances and comprised 0.15% of the dataset.