# Notebook from emiliom/echopype-examples
Path: notebooks/ms_PacificHake_EK60_cruisetracks.ipynb

# Echopype demonstration notebook 1
## Exploring ship echosounder data from the Pacific Hake survey_____no_output_____Jupyter notebook accompanying the manuscript:

_Echopype: A Python library for interoperable and scalable processing of ocean sonar data for biological information_
_Authors: Wu-Jung Lee, Emilio Mayorga, Landung Setiawan, Kavin Nguyen, Imran Majeed, Valentina Staneva_

### Goals

- Illustrate a common workflow for echosounder data conversion, calibration and use. This workflow leverages the standardization applied by echopype and the power, ease of use and familiarity of libraries in the scientific Python ecosystem.
- Extract and visualize data with relative ease using geospatial and temporal filters.

### Description

This notebook uses EK60 echosounder data collected during the [2017 Joint U.S.-Canada Integrated Ecosystem and Pacific Hake Acoustic Trawl Survey ('Pacific Hake Survey')](https://www.fisheries.noaa.gov/west-coast/science-data/joint-us-canada-integrated-ecosystem-and-pacific-hake-acoustic-trawl-survey) to illustrate a common workflow for data conversion, calibration and analysis using [`echopype`](https://echopype.readthedocs.io) and core scientific Python software packages, particularly `xarray`, `GeoPandas`, `pandas` and `NumPy`.

Two days of cloud-hosted `.raw` data files are accessed by echopype directly from an [Amazon Web Services (AWS)](https://aws.amazon.com/) S3 "bucket" maintained by the [NOAA NCEI Water-Column Sonar Data Archive](https://registry.opendata.aws/ncei-wcsd-archive/). The total data used are 170 `.raw` files at approximately 25 MB each (1 Hz pinging rate from first light to dusk), corresponding to approximately 4.2 GB. With `echopype`, each file is converted to a standardized representation based on the [SONAR-netCDF4 v1.0 convention](https://github.com/ices-publications/SONAR-netCDF4/) and saved to the cloud-optimized [Zarr](https://zarr.readthedocs.io) format.

Data stored in the netCDF-based SONAR-netCDF4 convention can be conveniently and intuitively manipulated with `xarray` in combination with related scientific Python packages. [Mean Volume Backscattering Strength (MVBS)](http://www.fao.org/3/X5818E/x5818e07.htm) is computed with `echopype` from each raw data file and exported to a netCDF file. Here, we define two geographical bounding boxes encompassing two ship tracks and use these to extract corresponding timestamp intervals from the GPS data, and then the corresponding MVBS data based on those intervals. Finally, these extracted MVBS subsets are plotted as track echograms.

### Outline

1. Establish AWS S3 file system connection and generate list of target EK60 `.raw` files
2. Process S3-hosted raw files with `echopype`: convert, calibrate and export to standardized files
3. Extract and process GPS locations from the `Platform` group of converted raw files
4. Read MVBS and plot track echograms for time periods corresponding to two ship tracks

### Running the notebook

This notebook can be run with a conda environment created using the conda environment file https://github.com/OSOceanAcoustics/echopype-examples/blob/echopype_paper/binder/environment.yml. The notebook creates two directories, if not already present: `./exports/hakesurvey_convertednc` and `./exports/hakesurvey_calibratednc`. netCDF and Zarr files will be exported there._____no_output_____

from pathlib import Path

import fsspec
import numpy as np
import geopandas as gpd
import xarray as xr

import matplotlib.pyplot as plt
from shapely.geometry import box
import cartopy.crs as ccrs
import cartopy.io.img_tiles as cimgt
from cartopy.mpl.gridliner import LONGITUDE_FORMATTER, LATITUDE_FORMATTER

import echopype as ep

import warnings
warnings.simplefilter("ignore", category=DeprecationWarning)_____no_output_____

## Establish AWS S3 file system connection and generate list of target EK60 `.raw` files

Access and inspect the publicly accessible NCEI WCSD S3 bucket on the AWS cloud as if it were a local file system. This will be done through the Python [fsspec](https://filesystem-spec.readthedocs.io) file system and bytes storage interface. We will use `fsspec.filesystem.glob` (`fs.glob`) to generate a list of all EK60 `.raw` data files in the bucket, then filter on file names for target dates of interest.

The directory path on the [ncei-wcsd-archive S3 bucket](https://ncei-wcsd-archive.s3.amazonaws.com/index.html) is `s3://ncei-wcsd-archive/data/raw/Bell_M._Shimada/SH1707/EK60/`. All `.raw` files from the 2017 Hake survey cruise are found here._____no_output_____

fs = fsspec.filesystem('s3', anon=True)

bucket = "ncei-wcsd-archive"
rawdirpath = "data/raw/Bell_M._Shimada/SH1707/EK60"_____no_output_____s3rawfiles = fs.glob(f"{bucket}/{rawdirpath}/*.raw")

# print out the last two S3 raw file paths in the list
s3rawfiles[-2:]_____no_output_____

Generate list of target EK60 `.raw` files from AWS S3 bucket based on dates. The dates are found in the middle string token (e.g., "D20170913"). Select files from 2 days, 2017-07-28 and 2017-07-29._____no_output_____

s3rawfiles = [
    s3path for s3path in s3rawfiles
    if any([f"D2017{datestr}" in s3path for datestr in ['0728', '0729']])
]

print(f"There are {len(s3rawfiles)} target raw files available")There are 170 target raw files available

## Process S3-hosted raw files with `echopype`: convert, calibrate and export to standardized files

Loop through all the selected raw files on S3 and convert, calibrate and generate Mean Volume Backscattering Strength (MVBS). Save the raw converted and MVBS data to local files, as zarr and netCDF, respectively._____no_output_____

def populate_metadata(ed, raw_fname):
    """
    Manually populate into the "ed" EchoData object
    additional metadata about the dataset and the platform
    """

    # -- SONAR-netCDF4 Top-level Group attributes
    survey_name = (
        "2017 Joint U.S.-Canada Integrated Ecosystem and "
        "Pacific Hake Acoustic Trawl Survey ('Pacific Hake Survey')"
    )
    ed.top.attrs['title'] = f"{survey_name}, file {raw_fname}"
    ed.top.attrs['summary'] = (
        f"EK60 raw file {raw_fname} from the {survey_name}, converted to a SONAR-netCDF4 file using echopype."
        "Information about the survey program is available at "
        "https://www.fisheries.noaa.gov/west-coast/science-data/"
        "joint-us-canada-integrated-ecosystem-and-pacific-hake-acoustic-trawl-survey"
    )

    # -- SONAR-netCDF4 Platform Group attributes
    # Per SONAR-netCDF4, for platform_type see https://vocab.ices.dk/?ref=311
    ed.platform.attrs['platform_type'] = "Research vessel"
    ed.platform.attrs['platform_name'] = "Bell M. Shimada"  # A NOAA ship
    ed.platform.attrs['platform_code_ICES'] = "315"_____no_output_____

Create the directories where the exported files will be saved, if these directories don't already exist._____no_output_____

base_dpath = Path('./exports')
base_dpath.mkdir(exist_ok=True)

converted_dpath = Path(base_dpath / 'hakesurvey_convertednc')
converted_dpath.mkdir(exist_ok=True)
calibrated_dpath = (base_dpath / 'hakesurvey_calibratednc')
calibrated_dpath.mkdir(exist_ok=True)_____no_output_____

### echopype processing

`EchoData` is an echopype object for conveniently handling raw converted data from either raw instrument files or previously converted and standardized raw netCDF4 and Zarr files. It is essentially a container for multiple `xarray.Dataset` objects, each corresponds to one of the netCDF4 groups specified in the SONAR-netCDF4 convention -- the convention followed by echopype. The `EchoData` object can be used to conveniently accesse and explore the echosounder raw data and for calibration and other processing.

The cell below contains the main echopype workflow steps. For each raw file:
- Access file directly from S3 via `ep.open_raw` to create a converted `EchoData` object in memory
- Add global and platform attributes to `EchoData` object
- Export to a local Zarr dataset (a collection of files encapsulated in a directory)
- Generate calibrated `Sv` and then `MVBS` from the raw data in the `EchoData` object
- Export `MVBS` to a local netcdf file

Note: Depending on your internet speed, this cell may take some time to run (potentially 20-30 mins)._____no_output_____

%%time

for s3rawfpath in s3rawfiles:
    raw_fpath = Path(s3rawfpath)
    try:
        # Access file directly from S3 to create a converted EchoData object in memory
        ed = ep.open_raw(
            f"s3://{s3rawfpath}",
            sonar_model='EK60',
            storage_options={'anon': True}
        )
        # Manually populate additional metadata about the dataset and the platform
        populate_metadata(ed, raw_fpath.name)

        # Save to converted Zarr format
        ed.to_zarr(save_path=converted_dpath, overwrite=True)

        # Use the EchoData object "ed" to generate calibrated and
        # computed MVBS files that will be saved to netcdf
        ds_Sv = ep.calibrate.compute_Sv(ed)
        ds_MVBS = ep.preprocess.compute_MVBS(
            ds_Sv,
            range_meter_bin=5,  # in meters
            ping_time_bin='20s'  # in seconds
        )
        ds_MVBS.to_netcdf(calibrated_dpath / f"MVBS_{raw_fpath.stem}.nc")
    except Exception as e:
        print(f"Failed to process raw file {raw_fpath.name}: {e}")01:43:16  parsing file Summer2017-D20170728-T000534.raw, time of first ping: 2017-Jul-28 00:05:34
01:43:26  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T000534.zarr
01:43:33  parsing file Summer2017-D20170728-T002507.raw, time of first ping: 2017-Jul-28 00:25:07
01:43:42  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T002507.zarr
01:43:46  parsing file Summer2017-D20170728-T004438.raw, time of first ping: 2017-Jul-28 00:44:38
01:43:55  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T004438.zarr
01:43:59  parsing file Summer2017-D20170728-T010354.raw, time of first ping: 2017-Jul-28 01:03:54
01:44:12  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T010354.zarr
01:44:17  parsing file Summer2017-D20170728-T012241.raw, time of first ping: 2017-Jul-28 01:22:41
01:44:25  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T012241.zarr
01:44:29  parsing file Summer2017-D20170728-T014121.raw, time of first ping: 2017-Jul-28 01:41:21
01:44:38  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T014121.zarr
01:44:42  parsing file Summer2017-D20170728-T015946.raw, time of first ping: 2017-Jul-28 01:59:46
01:44:47  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T015946.zarr
01:44:52  parsing file Summer2017-D20170728-T021051.raw, time of first ping: 2017-Jul-28 02:10:51
01:45:02  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T021051.zarr
01:45:07  parsing file Summer2017-D20170728-T022147.raw, time of first ping: 2017-Jul-28 02:21:47
01:45:15  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T022147.zarr
01:45:20  parsing file Summer2017-D20170728-T023241.raw, time of first ping: 2017-Jul-28 02:32:41
01:45:36  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T023241.zarr
01:45:40  parsing file Summer2017-D20170728-T024336.raw, time of first ping: 2017-Jul-28 02:43:36
01:45:49  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T024336.zarr
01:45:53  parsing file Summer2017-D20170728-T025431.raw, time of first ping: 2017-Jul-28 02:54:31
01:46:02  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T025431.zarr
01:46:06  parsing file Summer2017-D20170728-T030526.raw, time of first ping: 2017-Jul-28 03:05:26
01:46:19  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T030526.zarr
01:46:24  parsing file Summer2017-D20170728-T031620.raw, time of first ping: 2017-Jul-28 03:16:20
01:46:39  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T031620.zarr
01:46:44  parsing file Summer2017-D20170728-T032715.raw, time of first ping: 2017-Jul-28 03:27:15
01:46:53  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T032715.zarr
01:46:57  parsing file Summer2017-D20170728-T033809.raw, time of first ping: 2017-Jul-28 03:38:09
01:47:07  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T033809.zarr
01:47:11  parsing file Summer2017-D20170728-T034904.raw, time of first ping: 2017-Jul-28 03:49:04
01:47:21  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T034904.zarr
01:47:27  parsing file Summer2017-D20170728-T035959.raw, time of first ping: 2017-Jul-28 03:59:59
01:47:36  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T035959.zarr
01:47:41  parsing file Summer2017-D20170728-T041053.raw, time of first ping: 2017-Jul-28 04:10:53
01:47:51  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T041053.zarr
01:47:56  parsing file Summer2017-D20170728-T042149.raw, time of first ping: 2017-Jul-28 04:21:49
01:48:06  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T042149.zarr
01:48:11  parsing file Summer2017-D20170728-T043243.raw, time of first ping: 2017-Jul-28 04:32:43
01:48:22  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T043243.zarr
01:48:26  parsing file Summer2017-D20170728-T044338.raw, time of first ping: 2017-Jul-28 04:43:38
01:48:37  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T044338.zarr
01:48:42  parsing file Summer2017-D20170728-T045433.raw, time of first ping: 2017-Jul-28 04:54:33
01:48:52  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T045433.zarr
01:48:57  parsing file Summer2017-D20170728-T050528.raw, time of first ping: 2017-Jul-28 05:05:28
01:49:08  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T050528.zarr
01:49:13  parsing file Summer2017-D20170728-T051622.raw, time of first ping: 2017-Jul-28 05:16:22
01:49:23  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T051622.zarr
01:49:28  parsing file Summer2017-D20170728-T052717.raw, time of first ping: 2017-Jul-28 05:27:17
01:49:39  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T052717.zarr
01:49:44  parsing file Summer2017-D20170728-T053812.raw, time of first ping: 2017-Jul-28 05:38:12
01:49:54  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T053812.zarr
01:49:59  parsing file Summer2017-D20170728-T054907.raw, time of first ping: 2017-Jul-28 05:49:07
01:50:09  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T054907.zarr
01:50:14  parsing file Summer2017-D20170728-T060002.raw, time of first ping: 2017-Jul-28 06:00:02
01:50:32  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T060002.zarr
01:50:37  parsing file Summer2017-D20170728-T061057.raw, time of first ping: 2017-Jul-28 06:10:57
01:50:48  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T061057.zarr
01:50:53  parsing file Summer2017-D20170728-T062152.raw, time of first ping: 2017-Jul-28 06:21:52
01:51:04  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T062152.zarr
01:51:09  parsing file Summer2017-D20170728-T063247.raw, time of first ping: 2017-Jul-28 06:32:47
01:51:19  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T063247.zarr
01:51:24  parsing file Summer2017-D20170728-T064342.raw, time of first ping: 2017-Jul-28 06:43:42
01:51:35  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T064342.zarr
01:51:40  parsing file Summer2017-D20170728-T065438.raw, time of first ping: 2017-Jul-28 06:54:38
01:51:51  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T065438.zarr
01:51:56  parsing file Summer2017-D20170728-T070533.raw, time of first ping: 2017-Jul-28 07:05:33
01:52:08  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T070533.zarr
01:52:12  parsing file Summer2017-D20170728-T071628.raw, time of first ping: 2017-Jul-28 07:16:28
01:52:22  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T071628.zarr
01:52:26  parsing file Summer2017-D20170728-T072728.raw, time of first ping: 2017-Jul-28 07:27:28
01:52:37  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T072728.zarr
01:52:41  parsing file Summer2017-D20170728-T073827.raw, time of first ping: 2017-Jul-28 07:38:27
01:52:51  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T073827.zarr
01:52:55  parsing file Summer2017-D20170728-T074927.raw, time of first ping: 2017-Jul-28 07:49:27
01:53:05  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T074927.zarr
01:53:10  parsing file Summer2017-D20170728-T080026.raw, time of first ping: 2017-Jul-28 08:00:26
01:53:19  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T080026.zarr
01:53:23  parsing file Summer2017-D20170728-T081126.raw, time of first ping: 2017-Jul-28 08:11:26
01:53:34  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T081126.zarr
01:53:38  parsing file Summer2017-D20170728-T082225.raw, time of first ping: 2017-Jul-28 08:22:25
01:53:48  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T082225.zarr
01:53:53  parsing file Summer2017-D20170728-T083325.raw, time of first ping: 2017-Jul-28 08:33:25
01:54:06  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T083325.zarr
01:54:11  parsing file Summer2017-D20170728-T084424.raw, time of first ping: 2017-Jul-28 08:44:24
01:54:23  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T084424.zarr
01:54:27  parsing file Summer2017-D20170728-T085524.raw, time of first ping: 2017-Jul-28 08:55:24
01:54:38  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T085524.zarr
01:54:42  parsing file Summer2017-D20170728-T090623.raw, time of first ping: 2017-Jul-28 09:06:23
01:54:53  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T090623.zarr
01:54:57  parsing file Summer2017-D20170728-T091722.raw, time of first ping: 2017-Jul-28 09:17:22
01:55:10  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T091722.zarr
01:55:14  parsing file Summer2017-D20170728-T092822.raw, time of first ping: 2017-Jul-28 09:28:22
01:55:26  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T092822.zarr
01:55:31  parsing file Summer2017-D20170728-T093921.raw, time of first ping: 2017-Jul-28 09:39:21
01:55:45  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T093921.zarr
01:55:49  parsing file Summer2017-D20170728-T095020.raw, time of first ping: 2017-Jul-28 09:50:20
01:55:59  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T095020.zarr
01:56:04  parsing file Summer2017-D20170728-T100117.raw, time of first ping: 2017-Jul-28 10:01:17
01:56:12  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T100117.zarr
01:56:16  parsing file Summer2017-D20170728-T101212.raw, time of first ping: 2017-Jul-28 10:12:12
01:56:33  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T101212.zarr
01:56:40  parsing file Summer2017-D20170728-T102306.raw, time of first ping: 2017-Jul-28 10:23:06
01:56:50  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T102306.zarr
01:56:54  parsing file Summer2017-D20170728-T103401.raw, time of first ping: 2017-Jul-28 10:34:01
01:57:03  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T103401.zarr
01:57:13  parsing file Summer2017-D20170728-T104456.raw, time of first ping: 2017-Jul-28 10:44:56
01:57:23  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T104456.zarr
01:57:28  parsing file Summer2017-D20170728-T105551.raw, time of first ping: 2017-Jul-28 10:55:51
01:57:37  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T105551.zarr
01:57:42  parsing file Summer2017-D20170728-T110646.raw, time of first ping: 2017-Jul-28 11:06:46
01:57:51  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T110646.zarr
01:57:56  parsing file Summer2017-D20170728-T111742.raw, time of first ping: 2017-Jul-28 11:17:42
01:58:10  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T111742.zarr
01:58:15  parsing file Summer2017-D20170728-T112836.raw, time of first ping: 2017-Jul-28 11:28:36
01:58:25  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T112836.zarr
01:58:29  parsing file Summer2017-D20170728-T113931.raw, time of first ping: 2017-Jul-28 11:39:31
01:58:43  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T113931.zarr
01:58:48  parsing file Summer2017-D20170728-T115026.raw, time of first ping: 2017-Jul-28 11:50:26
01:58:58  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T115026.zarr
01:59:03  parsing file Summer2017-D20170728-T120121.raw, time of first ping: 2017-Jul-28 12:01:21
01:59:13  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T120121.zarr
01:59:18  parsing file Summer2017-D20170728-T121216.raw, time of first ping: 2017-Jul-28 12:12:16
01:59:29  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T121216.zarr
01:59:33  parsing file Summer2017-D20170728-T122311.raw, time of first ping: 2017-Jul-28 12:23:11
01:59:33  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T122311.zarr
01:59:37  parsing file Summer2017-D20170728-T122525.raw, time of first ping: 2017-Jul-28 12:25:25
01:59:49  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T122525.zarr
01:59:54  parsing file Summer2017-D20170728-T124210.raw, time of first ping: 2017-Jul-28 12:42:10
02:00:07  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T124210.zarr
02:00:12  parsing file Summer2017-D20170728-T130032.raw, time of first ping: 2017-Jul-28 13:00:32
02:00:27  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T130032.zarr
02:00:32  parsing file Summer2017-D20170728-T131854.raw, time of first ping: 2017-Jul-28 13:18:54
02:00:41  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T131854.zarr
02:00:46  parsing file Summer2017-D20170728-T133734.raw, time of first ping: 2017-Jul-28 13:37:34
02:01:00  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T133734.zarr
02:01:04  parsing file Summer2017-D20170728-T135621.raw, time of first ping: 2017-Jul-28 13:56:21
02:01:13  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T135621.zarr
02:01:18  parsing file Summer2017-D20170728-T141519.raw, time of first ping: 2017-Jul-28 14:15:19
02:01:27  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T141519.zarr
02:01:31  parsing file Summer2017-D20170728-T143451.raw, time of first ping: 2017-Jul-28 14:34:51
02:01:40  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T143451.zarr
02:01:48  parsing file Summer2017-D20170728-T145441.raw, time of first ping: 2017-Jul-28 14:54:41
02:01:58  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T145441.zarr
02:02:03  parsing file Summer2017-D20170728-T151434.raw, time of first ping: 2017-Jul-28 15:14:34
02:02:12  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T151434.zarr
02:02:16  parsing file Summer2017-D20170728-T153425.raw, time of first ping: 2017-Jul-28 15:34:25
02:02:26  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T153425.zarr
02:02:30  parsing file Summer2017-D20170728-T155417.raw, time of first ping: 2017-Jul-28 15:54:17
02:02:41  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T155417.zarr
02:02:45  parsing file Summer2017-D20170728-T161408.raw, time of first ping: 2017-Jul-28 16:14:08
02:02:56  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T161408.zarr
02:03:00  parsing file Summer2017-D20170728-T163359.raw, time of first ping: 2017-Jul-28 16:33:59
02:03:10  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T163359.zarr
02:03:14  parsing file Summer2017-D20170728-T165350.raw, time of first ping: 2017-Jul-28 16:53:50
02:03:24  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T165350.zarr
02:03:28  parsing file Summer2017-D20170728-T171341.raw, time of first ping: 2017-Jul-28 17:13:41
02:03:38  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T171341.zarr
02:03:42  parsing file Summer2017-D20170728-T173332.raw, time of first ping: 2017-Jul-28 17:33:32
02:03:52  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T173332.zarr
02:03:57  parsing file Summer2017-D20170728-T175338.raw, time of first ping: 2017-Jul-28 17:53:38
02:04:07  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T175338.zarr
02:04:11  parsing file Summer2017-D20170728-T181619.raw, time of first ping: 2017-Jul-28 18:16:19
02:04:21  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T181619.zarr
02:04:25  parsing file Summer2017-D20170728-T184131.raw, time of first ping: 2017-Jul-28 18:41:31
02:04:35  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T184131.zarr
02:04:40  parsing file Summer2017-D20170728-T190728.raw, time of first ping: 2017-Jul-28 19:07:28
02:04:50  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T190728.zarr
02:04:56  parsing file Summer2017-D20170728-T193459.raw, time of first ping: 2017-Jul-28 19:34:59
02:05:03  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T193459.zarr
02:05:07  parsing file Summer2017-D20170728-T195219.raw, time of first ping: 2017-Jul-28 19:52:19
02:05:25  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T195219.zarr
02:05:31  parsing file Summer2017-D20170728-T202608.raw, time of first ping: 2017-Jul-28 20:26:08
02:05:44  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T202608.zarr
02:05:49  parsing file Summer2017-D20170728-T205857.raw, time of first ping: 2017-Jul-28 20:58:57
02:06:02  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T205857.zarr
02:06:07  parsing file Summer2017-D20170728-T212916.raw, time of first ping: 2017-Jul-28 21:29:16
02:06:20  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T212916.zarr
02:06:24  parsing file Summer2017-D20170728-T215347.raw, time of first ping: 2017-Jul-28 21:53:47
02:06:33  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T215347.zarr
02:06:37  parsing file Summer2017-D20170728-T221343.raw, time of first ping: 2017-Jul-28 22:13:43
02:06:47  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T221343.zarr
02:06:54  parsing file Summer2017-D20170728-T223346.raw, time of first ping: 2017-Jul-28 22:33:46
02:07:08  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T223346.zarr
02:07:13  parsing file Summer2017-D20170728-T225342.raw, time of first ping: 2017-Jul-28 22:53:42
02:07:21  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T225342.zarr
02:07:25  parsing file Summer2017-D20170728-T231334.raw, time of first ping: 2017-Jul-28 23:13:34
02:07:34  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T231334.zarr
02:07:39  parsing file Summer2017-D20170728-T233248.raw, time of first ping: 2017-Jul-28 23:32:48
02:07:51  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T233248.zarr
02:07:56  parsing file Summer2017-D20170728-T235144.raw, time of first ping: 2017-Jul-28 23:51:44
02:08:17  overwriting exports/hakesurvey_convertednc/Summer2017-D20170728-T235144.zarr
02:08:21  parsing file Summer2017-D20170729-T001030.raw, time of first ping: 2017-Jul-29 00:10:30
02:08:43  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T001030.zarr
02:08:51  parsing file Summer2017-D20170729-T002909.raw, time of first ping: 2017-Jul-29 00:29:09
02:09:02  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T002909.zarr
02:09:07  parsing file Summer2017-D20170729-T004732.raw, time of first ping: 2017-Jul-29 00:47:32
02:09:17  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T004732.zarr
02:09:21  parsing file Summer2017-D20170729-T010556.raw, time of first ping: 2017-Jul-29 01:05:56
02:09:35  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T010556.zarr
02:09:39  parsing file Summer2017-D20170729-T012420.raw, time of first ping: 2017-Jul-29 01:24:20
02:09:48  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T012420.zarr
02:09:52  parsing file Summer2017-D20170729-T014242.raw, time of first ping: 2017-Jul-29 01:42:42
02:10:01  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T014242.zarr
02:10:05  parsing file Summer2017-D20170729-T020106.raw, time of first ping: 2017-Jul-29 02:01:06
02:10:20  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T020106.zarr
02:10:24  parsing file Summer2017-D20170729-T021938.raw, time of first ping: 2017-Jul-29 02:19:38
02:10:41  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T021938.zarr
02:10:46  parsing file Summer2017-D20170729-T023826.raw, time of first ping: 2017-Jul-29 02:38:26
02:10:57  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T023826.zarr
02:11:02  parsing file Summer2017-D20170729-T025714.raw, time of first ping: 2017-Jul-29 02:57:14
02:11:14  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T025714.zarr
02:11:19  parsing file Summer2017-D20170729-T031607.raw, time of first ping: 2017-Jul-29 03:16:07
02:11:31  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T031607.zarr
02:11:36  parsing file Summer2017-D20170729-T033503.raw, time of first ping: 2017-Jul-29 03:35:03
02:11:41  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T033503.zarr
02:11:45  parsing file Summer2017-D20170729-T034649.raw, time of first ping: 2017-Jul-29 03:46:49
02:11:59  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T034649.zarr
02:12:03  parsing file Summer2017-D20170729-T040618.raw, time of first ping: 2017-Jul-29 04:06:18
02:12:15  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T040618.zarr
02:12:20  parsing file Summer2017-D20170729-T042604.raw, time of first ping: 2017-Jul-29 04:26:04
02:12:30  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T042604.zarr
02:12:37  parsing file Summer2017-D20170729-T044550.raw, time of first ping: 2017-Jul-29 04:45:50
02:12:51  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T044550.zarr
02:12:57  parsing file Summer2017-D20170729-T050541.raw, time of first ping: 2017-Jul-29 05:05:41
02:13:07  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T050541.zarr
02:13:12  parsing file Summer2017-D20170729-T052531.raw, time of first ping: 2017-Jul-29 05:25:31
02:13:23  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T052531.zarr
02:13:27  parsing file Summer2017-D20170729-T054522.raw, time of first ping: 2017-Jul-29 05:45:22
02:13:38  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T054522.zarr
02:13:42  parsing file Summer2017-D20170729-T060513.raw, time of first ping: 2017-Jul-29 06:05:13
02:13:55  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T060513.zarr
02:14:00  parsing file Summer2017-D20170729-T062501.raw, time of first ping: 2017-Jul-29 06:25:01
02:14:11  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T062501.zarr
02:14:16  parsing file Summer2017-D20170729-T064450.raw, time of first ping: 2017-Jul-29 06:44:50
02:14:26  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T064450.zarr
02:14:31  parsing file Summer2017-D20170729-T070438.raw, time of first ping: 2017-Jul-29 07:04:38
02:14:41  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T070438.zarr
02:14:46  parsing file Summer2017-D20170729-T072426.raw, time of first ping: 2017-Jul-29 07:24:26
02:14:56  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T072426.zarr
02:15:01  parsing file Summer2017-D20170729-T074413.raw, time of first ping: 2017-Jul-29 07:44:13
02:15:10  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T074413.zarr
02:15:16  parsing file Summer2017-D20170729-T080357.raw, time of first ping: 2017-Jul-29 08:03:57
02:15:26  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T080357.zarr
02:15:31  parsing file Summer2017-D20170729-T082342.raw, time of first ping: 2017-Jul-29 08:23:42
02:15:40  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T082342.zarr
02:15:45  parsing file Summer2017-D20170729-T084328.raw, time of first ping: 2017-Jul-29 08:43:28
02:15:59  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T084328.zarr
02:16:03  parsing file Summer2017-D20170729-T090314.raw, time of first ping: 2017-Jul-29 09:03:14
02:16:15  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T090314.zarr
02:16:20  parsing file Summer2017-D20170729-T092259.raw, time of first ping: 2017-Jul-29 09:22:59
02:16:29  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T092259.zarr
02:16:34  parsing file Summer2017-D20170729-T094245.raw, time of first ping: 2017-Jul-29 09:42:45
02:16:45  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T094245.zarr
02:16:49  parsing file Summer2017-D20170729-T100230.raw, time of first ping: 2017-Jul-29 10:02:30
02:17:00  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T100230.zarr
02:17:05  parsing file Summer2017-D20170729-T102215.raw, time of first ping: 2017-Jul-29 10:22:15
02:17:15  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T102215.zarr
02:17:20  parsing file Summer2017-D20170729-T104201.raw, time of first ping: 2017-Jul-29 10:42:01
02:17:29  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T104201.zarr
02:17:34  parsing file Summer2017-D20170729-T110147.raw, time of first ping: 2017-Jul-29 11:01:47
02:17:45  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T110147.zarr
02:17:49  parsing file Summer2017-D20170729-T112132.raw, time of first ping: 2017-Jul-29 11:21:32
02:17:59  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T112132.zarr
02:18:04  parsing file Summer2017-D20170729-T114118.raw, time of first ping: 2017-Jul-29 11:41:18
02:18:16  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T114118.zarr
02:18:21  parsing file Summer2017-D20170729-T120104.raw, time of first ping: 2017-Jul-29 12:01:04
02:18:31  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T120104.zarr
02:18:34  parsing file Summer2017-D20170729-T122049.raw, time of first ping: 2017-Jul-29 12:20:49
02:18:34  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T122049.zarr
02:18:37  parsing file Summer2017-D20170729-T122320.raw, time of first ping: 2017-Jul-29 12:23:20
02:18:37  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T122320.zarr
02:18:41  parsing file Summer2017-D20170729-T122642.raw, time of first ping: 2017-Jul-29 12:26:42
02:18:50  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T122642.zarr
02:18:55  parsing file Summer2017-D20170729-T124630.raw, time of first ping: 2017-Jul-29 12:46:30
02:19:05  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T124630.zarr
02:19:09  parsing file Summer2017-D20170729-T130615.raw, time of first ping: 2017-Jul-29 13:06:15
02:19:19  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T130615.zarr
02:19:24  parsing file Summer2017-D20170729-T132601.raw, time of first ping: 2017-Jul-29 13:26:01
02:19:34  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T132601.zarr
02:19:41  parsing file Summer2017-D20170729-T134546.raw, time of first ping: 2017-Jul-29 13:45:46
02:19:54  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T134546.zarr
02:19:58  parsing file Summer2017-D20170729-T140536.raw, time of first ping: 2017-Jul-29 14:05:36
02:20:14  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T140536.zarr
02:20:19  parsing file Summer2017-D20170729-T142749.raw, time of first ping: 2017-Jul-29 14:27:49
02:20:34  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T142749.zarr
02:20:39  parsing file Summer2017-D20170729-T144927.raw, time of first ping: 2017-Jul-29 14:49:27
02:20:50  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T144927.zarr
02:20:54  parsing file Summer2017-D20170729-T150925.raw, time of first ping: 2017-Jul-29 15:09:25
02:21:05  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T150925.zarr
02:21:10  parsing file Summer2017-D20170729-T152925.raw, time of first ping: 2017-Jul-29 15:29:25
02:21:20  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T152925.zarr
02:21:24  parsing file Summer2017-D20170729-T154923.raw, time of first ping: 2017-Jul-29 15:49:23
02:21:36  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T154923.zarr
02:21:40  parsing file Summer2017-D20170729-T160924.raw, time of first ping: 2017-Jul-29 16:09:24
02:21:50  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T160924.zarr
02:21:54  parsing file Summer2017-D20170729-T162927.raw, time of first ping: 2017-Jul-29 16:29:27
02:22:03  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T162927.zarr
02:22:08  parsing file Summer2017-D20170729-T165128.raw, time of first ping: 2017-Jul-29 16:51:28
02:22:18  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T165128.zarr
02:22:22  parsing file Summer2017-D20170729-T172207.raw, time of first ping: 2017-Jul-29 17:22:07
02:22:33  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T172207.zarr
02:22:37  parsing file Summer2017-D20170729-T175542.raw, time of first ping: 2017-Jul-29 17:55:42
02:22:48  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T175542.zarr
02:22:53  parsing file Summer2017-D20170729-T182943.raw, time of first ping: 2017-Jul-29 18:29:43
02:23:05  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T182943.zarr
02:23:10  parsing file Summer2017-D20170729-T185953.raw, time of first ping: 2017-Jul-29 18:59:53
02:23:22  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T185953.zarr
02:23:29  parsing file Summer2017-D20170729-T192034.raw, time of first ping: 2017-Jul-29 19:20:34
02:23:41  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T192034.zarr
02:23:46  parsing file Summer2017-D20170729-T194026.raw, time of first ping: 2017-Jul-29 19:40:26
02:23:57  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T194026.zarr
02:24:02  parsing file Summer2017-D20170729-T200020.raw, time of first ping: 2017-Jul-29 20:00:20
02:24:12  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T200020.zarr
02:24:16  parsing file Summer2017-D20170729-T202014.raw, time of first ping: 2017-Jul-29 20:20:14
02:24:26  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T202014.zarr
02:24:30  parsing file Summer2017-D20170729-T204008.raw, time of first ping: 2017-Jul-29 20:40:08
02:24:40  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T204008.zarr
02:24:44  parsing file Summer2017-D20170729-T210002.raw, time of first ping: 2017-Jul-29 21:00:02
02:24:54  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T210002.zarr
02:24:59  parsing file Summer2017-D20170729-T211958.raw, time of first ping: 2017-Jul-29 21:19:58
02:25:14  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T211958.zarr
02:25:18  parsing file Summer2017-D20170729-T213950.raw, time of first ping: 2017-Jul-29 21:39:50
02:25:29  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T213950.zarr
02:25:33  parsing file Summer2017-D20170729-T215945.raw, time of first ping: 2017-Jul-29 21:59:45
02:25:49  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T215945.zarr
02:25:53  parsing file Summer2017-D20170729-T221931.raw, time of first ping: 2017-Jul-29 22:19:31
02:26:03  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T221931.zarr
02:26:07  parsing file Summer2017-D20170729-T223917.raw, time of first ping: 2017-Jul-29 22:39:17
02:26:16  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T223917.zarr
02:26:20  parsing file Summer2017-D20170729-T225900.raw, time of first ping: 2017-Jul-29 22:59:00
02:26:30  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T225900.zarr
02:26:35  parsing file Summer2017-D20170729-T231846.raw, time of first ping: 2017-Jul-29 23:18:46
02:26:49  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T231846.zarr
02:26:56  parsing file Summer2017-D20170729-T233829.raw, time of first ping: 2017-Jul-29 23:38:29
02:27:10  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T233829.zarr
02:27:15  parsing file Summer2017-D20170729-T235813.raw, time of first ping: 2017-Jul-29 23:58:13
02:27:24  overwriting exports/hakesurvey_convertednc/Summer2017-D20170729-T235813.zarr
CPU times: user 14min 57s, sys: 1min 9s, total: 16min 7s
Wall time: 44min 13s

### Test for time reversals

Small time reversals are found in EK60 datasets, including the 2017 Pacific Hake survey, where the `ping_time` (or GPS `location_time`) value may be lower (older) than the preceding `ping_time` by a second. Such discontinuities can interfere with concatenating individual raw files to procuce an aggregated dataset. The capability to identify and address these reversals is in the `echopype.qc` subpackage._____no_output_____

for datapath in converted_dpath.glob('*'):
    ed = ep.open_converted(datapath)
    # Test for a negative ping_time increment in sequential timestamps, in the Beam group
    if ep.qc.exist_reversed_time(ds=ed.beam, time_name="ping_time"):
        print(f"Reversed time in {datapath}")_____no_output_____

There are no time reversals in this two-day dataset, fortunately._____no_output_____### Examine the EchoData object for one of the data files

echopype provides a user-friendly, convenient representation of an `EchoData` object that leverages the user-friendly xarray `Dataset` HTML representation. Since an `EchoData` object is effectively a container for multiple `xarray.Dataset` objects corresponding to netCDF4 groups, the notebook "print out" provides a summary view of all the groups and interactive access to summaries of each group.

Here, `ed` is the last object opened in the time reversal test, in the preceding cell._____no_output_____

ed_____no_output_____

## Extract and process GPS locations from the `Platform` group of converted raw files

Use `xarray.open_mfdataset` to open the `Platform` group from all the converted raw netcdf files as a single concatenated (combined) `xarray` dataset. Then extract GPS `location_time` (time stamp), `latitude` and `longitude` from this group and transform that data into a [GeoPandas](https://geopandas.org/) `GeoDataFrame` containing point-geometry objects that are readily manipulated via geospatial operations. A `GeoDataFrame` adds geospatial capabilities to a Pandas `DataFrame`._____no_output_____Due to the presence of two time coordinates in this group (`ping_time` and `location_time`), care must be taken in defining how the concatenation (combine) operation is to be performed. This is captured in the arguments passed to `open_mfdataset`._____no_output_____

%%time
platform_ds = xr.open_mfdataset(
    str(converted_dpath / '*.zarr'), group='Platform',
    engine='zarr',
    concat_dim=["location_time", "ping_time"],
    data_vars='minimal', coords='minimal'
)CPU times: user 32.7 s, sys: 370 ms, total: 33 s
Wall time: 32.8 s
platform_ds_____no_output_____

We can use `location_time` (or `ping_time`) to examine the exact timestamp interval spanned by the combined dataset._____no_output_____

print(f"{platform_ds.location_time.values.min()}, {platform_ds.location_time.values.max()}")2017-07-28T00:05:36.103000064, 2017-07-30T00:17:59.381000192

To create the GeoPandas GeoDataFrame, first transform the `latitude` and `longitude` arrays to a single Pandas DataFrame that retains the `location_time` coordinate as a common index. This is done by using the DataFrame `to_dataframe` method together with a Pandas `join` operation. Then, a point GeoDataFrame is created from this DataFrame._____no_output_____

gps_df = platform_ds.latitude.to_dataframe().join(platform_ds.longitude.to_dataframe())

gps_df.head(3)_____no_output_____gps_gdf = gpd.GeoDataFrame(
    gps_df,
    geometry=gpd.points_from_xy(gps_df['longitude'], gps_df['latitude']),
    crs="epsg:4326"
)_____no_output_____

A simple, easily generated map plot of the point GeoDataFrame_____no_output_____

gps_gdf.plot(markersize=2)_____no_output_____

## Read MVBS and plot track echograms for time periods corresponding to two ship tracks_____no_output_____### Read MVBS as a concatenated dataset

Use `xarray.open_mfdataset` again to read and concatenate (combine) data files into a single xarray Dataset. This time, we're reading the MVBS netCDF files._____no_output_____

%%time
MVBS_ds = xr.open_mfdataset(
    str(calibrated_dpath / 'MVBS_*.nc'),
    combine='by_coords'
)CPU times: user 1.37 s, sys: 56.4 ms, total: 1.43 s
Wall time: 1.45 s
MVBS_ds_____no_output_____

### Extract MVBS along two N-S tracks selected via geographical bounding boxes_____no_output_____Define rectangular bounding boxes around two tracks oriented North-South, then plot a reference map showing all the GPS points (in red), the two bounding boxes (black), and the OOI mooring location (CE04 Oregon Offshore, yellow star) examined in the accompanying Jupyter notebook._____no_output_____

tracksouth_bbox = gpd.GeoSeries(box(-125.17, 43.65, -125.14, 43.84), crs=gps_gdf.crs)
tracknorth_bbox = gpd.GeoSeries(box(-125.17, 43.98, -125.14, 44.17), crs=gps_gdf.crs)_____no_output_____basemap = cimgt.OSM()

_, ax = plt.subplots(
    figsize=(7, 7), subplot_kw={"projection": basemap.crs}
)
bnd = gps_gdf.geometry.bounds
ax.set_extent([bnd.minx.min() - 1, bnd.maxx.max() + 2,
               bnd.miny.min() - 0.8, bnd.maxy.max() + 2.2])
ax.add_image(basemap, 7)
ax.gridlines(draw_labels=True, xformatter=LONGITUDE_FORMATTER, yformatter=LATITUDE_FORMATTER)

# GPS points
gps_gdf.plot(ax=ax, markersize=0.1, color='red',
             transform=ccrs.PlateCarree())

# Bounding box for selected tracks
tracksouth_bbox.plot(ax=ax, edgecolor="black", linewidth=1.2, facecolor='none',
                     transform=ccrs.PlateCarree())
tracknorth_bbox.plot(ax=ax, edgecolor="black", linewidth=1.2, facecolor='none',
                     transform=ccrs.PlateCarree())

# OOI CE04 Oregon Offshore mooring location
plt.plot(-124.95, 44.37, marker='*', color='yellow', markersize=13,
         transform=ccrs.PlateCarree());_____no_output_____

Clip the GPS locations GeoPandas GeoDataFrame `gps_gdf` generated from the `Platform` group with the two rectangular regions, `tracksouth_bbox` and `tracknorth_bbox`. Extract from those clipped GeoDataFrames (`tracksouth_gps_gdf` and `tracknorth_gps_gdf`) the minimum and maximum `location_time` timestamps for each track and use them to select the corresponding time span in the MVBS data.

Finally, create a MVBS Dataset subset for each track (`tracksouth_MVBS_ds` and `tracknorth_MVBS_ds`) by taking advantage of xarray's "label"-based selection capability via the `sel` method. We select `MVBS_ds` data with `ping_time` values within the timestamp interval "slice" extracted from the geographical track. As we can see above, `ping_time` is a coordinate variable in the `MVBS_ds` Dataset._____no_output_____

tracksouth_gps_gdf = gpd.clip(gps_gdf, tracksouth_bbox)
tracksouth_MVBS_ds = MVBS_ds.sel(
    ping_time=slice(tracksouth_gps_gdf.index.min(), tracksouth_gps_gdf.index.max())
)

tracknorth_gps_gdf = gpd.clip(gps_gdf, tracknorth_bbox)
tracknorth_MVBS_ds = MVBS_ds.sel(
    ping_time=slice(tracknorth_gps_gdf.index.min(), tracknorth_gps_gdf.index.max())
)_____no_output_____

### Plot MVBS echograms for the two N-S tracks, for all 3 frequencies

The final step is to create echogram plots (range vs ping_time) for each echosounder frequency and each of the two selected ship tracks. That's 6 subplots. We first define two functions to simplify the task. `plot_echograms` plots the echograms of the 3 frequencies as a column of subplots, extracting the data for each frequency by using xarray's `isel` index selector method, which uses index counts rather than values. As we can see above, `frequency` is a coordinate of the `Sv` (MVBS) DataArray._____no_output_____

def track_interval_str(trackdt):
    """ Create the timestamp interval title string for a plot column
    """
    track_interval_title_str = (
        f"{trackdt.index.min().strftime('%b-%d %H:%MZ')}"
        f" to {trackdt.index.max().strftime('%b-%d %H:%MZ')}"
    )
    return track_interval_title_str

def plot_echograms(ds, freq_len, column_idx):
    """Plot echograms of the 3 frequencies for xarray dataset ds,
       as a column of subplots"""
    for f in range(freq_len):
        ax = axes[f][column_idx]
        # Select Sv data by frequency using the frequency coordinate index "f",
        # then plot the echogram of the selected data
        ds.Sv.isel(frequency=f).plot(
            ax=ax,
            x='ping_time',
            y='range',
            yincrease=False,
            vmin=-80,
            vmax=-50,
        )
        if f < 2:
            ax.set_xlabel(None)
        ax.set_ylabel('range (meters)');_____no_output_____freq_len = len(MVBS_ds.frequency)

fig, axes = plt.subplots(nrows=freq_len, ncols=2, constrained_layout=True, figsize=(16, 16))

fig.suptitle(
    (f"    South Track, {track_interval_str(tracksouth_gps_gdf)}"
     "                                         "
     f"North Track, {track_interval_str(tracknorth_gps_gdf)}"),
    fontsize=16)

plot_echograms(tracksouth_MVBS_ds, freq_len, column_idx=0) # left column
plot_echograms(tracknorth_MVBS_ds, freq_len, column_idx=1) # right column_____no_output_____

## Package versions_____no_output_____

print(f"echopype: {ep.__version__}, xarray: {xr.__version__}, geopandas: {gpd.__version__}, fsspec: {fsspec.__version__}")echopype: 0.5.4, xarray: 0.19.0, geopandas: 0.10.2, fsspec: 2021.07.0
import datetime
print(f"{datetime.datetime.utcnow()} +00:00")2021-10-26 09:29:09.475883 +00:00


# Notebook from wrathofquan/dats
Path: code_examples/SQL/SQL_Tutorial-0.ipynb

# Intro to SQL - The Hacker Within (2018-02-21)
This tutorial will show you the basics of SQL:
* creating a table
* inserting rows
* querying with conditions
* grouping and ordering
* joining tables with a common field
* aggregating rows using COUNT, MAX, etc.
* alternates to SQL like pandas and Django
* indexing frequently queried columns for performance
* alter existing table to add, drop, or rename a column
* drop a table
* vacuum a database

## SQLite
We are going to use [Python](https://docs.python.org/3/library/sqlite3.html) because it comes with the popular [SQLite](https://sqlite.org/index.html) (_aka_ [`etilqs`](https://www.google.com/search?q=etilqs)) relational database builtin.

## Requirements
The following examples assume you have Python 3 installed on your computer. Any distribution will do. If you have [Anaconda](https://www.anaconda.com/distribution/) or [Miniconda](https://conda.io/miniconda.html), you can create a new conda environment with the requirements for this tutorial from a terminal. For example with miniconda on Mac OS X, you might use the following to activate the root conda environment and create a new one called "py36-thw-sql".

    ~$ source ~/miniconda/bin/activate
    (root) ~$ conda create -n py36-thw-sql python=3.6.3 psycopg2 django pandas

Or on Windows with Anaconda, you might use the following in a CMD terminal.

    ~> %LOCALAPPDATA%\Continuum\anaconda3\Scripts\activate
    (root) ~> conda create -n py36-thw-sql python=3.6.3 psycopg2 django pandas_____no_output_____# Python DB-API 2.0
[PEP 249](https://www.python.org/dev/peps/pep-0249/) specifies a `connection` to a database, and a `cursor` from the connection to execute SQL._____no_output_____

# imports
import io  # we'll need this way later
import os
import sqlite3  # this is the module that binds to SQLite
import numpy as np  # never know when you might need NumPy, oh, right, always!
import pandas as pd  # you'll see why we can use this later

DBFILE = 'sqlite3.db'  # this will be our database
BASEDIR = %pwd  # os.path.abspath(os.path.dirname(__file__))
DBPATH = os.path.join(BASEDIR, DBFILE)

# we may need to delete the existing file first
if os.path.exists(DBPATH):
    os.remove(DBPATH)

# open a connection to the database for this tutorial
conn = sqlite3.connect(DBPATH)

# get a cursor to the database
cur = conn.cursor()_____no_output_____

## Notes
If a file with the same name doesn't already exist, then this creates a new database otherwise it connects to the existing database contained in the file. You can also use ':memory:' to create an "in-memory" database that has no file, but then you can't connect to that from another process.

We'll have to close the connection and cursor later. Next time we could use a [`with` context to automatically close the connection](https://docs.python.org/3.6/library/sqlite3.html#using-the-connection-as-a-context-manager).

    with sqlite3.connect('sqlite3.db') as conn:
        cur = conn.execute('SQL QUERY ...')  # e.g.: 'SELECT * FROM table_name;'
        output = cur.fetchall()  # get the results

Closing the connection automatically closes the cursor. [Other bindings may offer similar context managers.](http://initd.org/psycopg/docs/usage.html#with-statement) to commit and close changes or rollback changes and raise an exception._____no_output_____# Creating tables
A relational database or SQL database is a tabular structure consisting of rows of data with columns of fields. The [data definition language or DDL](https://en.wikipedia.org/wiki/Data_definition_language) used to create the table is the same language used to query it, called [SQL or Structured Query Language](https://en.wikipedia.org/wiki/SQL).

Although [the basic SQL commands](https://sqlite.org/lang.html) are nearly the same for [other relational databases](https://www.postgresql.org/docs/10/static/sql.html), the data types may be different. [SQLite only has 5 datatypes](https://sqlite.org/datatype3.html): `NULL`, `INTEGER`, `REAL`, `TEXT`, `BLOB`. For [boolean, use integer zero for false, and one for true](https://sqlite.org/datatype3.html#boolean_datatype). For [dates and times use text and ISO8601](https://sqlite.org/datatype3.html#date_and_time_datatype), _e.g._: `"2018-02-21T17:05-0800"`. By comparison, [PostgreSQL has too many to list here](https://www.postgresql.org/docs/10/static/datatype.html) including booleans, date, time, arrays, JSON, etc.

## `CREATE`
The [basic SQL command to create a table](https://sqlite.org/lang_createtable.html) is

    CREATE TABLE <table_name> (<field_name> <TYPE> <CONSTRAINTS>, <field_name> <TYPE> <CONSTRAINTS>, <CONSTRAINTS>, ...);

Some languages enforce the semicolon, some don't. The syntax is nearly the same for [other relational databases](https://www.postgresql.org/docs/10/static/ddl-basics.html).

### Constraints, Defaults, and Options
Constraints are optional and set conditions, limitations, or options for columns and the table. The most common constraints are: `PRIMARY KEY`, `UNIQUE`, `NOT NULL`, `DEFAULT`, `FOREIGN KEY`, `REFERENCES`, etc. The syntax is nearly the same for [other relational databases](https://www.postgresql.org/docs/10/static/ddl-constraints.html).

#### `PRIMARY KEY`
The most important of these is `PRIMARY KEY` which is equivalent to `UNIQUE NOT NULL`. A [primary key](https://www.postgresql.org/docs/10/static/ddl-constraints.html#DDL-CONSTRAINTS-PRIMARY-KEYS) is a unique references that identifies each record in the table. Although it is not required, every table should have a primary key. Only one primary key is allowed, and it can be constructed from multiple columns, `PRIMARY KEY (<field_A>, <field_B)`, to create a unique together, non-null identifier. [In SQLite, if missing then a integer primary key named, `rowid`, is created by default](https://sqlite.org/lang_createtable.html#rowid). Also in SQLite, any integer primary key is automatically incremented, so [the _AUTOINCREMENT_ command is usually **not** needed](https://sqlite.org/autoinc.html). [In PostgreSQL the `SERIAL` command is used](https://www.postgresql.org/docs/10/static/datatype-numeric.html#DATATYPE-SERIAL) to create a corresponding [sequence](https://www.postgresql.org/docs/10/static/functions-sequence.html) for the primary key.

## Practice
The other constraints and options are also important, but we'll discover those as we learn. Let's create some simple databases with fictitious data to practice. Imagine you are testing several different materials with different properties $\alpha$ and $\beta$ under different stresses like different temperatures and  light intensity and changing thickness. How would you organize this data? Take a moment to design a schema or structure for your data. The schema consists of the column names and data types and the column and table constraints._____no_output_____

# we can use Python triple quoted strings to span multiple lines, but use single quotes, since SQL only uses double quotes

# first create a materials table
cur.execute('''CREATE TABLE materials (
    material_id TEXT PRIMARY KEY,
    long_name TEXT UNIQUE NOT NULL,
    alpha REAL NOT NULL,
    beta REAL NOT NULL,
    material_type TEXT NOT NULL
)''')
conn.commit()
# if you don't commit the changes, they won't be written to the file, and won't be visible to other connections

# then create an experiments table
cur.execute('''CREATE TABLE experiments (
    experiment_id INTEGER PRIMARY KEY,
    temperature REAL DEFAULT 298.15,
    irradiance REAL DEFAULT 1000.0,
    uv_filter INTEGER DEFAULT 0,
    material_id NOT NULL REFERENCES materials ON UPDATE CASCADE ON DELETE CASCADE,
    thickness REAL DEFAULT 0.005,
    UNIQUE (temperature, irradiance, uv_filter, material_id)
)''')
conn.commit()

# and finally create a trials table
cur.execute('''CREATE TABLE trials (
    trial_id INTEGER PRIMARY KEY,
    experiment_id NOT NULL REFERENCES experiments ON UPDATE CASCADE ON DELETE CASCADE,
    results BLOB NOT NULL,
    duration REAL NOT NULL,
    avg_temperature REAL NOT NULL,
    std_temperature REAL NOT NULL,
    avg_irradiance REAL NOT NULL,
    std_irradiance REAL NOT NULL,
    init_visible_transmittance REAL NOT NULL,
    final_visible_transmittance REAL NOT NULL,
    notes TEXT
)''')
conn.commit()_____no_output_____

## `FOREIGN KEY`
A [foreign key constraint](https://sqlite.org/foreignkeys.html) creates a relationship between two tables. The `FOREIGN KEY` is implied when the `REFERENCES` column constraint is applied. In the experiments table above, the column constraint on `material_id` is the same as adding this table constriant:

    FOREIGN KEY (material_id) REFERENCES materials (material_id)

Specifying the referenced column in the table constraint isn't necessary, and if omitted defaults to the primary key of the referenced table. The syntax is nearly the same for [other relational databases](https://www.postgresql.org/docs/10/static/ddl-constraints.html#DDL-CONSTRAINTS-FK).

You can use the same name for the foreign key and it's related field, but it may make joining tables more difficult because you will need to use the table name to avoid an ambigous column name. _E.G._: you can use `trials.experiment_id` and `experiments.experiment_id` to differentiate between them. You can also use `AS` to create a temporary name like `trials.experiment_id AS experiment`. Or you could just use different names for the foreign key and it's related field like `FOREIGN KEY (material) REFERENCES materials (material_id)` and then there's no ambiguity. Your call.

### `DELETE` and `UPDATE`
What happens if the reference of a foreign key is deleted or updated? That's up to you: in SQLite the default is to do nothing, but typically you want the action to cascade. Add the desired [`ON DELETE` or `ON UPDATE` action](https://sqlite.org/foreignkeys.html#fk_actions) to the constraint.

### Bonus Questions
1. What is the difference between a column constraint and a table constraint?
2. What [other table constraint](https://www.postgresql.org/docs/9.1/static/ddl-constraints.html#AEN2496) is in the experiments table?
3. What other constraints or defaults are applied in the tables?
4. What part of the materials table schema is fragile and can be improved?_____no_output_____## `INSERT`
The [basic SQL command to put data into a table is](https://sqlite.org/lang_insert.html)

    INSERT INTO <table_name> (<field_name>, <field_name>, ...) VALUES (<value>, <value>, ...)

[Other relational databases use the same SQL syntax.](https://www.postgresql.org/docs/current/static/dml-insert.html)

Let's add some pretend data to the database_____no_output_____

# add a EVA as a material
cur.execute('INSERT INTO materials VALUES ("EVA", "ethylene vinyl acetate", 0.123, 4.56, "polymer")')
conn.commit()  # you must commit for it to become permanent
cur.rowcount  # tells you how many rows written, sometimes, it's quirky_____no_output_____

## Placeholders
You can use placeholders to loop over insert statements to add multiple records.

>*WARNING*: Never use string formatters to in lieu of placeholders or you may be subject to [a SQL injection attack](https://xkcd.com/327/).

SQLite uses `?` but [other relational databases may use `%s`](http://initd.org/psycopg/docs/usage.html#passing-parameters-to-sql-queries) or another placeholder.

Also, in `sqlite3` [`executemany`](https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor.executemany) is a convenient shortcut, [but it may not be convenient for all database bindings](http://initd.org/psycopg/docs/extras.html#fast-exec)._____no_output_____

# add some more fake materials
fake_materials = [
    ('PMMC', 'poly methyl methacrylate', 0.789, 10.11, 'polymer'),
    ('KBr', 'potassium bromide', 1.213, 14.15, 'crystal')
]
for mat in fake_materials:
    # must have same number of place holders as values
    cur.execute('INSERT INTO materials VALUES (?, ?, ?, ?, ?)', mat)  # use place holders
conn.commit()  # you can commit all of the changes at the end of the loop

# use the executemany shortcut
fake_materials = [
    ('SiO2', 'silicon dioxide', 1.617, 18.19, 'crystal'),
    ('CaF2', 'calcium flouride', 2.0, 21.22, 'crystal')
]
cur.executemany('INSERT INTO materials VALUES (?, ?, ?, ?, ?)', fake_materials)
conn.commit()
print('rowcount = %d' % cur.rowcount)  # with executemany, cur.rowcount shows total number of rowsrowcount = 2

## `DELETE` and `UPDATE`
Oops I made a mistake. How do I fix it? The opposite of `INSERT` is [`DELETE`](https://sqlite.org/lang_delete.html). But don't throw the baby out with the bathwater, you can also [`UPDATE`] a record. [Other relational databases use the same SQL syntax to manipulate data](https://www.postgresql.org/docs/10/static/dml.html)._____no_output_____

cur.execute('DELETE FROM materials WHERE material_id = "SiO2"')
cur.execute('UPDATE materials SET alpha=1.23E-4, beta=8.910E+11 WHERE material_id = "CaF2"')
conn.commit()_____no_output_____

# Queries
The way you [select](https://sqlite.org/lang_select.html) data is by executing queries. [The language is the same for all relational databases.](https://www.postgresql.org/docs/10/static/queries.html) The star `*` means select all columns, or you can give the columns explicitly._____no_output_____

cur.execute('SELECT * FROM materials')
cur.fetchall()  # fetch all the results of the query_____no_output_____

## Conditions
You can limit a query using [`WHERE`](https://www.postgresql.org/docs/10/static/queries-table-expressions.html#QUERIES-FROM) and [`LIMIT`](https://www.postgresql.org/docs/10/static/queries-limit.html). You can combine `WHERE` with a conditional expression, `IN` to check a set, or `LIKE` to compare with strings. Use `AND` and `OR` to combine conditions.

### Python DB-API Cursor Methods
The Python DB-API cursor can be used as an iterator or you can call it's [fetch methods](https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor.fetchone)._____no_output_____

# limit the query using WHERE and LIMIT
cur.execute('SELECT material_id, long_name FROM materials WHERE alpha < 1 LIMIT 2')
for c in cur: print('{} is {}'.format(*c))  # user the cursor as an iteratorEVA is ethylene vinyl acetate
PMMC is poly methyl methacrylate
materials_list = ("EVA", "PMMC")
cur.execute('SELECT alpha, beta FROM materials WHERE material_id IN (?, ?)', materials_list)
[(mat, cur.fetchone()) for mat in materials_list]  # use the cursor fetchone() method to get next item_____no_output_____

### Aggregates
Your query can aggregate results like `AVG`, `SUM`, `COUNT`, `MAX`, `MIN`, etc._____no_output_____

cur.execute('SELECT COUNT(*) FROM materials')
print(cur.fetchone())(4,)

### `GROUP BY`
You can group queries by a column or a condition such as an expression, `IN`, or `LIKE`, if your selection is an aggregate._____no_output_____

cur.execute('SELECT material_type, COUNT(*), AVG(alpha), MAX(beta) FROM materials GROUP BY material_type')
cur.fetchmany(2)  # use fetchmany() with size parameter, just for fun_____no_output_____

## More Practice
Add a fictitious experiment schedule and doctor up some data!_____no_output_____

# use defaults, let primary key auto-increment, just supply material ID
cur.execute('INSERT INTO experiments (material_id) VALUES ("EVA")')  # use defaults,
conn.commit()_____no_output_____# set up a test matrix for EVA
temp = range(300, 400, 25)
irrad = range(400, 800, 100)
try:
    for T in temp:
        for E in irrad:
            cur.execute('INSERT INTO experiments (temperature, irradiance) VALUES (?, ?)', (T, E))
except sqlite3.IntegrityError as exc:
    print('sqlite3.IntegrityError: %s', exc)

# Oops! We forgot to specify the material, there is not default, and it is constrained as NOT NULL!
conn.rollback()  # undo any changes
try:
    for T in temp:
        for E in irrad:
            cur.execute('INSERT INTO experiments (temperature, irradiance, material_id) VALUES (?, ?, "EVA")', (T, E))
except sqlite3.IntegrityError as exc:
    print(exc)
conn.commit()  # commit! commit! commit!sqlite3.IntegrityError: %s NOT NULL constraint failed: experiments.material_id
# this list is hard to read
list(cur.execute('SELECT * FROM experiments'))_____no_output_____# not only is Pandas much nicer, it also executes queries!
pd.read_sql('SELECT * FROM experiments', conn, index_col='experiment_id')_____no_output_____

## `ORDER BY`
Does what it says; order the query results by a column. Default is ascending, but use `ASC` or `DESC` to change the order._____no_output_____

# Python's SQLite let's you use either '==' or '=', but I think SQL only allows '=', okay?
pd.read_sql('SELECT * FROM experiments WHERE irradiance = 700 ORDER BY temperature', conn, index_col='experiment_id')_____no_output_____# descending order
pd.read_sql('SELECT * FROM experiments WHERE temperature = 375 ORDER BY irradiance DESC', conn, index_col='experiment_id')_____no_output_____

### [Dr. Data](http://www.startrek.com/database_article/data)
![Dr. Data](http://www.startrek.com/uploads/assets/db_articles/26da32597d9bd37fde9da22660aa524f24fd725c.jpg)_____no_output_____

# Dr. Data
start_time, end_time = '2018-02-21T17:00-0800', '2018-02-21T18:30-0800'
timestamps = pd.DatetimeIndex(start=start_time, end=end_time, freq='T')
# use http://poquitopicante.blogspot.com/2016/11/panda-pop.html to help you recall what offset alias to use
size = len(timestamps)
data = {
    'temperature': np.random.randn(size) + 298.15,
    'irradiance': np.random.randn(size) + 1000,
    'visible_transmittance': np.logspace(np.log10(0.9), np.log10(0.8), size) + np.random.randn(size) / 100
}
results = pd.DataFrame(data, index=timestamps)
duration = (results.index[-1] - results.index[0]).value  # [ns]
avg_temperature = results.temperature.mean()  # [K]
std_temperature = results.temperature.std()  # [K]
avg_irradiance = results.irradiance.mean()  # [W/m^2]
std_irradiance = results.irradiance.std()  # [W/m^2]
init_visible_transmittance = results.visible_transmittance[start_time]
final_visible_transmittance = results.visible_transmittance[end_time]
values = (1, results.to_csv(), duration, avg_temperature, std_temperature,
          avg_irradiance, std_irradiance, init_visible_transmittance, final_visible_transmittance,
         'this is doctored data')
cur.execute('''INSERT INTO trials (
    experiment_id, results, duration, avg_temperature, std_temperature,
    avg_irradiance, std_irradiance, init_visible_transmittance,
    final_visible_transmittance, notes
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)''', values)
conn.commit()  # commit! commit! commit!_____no_output_____# check the blob, is it really there?
cur.execute('SELECT results FROM trials WHERE trial_id = 1')
trial1_results = cur.fetchone()
pd.read_csv(io.StringIO(trial1_results[0]), index_col=0)
# yay! it works!_____no_output_____

# `JOIN`
The foreign keys relate tables, but how do we use this relation? By joining the tables._____no_output_____

# add the results for experiment 17: T=375[K], E=700[W/m^2]
experiment_id, temperature, irradiance = list(cur.execute(
    'SELECT experiment_id, temperature, irradiance FROM experiments WHERE (temperature = 375 AND irradiance = 700)'
))[0]
start_time, end_time = '2018-02-28T17:00-0800', '2018-02-28T18:30-0800'
timestamps = pd.DatetimeIndex(start=start_time, end=end_time, freq='T')
# use http://poquitopicante.blogspot.com/2016/11/panda-pop.html to help you recall what offset alias to use
size = len(timestamps)
data = {
    'temperature': np.random.randn(size) + temperature,
    'irradiance': np.random.randn(size) + irradiance,
    'visible_transmittance': np.logspace(np.log10(0.9), np.log10(0.7), size) + np.random.randn(size) / 100
}
results = pd.DataFrame(data, index=timestamps)
duration = (results.index[-1] - results.index[0]).value  # [ns]
avg_temperature = results.temperature.mean()  # [K]
std_temperature = results.temperature.std()  # [K]
avg_irradiance = results.irradiance.mean()  # [W/m^2]
std_irradiance = results.irradiance.std()  # [W/m^2]
init_visible_transmittance = results.visible_transmittance[start_time]
final_visible_transmittance = results.visible_transmittance[end_time]
values = (experiment_id, results.to_csv(), duration, avg_temperature, std_temperature,
          avg_irradiance, std_irradiance, init_visible_transmittance, final_visible_transmittance,
         'this is doctored data')
cur.execute('''INSERT INTO trials (
    experiment_id, results, duration, avg_temperature, std_temperature,
    avg_irradiance, std_irradiance, init_visible_transmittance,
    final_visible_transmittance, notes
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)''', values)
conn.commit()  # commit! commit! commit!_____no_output_____pd.read_sql('''
SELECT
    trial_id, trials.experiment_id AS experiment, init_visible_transmittance, final_visible_transmittance,
    experiments.material_id AS material, temperature, irradiance, alpha, beta, material_type
FROM trials
JOIN experiments ON experiments.experiment_id = experiment
JOIN materials ON materials.material_id = material
''', conn, index_col='trial_id')_____no_output_____cur.close()
conn.close()_____no_output_____

# Epilogue
Unfortunately that's all we have time for, but we covered a lot, even if we didn't get this far. If there's anything I hope you learned from this it's that:

1. You can read the SQLite and PostgreSQL manuals or use [StackOverflow](https://stackoverflow.com/questions/tagged/SQL) to teach yourself SQL. It's not rocket science. I was a mechanical engineer, and I learned it.
2. The basics of making a table, inserting data, and conducting queries.
3. How to interact programmatically with a database.

But there's still so much to learn! Hopefully you will continue on your own and try some of these intereting topics.
1. Use what you've learned on a more advanced database management system like PostgreSQL, MySQL, or MS SQL Server.
2. Use an Object Relational Mapper (ORM) like Django or SQLAlchemy to simplify your workflow, by creating and manipulating data in native Python.
3. Explore a NoSQL database like Cassandra or MongoDB and see if the flexible structure and large scale cluster computing allow you to explore big data with tools like [Spark](http://spark.apache.org/) and [Hadoop](http://hadoop.apache.org/).

Did you learn about everything you expected today? What did wish we had covered? Leave your comments or improve this tutorial by sending a PR to [The Hacker Within, Berkeley](https://github.com/thehackerwithin/berkeley).

Thanks!_____no_output_____