# Notebook from jaedong2019/mec647
Path: mec647_Fracture_4.ipynb

_____no_output_____

%%capture
import sys

try:
    import google.colab  # noqa: F401
except ImportError:
    import ufl  # noqa: F401
    import dolfinx  # noqa: F401
else:
    try:
        import ufl
        import dolfinx
    except ImportError:
        !wget "https://fem-on-colab.github.io/releases/fenicsx-install.sh" -O "/tmp/fenicsx-install.sh" && bash "/tmp/fenicsx-install.sh";
        import ufl  # noqa: F401
        import dolfinx  # noqa: F401

try:
    import pyvista
except ImportError:
    !{sys.executable} -m pip install --upgrade pyvista itkwidgets;
    import pyvista  # noqa: F401
    from pyvista.utilities import xvfb

try:
    import gmsh
except ImportError:
    !{sys.executable} -m pip install gmsh
    import gmsh

!sudo apt install libgl1-mesa-glx xvfb;
!{sys.executable} -m pip install pythreejs;
!{sys.executable} -m pip install ipygany;
!{sys.executable} -m pip install --upgrade PyYAML
# !pip install --ignore-installed PyYAML

_____no_output_____%%capture
!sudo apt install libgl1-mesa-glx xvfb;
!{sys.executable} -m pip install pythreejs;
!{sys.executable} -m pip install ipygany;
!{sys.executable} -m pip install --upgrade PyYAML
# !pip install --ignore-installed PyYAML

_____no_output_____

# Fracture

Let $\Omega \subset (0, L)^D$, with $D=1, 2, 3$, $L$ finite, being the (or one) characteristic length of the specimen.
For any
- displacement field $u\in V_t : H^1(\Omega, R^n) + bcs(t)$ with $n=1, 2$ or $3$, and
- damage field $\alpha \in H^1(\Omega, R)$,

consider the energy $E(u, \alpha)$ defined as
$$
E_\ell(u, \alpha)=\frac{1}{2}\int_\Omega a(\alpha) W(u) dx + \underbrace{\frac{G_c}{c_w} \int \left(\frac{1}{\ell}w(\alpha) + \ell |\nabla \alpha|^2 \right)dx}_{\text{Surface energy}}- \int_\Omega f.u dx$$

In practice, $\ell \ll L$.

Above, $W$ is the elastic energy density, reading (in linearised elasticity as)
$$
W(u) = Ae(u):e(u)
$$
where $A$ is the 4-th order tensor of elasticity, in the isotropic and homogeneous case, it corresponds to a linear combination with two coefficients, say, $A_0$ the stiffness (dimensional), and $\nu$ the Poisson ratio (non-dimensional).

Further, $w(\alpha)$ corresponds to the dissipated energy to damage, homogeneously, the specimen, the gradient term accounts for spatial variations.

**Keypoint:** these two terms are weighted by $\ell$, a parameter that is homogeneous to a length and is understood as a *material* quantity (as opposed to *numerical*).

Define $D(\alpha_0):=\left\{ \alpha \in H^1(\Omega), \alpha \geq \alpha_0 \right\}$, for some $\alpha_0(x)\geq 0$ pointwise.

We solve two types of problems (by increasing difficulty):
- **The static problem**: Given a load (boundary conditions) and an initial state of damage $\alpha_0$, what is the equilibrium displacement and repartition of damage?
In other terms:

$
\text{ min loc} \left\{ E_\ell(u, \alpha):
 u \in V_t, \alpha \in D(\alpha_0) \right\}.
$

- **The evolution problem**: Given a load **history** (boundary conditions as a function of $t$) and an initial state of damage $\alpha_0$, what is the *evolution* of equilibrium displacement and repartition of damage, i.e.
the map $t\mapsto (u_t, \alpha_t)$, such that
  - (Irrevers.) $\alpha_t \nearrow t$,
  - (Stability) $(u_t, \alpha_t) = \operatorname{arg loc min} \left\{ E_\ell (v, \beta), (v, \beta) \in V_t \times D(\alpha_t) \right\}$  -->

_____no_output_____### Parameters

In the energy above:
- Two elasticity parameters, such as
  - $A_0$ the stiffness of the sound material
  - $\nu$ the Poisson ratio
  - equivalently, $\mu, \lambda$ the Lamé parameters
- Two fracture/damage parameters:
  - $\ell$ the internal damage length
  - $G_c$ the material toughness

### Back of the envelope computation.

1. Show that the energy above can be written as a function of only two non-dimensional parameters (ex.: $\nu, \tilde \ell)$, by dimensional analysis._____no_output_____

branch_name = 'main'

!rm -rf mec647
try:
  !git clone -b {branch_name} https://github.com/kumiori/mec647.git
except Exception:
  print('Something went wrong')
  !rm -rf mec647
  !git clone https://github.com/kumiori/mec647.git
Cloning into 'mec647'...
remote: Enumerating objects: 684, done.
remote: Counting objects: 100% (684/684), done.
remote: Compressing objects: 100% (559/559), done.
remote: Total 684 (delta 305), reused 303 (delta 106), pack-reused 0
Receiving objects: 100% (684/684), 12.63 MiB | 17.62 MiB/s, done.
Resolving deltas: 100% (305/305), done.
sys.path.append('mec647/')
_____no_output_____import matplotlib.pyplot as plt
from petsc4py import PETSc
import numpy as np
from mpi4py import MPI

comm = MPI.COMM_WORLD

from dolfinx.fem import (
    Constant,
    Function,
    FunctionSpace,
    assemble_scalar,
    dirichletbc,
    form,
    locate_dofs_geometrical,
    set_bc,
)

# meshes
import meshes
from meshes import primitives_____no_output_____# Parameters

parameters = {
    'loading': {
        'min': 0,
        'max': 1.5,
        'steps': 10
    },
    'geometry': {
        'geom_type': 'bar',
        'Lx': 1.,
        'Ly': 0.1
    },
    'model': {
        'E': 1.,
        'nu': 0.,
        'w1': 1.,
        'ell': 0.05,
        'k_res': 1.e-8
    },
    'solvers': {
          'elasticity': {
            'snes': {
                'snes_type': 'newtontr',
                'snes_stol': 1e-8,
                'snes_atol': 1e-8,
                'snes_rtol': 1e-8,
                'snes_max_it': 100,
                'snes_monitor': "",
                'ksp_type': 'preonly',
                'pc_type': 'lu',
                'pc_factor_mat_solver_type': 'mumps'
            }
        },
          'damage': {
            'snes': {
                'snes_type': 'vinewtonrsls',
                'snes_stol': 1e-5,
                'snes_atol': 1e-5,
                'snes_rtol': 1e-8,
                'snes_max_it': 100,
                'snes_monitor': "",
                'ksp_type': 'preonly',
                'pc_type': 'lu',
                'pc_factor_mat_solver_type': 'mumps'
            },
        },
          'damage_elasticity': {
            "max_it": 100,
            "alpha_rtol": 1.0e-5,
            "criterion": "alpha_H1"
          }
    }
}_____no_output_____# Equivalently
import yaml

with open("mec647/test/parameters.yml") as f:
    parameters = yaml.load(f, Loader=yaml.FullLoader)
_____no_output_____parameters.get("model").get("ell")_____no_output_____# Mesh

Lx = parameters["geometry"]["Lx"]
Ly = parameters["geometry"]["Ly"]
geom_type = parameters["geometry"]["geom_type"]

gmsh_model, tdim = primitives.mesh_bar_gmshapi(geom_type,
                                    Lx,
                                    Ly,
                                    parameters.get("model").get("ell")/3,
                                    tdim=2)

mesh, mts = meshes.gmsh_model_to_mesh(gmsh_model,
                               cell_data=False,
                               facet_data=True,
                               gdim=2)

from utils.viz import plot_mesh

plt.figure()
ax = plot_mesh(mesh)
fig = ax.get_figure()
fig.savefig(f"mesh.png")
_____no_output_____# Functional Setting

element_u = ufl.VectorElement("Lagrange", mesh.ufl_cell(),
                              degree=1, dim=2)

element_alpha = ufl.FiniteElement("Lagrange", mesh.ufl_cell(),
                              degree=1)

V_u = dolfinx.fem.FunctionSpace(mesh, element_u)
V_alpha = dolfinx.fem.FunctionSpace(mesh, element_alpha)

u = dolfinx.fem.Function(V_u, name="Displacement")
u_ = dolfinx.fem.Function(V_u, name="BoundaryDisplacement")

alpha = dolfinx.fem.Function(V_alpha, name="Damage")

# Pack state
state = {"u": u, "alpha": alpha}

# Bounds
alpha_ub = dolfinx.fem.Function(V_alpha, name="UpperBoundDamage")
alpha_lb = dolfinx.fem.Function(V_alpha, name="LowerBoundDamage")

dx = ufl.Measure("dx", domain = mesh)
ds = ufl.Measure("ds", domain = mesh)_____no_output_____# Boundary sets

dofs_alpha_left = locate_dofs_geometrical(V_alpha, lambda x: np.isclose(x[0], 0.))
dofs_alpha_right = locate_dofs_geometrical(V_alpha, lambda x: np.isclose(x[0], Lx))

dofs_u_left = locate_dofs_geometrical(V_u, lambda x: np.isclose(x[0], 0.))
dofs_u_right = locate_dofs_geometrical(V_u, lambda x: np.isclose(x[0], Lx))_____no_output_____# Boundary data

u_.interpolate(lambda x: (np.ones_like(x[0]), np.zeros_like(x[1])))

# Bounds (nontrivial)

alpha_lb.interpolate(lambda x: np.zeros_like(x[0]))
alpha_ub.interpolate(lambda x: np.ones_like(x[0]))_____no_output_____# Boundary conditions

bcs_u = [
         dirichletbc(np.array([0., 0.], dtype=PETSc.ScalarType),
                      dofs_u_left,
                      V_u),
         dirichletbc(u_, dofs_u_right)
         ]

bcs_alpha = [
             dirichletbc(np.array(0., dtype = PETSc.ScalarType),
                         np.concatenate([dofs_alpha_left, dofs_alpha_right]),
                         V_alpha)
]
_____no_output_____import models
from models import DamageElasticityModel as Brittle_____no_output_____bcs = {"bcs_u": bcs_u, "bcs_alpha": bcs_alpha}

model = Brittle(parameters["model"])

total_energy = model.total_energy_density(state) * dx_____no_output_____import algorithms
from algorithms import am_____no_output_____solver = am.AlternateMinimisation(total_energy,
                         state,
                         bcs,
                         parameters.get("solvers"),
                         bounds=(alpha_lb, alpha_ub)
                         )_____no_output_____# Loop for evolution

loads = np.linspace(parameters.get("loading").get("min"),
                    parameters.get("loading").get("max"),
                    parameters.get("loading").get("steps"))

data = {
    'elastic': [],
    'surface': [],
    'total': [],
    'load': []
}
for (i_t, t) in enumerate(loads):
  # update boundary conditions

  u_.interpolate(lambda x: (t * np.ones_like(x[0]), np.zeros_like(x[1])))
  u_.vector.ghostUpdate(addv=PETSc.InsertMode.INSERT,
                        mode=PETSc.ScatterMode.FORWARD)

  # update lower bound for damage
  alpha.vector.copy(alpha_lb.vector)
  alpha.vector.ghostUpdate(addv=PETSc.InsertMode.INSERT,
                        mode=PETSc.ScatterMode.FORWARD)

  # solve for current load step
  print(f"Solving timestep {i_t}, load: {t}")

  solver.solve()

  # postprocessing
  # global

  surface_energy = comm.allreduce(
          assemble_scalar(
            dolfinx.fem.form(model.damage_dissipation_density(state) * dx)),
          op=MPI.SUM
          )

  elastic_energy = comm.allreduce(
      assemble_scalar(
        dolfinx.fem.form(model.elastic_energy_density(state) * dx)),
          op=MPI.SUM
        )

  data.get('elastic').append(elastic_energy)
  data.get('surface').append(surface_energy)
  data.get('total').append(surface_energy+elastic_energy)
  data.get('load').append(t)

  print(f"Solved timestep {i_t}, load: {t}")
  print(f"Elastic Energy {elastic_energy:.3g}, Surface energy: {surface_energy:.3g}")
  print("\n\n")

  # savings?

Solving timestep 0, load: 0.0
  0 SNES Function norm 0.000000000000e+00
  0 SNES Function norm 0.000000000000e+00
Solved timestep 0, load: 0.0
Elastic Energy 0, Surface energy: 0

Solving timestep 1, load: 0.16666666666666666
  0 SNES Function norm 4.550226413810e-01
  1 SNES Function norm 3.796504297722e-01
  2 SNES Function norm 2.289060065546e-01
  3 SNES Function norm 2.919139040773e-16
  0 SNES Function norm 0.000000000000e+00
Solved timestep 1, load: 0.16666666666666666
Elastic Energy 0.00139, Surface energy: 0

Solving timestep 2, load: 0.3333333333333333
  0 SNES Function norm 4.550226413810e-01
  1 SNES Function norm 3.640181131048e-01
  2 SNES Function norm 1.820090565524e-01
  3 SNES Function norm 3.234800395702e-16
  0 SNES Function norm 0.000000000000e+00
Solved timestep 2, load: 0.3333333333333333
Elastic Energy 0.00556, Surface energy: 0

Solving timestep 3, load: 0.5
  0 SNES Function norm 4.550226413810e-01
  1 SNES Function norm 2.730135848286e-01
  2 SNES Function norm 4.007272373815e-16
  0 SNES Function norm 0.000000000000e+00
Solved timestep 3, load: 0.5
Elastic Energy 0.0125, Surface energy: 0

Solving timestep 4, load: 0.6666666666666666
  0 SNES Function norm 4.550226413810e-01
  1 SNES Function norm 1.820090565524e-01
  2 SNES Function norm 5.201875852681e-16
  0 SNES Function norm 0.000000000000e+00
Solved timestep 4, load: 0.6666666666666666
Elastic Energy 0.0222, Surface energy: 0

Solving timestep 5, load: 0.8333333333333333
  0 SNES Function norm 4.550226413810e-01
  1 SNES Function norm 9.100452827621e-02
  2 SNES Function norm 6.367368187746e-16
  0 SNES Function norm 0.000000000000e+00
Solved timestep 5, load: 0.8333333333333333
Elastic Energy 0.0347, Surface energy: 0

Solving timestep 6, load: 1.0
  0 SNES Function norm 4.550226413810e-01
  1 SNES Function norm 2.270564208927e-15
  0 SNES Function norm 1.658739170580e-17
Solved timestep 6, load: 1.0
Elastic Energy 0.05, Surface energy: 0

Solving timestep 7, load: 1.1666666666666665
  0 SNES Function norm 4.550226413811e-01
  1 SNES Function norm 1.124080193832e-15
  0 SNES Function norm 2.982089522046e-03
  1 SNES Function norm 2.513119533480e-17
  0 SNES Function norm 1.177732169653e-02
  1 SNES Function norm 7.147742488360e-16
  0 SNES Function norm 2.380186519919e-03
  1 SNES Function norm 1.562006035395e-17
  0 SNES Function norm 8.517116950112e-03
  1 SNES Function norm 7.930146602390e-16
  0 SNES Function norm 3.187492579934e-03
  1 SNES Function norm 1.431736116473e-03
  2 SNES Function norm 5.703509676775e-04
  3 SNES Function norm 1.609550783899e-17
  0 SNES Function norm 1.119442641424e-02
  1 SNES Function norm 7.276004766315e-16
  0 SNES Function norm 5.495800936288e-03
  1 SNES Function norm 1.882325531008e-03
  2 SNES Function norm 1.642617984877e-03
  3 SNES Function norm 4.753829955734e-04
  4 SNES Function norm 1.485924605976e-17
  0 SNES Function norm 1.297660834592e-02
  1 SNES Function norm 8.742630036749e-16
  0 SNES Function norm 1.009783506233e-02
  1 SNES Function norm 1.872540427177e-03
  2 SNES Function norm 9.475487094566e-04
  3 SNES Function norm 1.730891879735e-17
  0 SNES Function norm 1.246462150887e-02
  1 SNES Function norm 1.132986564202e-15
  0 SNES Function norm 1.638553682161e-02
  1 SNES Function norm 1.246423030310e-03
  2 SNES Function norm 6.970198002799e-06
  0 SNES Function norm 8.599400093241e-03
  1 SNES Function norm 9.697169681881e-16
  0 SNES Function norm 1.372737529078e-02
  1 SNES Function norm 1.301711159150e-03
  2 SNES Function norm 2.242195888336e-17
  0 SNES Function norm 1.913831014092e-03
  1 SNES Function norm 7.897498989577e-16
  0 SNES Function norm 1.477893266842e-03
  1 SNES Function norm 4.230407838293e-17
  0 SNES Function norm 8.899337880250e-05
  1 SNES Function norm 1.071952892969e-15
  0 SNES Function norm 9.007281741696e-05
  1 SNES Function norm 3.722668786904e-17
  0 SNES Function norm 6.428625056420e-06
  1 SNES Function norm 1.090452594983e-15
  0 SNES Function norm 7.877830630469e-06
Solved timestep 7, load: 1.1666666666666665
Elastic Energy 0.00103, Surface energy: 0.0301

Solving timestep 8, load: 1.3333333333333333
  0 SNES Function norm 4.550226413810e-01
  1 SNES Function norm 1.240606413394e-15
  0 SNES Function norm 3.883069611357e-03
  1 SNES Function norm 9.098681864584e-17
  0 SNES Function norm 2.813438553462e-04
  1 SNES Function norm 1.053951816549e-15
  0 SNES Function norm 2.883875617628e-04
  1 SNES Function norm 7.001612764772e-17
  0 SNES Function norm 1.965447386683e-05
  1 SNES Function norm 9.562088135488e-16
  0 SNES Function norm 3.037782980406e-05
  1 SNES Function norm 4.332954671611e-17
  0 SNES Function norm 5.333386361954e-06
  1 SNES Function norm 1.125518973366e-15
  0 SNES Function norm 5.923137072769e-05
  1 SNES Function norm 5.356521104997e-17
  0 SNES Function norm 1.289681464062e-05
  1 SNES Function norm 1.170998331489e-15
  0 SNES Function norm 1.475258044548e-04
  1 SNES Function norm 1.021905122547e-16
  0 SNES Function norm 3.211836664222e-05
  1 SNES Function norm 1.218804277085e-15
  0 SNES Function norm 3.672705547308e-04
  1 SNES Function norm 6.941915036260e-17
  0 SNES Function norm 7.994917495364e-05
  1 SNES Function norm 9.708441862100e-16
  0 SNES Function norm 9.112786844569e-04
  1 SNES Function norm 3.089762196903e-17
  0 SNES Function norm 1.985050678260e-04
  1 SNES Function norm 1.069529462790e-15
  0 SNES Function norm 2.216939250253e-03
  1 SNES Function norm 1.086461731644e-04
  2 SNES Function norm 1.740667932525e-17
  0 SNES Function norm 3.648370807466e-04
  1 SNES Function norm 1.045390095723e-15
  0 SNES Function norm 2.976477889478e-03
  1 SNES Function norm 5.214663058873e-17
  0 SNES Function norm 3.074329705038e-04
  1 SNES Function norm 1.219808793136e-15
  0 SNES Function norm 2.554769246154e-03
  1 SNES Function norm 6.541497848596e-05
  2 SNES Function norm 1.430376630460e-17
  0 SNES Function norm 1.952996992109e-04
  1 SNES Function norm 8.893174304641e-16
  0 SNES Function norm 1.381147891197e-03
  1 SNES Function norm 3.155452101022e-17
  0 SNES Function norm 1.038358408477e-04
  1 SNES Function norm 1.186510639732e-15
  0 SNES Function norm 7.246635200119e-04
  1 SNES Function norm 1.791601033075e-17
  0 SNES Function norm 4.888017377517e-05
  1 SNES Function norm 1.080727486529e-15
  0 SNES Function norm 3.370101121299e-04
  1 SNES Function norm 5.411838968829e-06
  0 SNES Function norm 2.148214761507e-05
  1 SNES Function norm 1.069573461929e-15
  0 SNES Function norm 1.471770140897e-04
  1 SNES Function norm 1.958217775431e-17
  0 SNES Function norm 9.152574421424e-06
  1 SNES Function norm 1.165897304767e-15
  0 SNES Function norm 6.252266191107e-05
  1 SNES Function norm 2.856342918145e-17
  0 SNES Function norm 3.841883647907e-06
  1 SNES Function norm 9.848079225680e-16
  0 SNES Function norm 2.618290586984e-05
  1 SNES Function norm 1.008453735981e-17
  0 SNES Function norm 1.601925177234e-06
  1 SNES Function norm 1.048464048332e-15
  0 SNES Function norm 1.090946097113e-05
  1 SNES Function norm 2.161515494771e-17
  0 SNES Function norm 6.664956806351e-07
  1 SNES Function norm 1.173580445744e-15
  0 SNES Function norm 4.537402517161e-06
Solved timestep 8, load: 1.3333333333333333
Elastic Energy 0.000312, Surface energy: 0.0309

Solving timestep 9, load: 1.5
  0 SNES Function norm 4.550226413810e-01
  1 SNES Function norm 1.149706724543e-15
  0 SNES Function norm 2.277694404744e-03
  1 SNES Function norm 2.110991328805e-17
  0 SNES Function norm 1.131324001603e-04
  1 SNES Function norm 1.146050127705e-15
  0 SNES Function norm 7.670724045086e-04
  1 SNES Function norm 3.695190411407e-17
  0 SNES Function norm 3.266176113117e-05
  1 SNES Function norm 1.157938279737e-15
  0 SNES Function norm 2.143485994612e-04
  1 SNES Function norm 2.646720769499e-17
  0 SNES Function norm 8.707140768176e-06
  1 SNES Function norm 1.019885687725e-15
  0 SNES Function norm 5.664920508884e-05
  1 SNES Function norm 2.998186593622e-17
  0 SNES Function norm 2.275322396500e-06
  1 SNES Function norm 1.090061037657e-15
  0 SNES Function norm 1.477846989223e-05
  1 SNES Function norm 2.811144358882e-17
  0 SNES Function norm 5.927179670118e-07
  1 SNES Function norm 1.206059335871e-15
  0 SNES Function norm 3.850138070015e-06
Solved timestep 9, load: 1.5
Elastic Energy 0.000245, Surface energy: 0.031

plt.plot(data.get('load'), data.get('surface'), label='surface')
plt.plot(data.get('load'), data.get('elastic'), label='elastic')
plt.plot(data.get('load'), [1./2. * t**2*Ly for t in data.get('load')], label='anal elast', ls=':', c='k')

plt.title('Traction bar energetics')
plt.legend()
plt.yticks([0, 1/20], [0, '$1/2.\sigma_c^2/E_0$'])
plt.xticks([0, 1], [0, 1])_____no_output_____# experiments
params0, data0 = parameters, data
_____no_output_____from test_viz import plot_vector, plot_scalar, plot_profile

_ell = parameters.get('model').get('ell')
tol = 1e-3
xs = np.linspace(0 + tol, Lx - tol, 101)
points = np.zeros((3, 101))
points[0] = xs

_plt, data = plot_profile(
    alpha,
    points,
    subplot=(0, 0),
    lineproperties={
        "c": "k",
        "label": f"$alpha$ with $\ell$ = {_ell:.2f}"
    },
)
ax = _plt.gca()
ax.axvline(0.0, c="k")
ax.axvline(2 * _ell, c="k", label='D=$2\ell$')
_plt.legend()
_plt.fill_between(data[0], data[1].reshape(len(data[1])))
_plt.title("Traction bar damage profile")
_____no_output_____


# Notebook from Nicholas-7/cudf
Path: docs/cudf/source/user_guide/10min.ipynb

10 Minutes to cuDF and Dask-cuDF
=======================

Modeled after 10 Minutes to Pandas, this is a short introduction to cuDF and Dask-cuDF, geared mainly for new users.

### What are these Libraries?

[cuDF](https://github.com/rapidsai/cudf) is a Python GPU DataFrame library (built on the Apache Arrow columnar memory format) for loading, joining, aggregating, filtering, and otherwise manipulating tabular data using a DataFrame style API.

[Dask](https://dask.org/) is a flexible library for parallel computing in Python that makes scaling out your workflow smooth and simple. On the CPU, Dask uses Pandas to execute operations in parallel on DataFrame partitions.

[Dask-cuDF](https://github.com/rapidsai/cudf/tree/main/python/dask_cudf) extends Dask where necessary to allow its DataFrame partitions to be processed by cuDF GPU DataFrames as opposed to Pandas DataFrames. For instance, when you call dask_cudf.read_csv(...), your cluster’s GPUs do the work of parsing the CSV file(s) with underlying cudf.read_csv().

### When to use cuDF and Dask-cuDF

If your workflow is fast enough on a single GPU or your data comfortably fits in memory on a single GPU, you would want to use cuDF. If you want to distribute your workflow across multiple GPUs, have more data than you can fit in memory on a single GPU, or want to analyze data spread across many files at once, you would want to use Dask-cuDF._____no_output_____

import os

import cupy as cp
import pandas as pd
import cudf
import dask_cudf

cp.random.seed(12)

#### Portions of this were borrowed and adapted from the
#### cuDF cheatsheet, existing cuDF documentation,
#### and 10 Minutes to Pandas._____no_output_____

Object Creation
---------------_____no_output_____Creating a `cudf.Series` and `dask_cudf.Series`._____no_output_____

s = cudf.Series([1,2,3,None,4])
s_____no_output_____ds = dask_cudf.from_cudf(s, npartitions=2)
ds.compute()_____no_output_____

Creating a `cudf.DataFrame` and a `dask_cudf.DataFrame` by specifying values for each column._____no_output_____

df = cudf.DataFrame({'a': list(range(20)),
                     'b': list(reversed(range(20))),
                     'c': list(range(20))
                    })
df_____no_output_____ddf = dask_cudf.from_cudf(df, npartitions=2)
ddf.compute()_____no_output_____

Creating a `cudf.DataFrame` from a pandas `Dataframe` and a `dask_cudf.Dataframe` from a `cudf.Dataframe`.

*Note that best practice for using Dask-cuDF is to read data directly into a `dask_cudf.DataFrame` with something like `read_csv` (discussed below).*_____no_output_____

pdf = pd.DataFrame({'a': [0, 1, 2, 3],'b': [0.1, 0.2, None, 0.3]})
gdf = cudf.DataFrame.from_pandas(pdf)
gdf_____no_output_____dask_gdf = dask_cudf.from_cudf(gdf, npartitions=2)
dask_gdf.compute()_____no_output_____

Viewing Data
-------------_____no_output_____Viewing the top rows of a GPU dataframe._____no_output_____

df.head(2)_____no_output_____ddf.head(2)_____no_output_____

Sorting by values._____no_output_____

df.sort_values(by='b')_____no_output_____ddf.sort_values(by='b').compute()_____no_output_____

Selection
------------

## Getting_____no_output_____Selecting a single column, which initially yields a `cudf.Series` or `dask_cudf.Series`. Calling `compute` results in a `cudf.Series` (equivalent to `df.a`)._____no_output_____

df['a']_____no_output_____ddf['a'].compute()_____no_output_____

## Selection by Label_____no_output_____Selecting rows from index 2 to index 5 from columns 'a' and 'b'._____no_output_____

df.loc[2:5, ['a', 'b']]_____no_output_____ddf.loc[2:5, ['a', 'b']].compute()_____no_output_____

## Selection by Position_____no_output_____Selecting via integers and integer slices, like numpy/pandas. Note that this functionality is not available for Dask-cuDF DataFrames._____no_output_____

df.iloc[0]_____no_output_____df.iloc[0:3, 0:2]_____no_output_____

You can also select elements of a `DataFrame` or `Series` with direct index access._____no_output_____

df[3:5]_____no_output_____s[3:5]_____no_output_____

## Boolean Indexing_____no_output_____Selecting rows in a `DataFrame` or `Series` by direct Boolean indexing._____no_output_____

df[df.b > 15]_____no_output_____ddf[ddf.b > 15].compute()_____no_output_____

Selecting values from a `DataFrame` where a Boolean condition is met, via the `query` API._____no_output_____

df.query("b == 3")_____no_output_____ddf.query("b == 3").compute()_____no_output_____

You can also pass local variables to Dask-cuDF queries, via the `local_dict` keyword. With standard cuDF, you may either use the `local_dict` keyword or directly pass the variable via the `@` keyword. Supported logical operators include `>`, `<`, `>=`, `<=`, `==`, and `!=`._____no_output_____

cudf_comparator = 3
df.query("b == @cudf_comparator")_____no_output_____dask_cudf_comparator = 3
ddf.query("b == @val", local_dict={'val':dask_cudf_comparator}).compute()_____no_output_____

Using the `isin` method for filtering._____no_output_____

df[df.a.isin([0, 5])]_____no_output_____

## MultiIndex_____no_output_____cuDF supports hierarchical indexing of DataFrames using MultiIndex. Grouping hierarchically (see `Grouping` below) automatically produces a DataFrame with a MultiIndex._____no_output_____

arrays = [['a', 'a', 'b', 'b'], [1, 2, 3, 4]]
tuples = list(zip(*arrays))
idx = cudf.MultiIndex.from_tuples(tuples)
idx_____no_output_____

This index can back either axis of a DataFrame._____no_output_____

gdf1 = cudf.DataFrame({'first': cp.random.rand(4), 'second': cp.random.rand(4)})
gdf1.index = idx
gdf1_____no_output_____gdf2 = cudf.DataFrame({'first': cp.random.rand(4), 'second': cp.random.rand(4)}).T
gdf2.columns = idx
gdf2_____no_output_____

Accessing values of a DataFrame with a MultiIndex. Note that slicing is not yet supported._____no_output_____

gdf1.loc[('b', 3)]_____no_output_____

Missing Data
------------_____no_output_____Missing data can be replaced by using the `fillna` method._____no_output_____

s.fillna(999)_____no_output_____ds.fillna(999).compute()_____no_output_____

Operations
------------_____no_output_____## Stats_____no_output_____Calculating descriptive statistics for a `Series`._____no_output_____

s.mean(), s.var()_____no_output_____ds.mean().compute(), ds.var().compute()_____no_output_____

## Applymap_____no_output_____Applying functions to a `Series`. Note that applying user defined functions directly with Dask-cuDF is not yet implemented. For now, you can use [map_partitions](http://docs.dask.org/en/stable/generated/dask.dataframe.DataFrame.map_partitions.html) to apply a function to each partition of the distributed dataframe._____no_output_____

def add_ten(num):
    return num + 10

df['a'].applymap(add_ten)_____no_output_____ddf['a'].map_partitions(add_ten).compute()_____no_output_____

## Histogramming_____no_output_____Counting the number of occurrences of each unique value of variable._____no_output_____

df.a.value_counts()_____no_output_____ddf.a.value_counts().compute()_____no_output_____

## String Methods_____no_output_____Like pandas, cuDF provides string processing methods in the `str` attribute of `Series`. Full documentation of string methods is a work in progress. Please see the cuDF API documentation for more information._____no_output_____

s = cudf.Series(['A', 'B', 'C', 'Aaba', 'Baca', None, 'CABA', 'dog', 'cat'])
s.str.lower()_____no_output_____ds = dask_cudf.from_cudf(s, npartitions=2)
ds.str.lower().compute()_____no_output_____

## Concat_____no_output_____Concatenating `Series` and `DataFrames` row-wise._____no_output_____

s = cudf.Series([1, 2, 3, None, 5])
cudf.concat([s, s])_____no_output_____ds2 = dask_cudf.from_cudf(s, npartitions=2)
dask_cudf.concat([ds2, ds2]).compute()_____no_output_____

## Join_____no_output_____Performing SQL style merges. Note that the dataframe order is not maintained, but may be restored post-merge by sorting by the index._____no_output_____

df_a = cudf.DataFrame()
df_a['key'] = ['a', 'b', 'c', 'd', 'e']
df_a['vals_a'] = [float(i + 10) for i in range(5)]

df_b = cudf.DataFrame()
df_b['key'] = ['a', 'c', 'e']
df_b['vals_b'] = [float(i+100) for i in range(3)]

merged = df_a.merge(df_b, on=['key'], how='left')
merged_____no_output_____ddf_a = dask_cudf.from_cudf(df_a, npartitions=2)
ddf_b = dask_cudf.from_cudf(df_b, npartitions=2)

merged = ddf_a.merge(ddf_b, on=['key'], how='left').compute()
merged_____no_output_____

## Append_____no_output_____Appending values from another `Series` or array-like object._____no_output_____

s.append(s)_____no_output_____ds2.append(ds2).compute()_____no_output_____

## Grouping_____no_output_____Like pandas, cuDF and Dask-cuDF support the Split-Apply-Combine groupby paradigm._____no_output_____

df['agg_col1'] = [1 if x % 2 == 0 else 0 for x in range(len(df))]
df['agg_col2'] = [1 if x % 3 == 0 else 0 for x in range(len(df))]

ddf = dask_cudf.from_cudf(df, npartitions=2)_____no_output_____

Grouping and then applying the `sum` function to the grouped data._____no_output_____

df.groupby('agg_col1').sum()_____no_output_____ddf.groupby('agg_col1').sum().compute()_____no_output_____

Grouping hierarchically then applying the `sum` function to grouped data._____no_output_____

df.groupby(['agg_col1', 'agg_col2']).sum()_____no_output_____ddf.groupby(['agg_col1', 'agg_col2']).sum().compute()_____no_output_____

Grouping and applying statistical functions to specific columns, using `agg`._____no_output_____

df.groupby('agg_col1').agg({'a':'max', 'b':'mean', 'c':'sum'})_____no_output_____ddf.groupby('agg_col1').agg({'a':'max', 'b':'mean', 'c':'sum'}).compute()_____no_output_____

## Transpose_____no_output_____Transposing a dataframe, using either the `transpose` method or `T` property. Currently, all columns must have the same type. Transposing is not currently implemented in Dask-cuDF._____no_output_____

sample = cudf.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]})
sample_____no_output_____sample.transpose()_____no_output_____

Time Series
------------
_____no_output_____`DataFrames` supports `datetime` typed columns, which allow users to interact with and filter data based on specific timestamps._____no_output_____

import datetime as dt

date_df = cudf.DataFrame()
date_df['date'] = pd.date_range('11/20/2018', periods=72, freq='D')
date_df['value'] = cp.random.sample(len(date_df))

search_date = dt.datetime.strptime('2018-11-23', '%Y-%m-%d')
date_df.query('date <= @search_date')_____no_output_____date_ddf = dask_cudf.from_cudf(date_df, npartitions=2)
date_ddf.query('date <= @search_date', local_dict={'search_date':search_date}).compute()_____no_output_____

Categoricals
------------_____no_output_____`DataFrames` support categorical columns._____no_output_____

gdf = cudf.DataFrame({"id": [1, 2, 3, 4, 5, 6], "grade":['a', 'b', 'b', 'a', 'a', 'e']})
gdf['grade'] = gdf['grade'].astype('category')
gdf_____no_output_____dgdf = dask_cudf.from_cudf(gdf, npartitions=2)
dgdf.compute()_____no_output_____

Accessing the categories of a column. Note that this is currently not supported in Dask-cuDF._____no_output_____

gdf.grade.cat.categories_____no_output_____

Accessing the underlying code values of each categorical observation._____no_output_____

gdf.grade.cat.codes_____no_output_____dgdf.grade.cat.codes.compute()_____no_output_____

Converting Data Representation
--------------------------------_____no_output_____## Pandas_____no_output_____Converting a cuDF and Dask-cuDF `DataFrame` to a pandas `DataFrame`._____no_output_____

df.head().to_pandas()_____no_output_____ddf.compute().head().to_pandas()_____no_output_____

## Numpy_____no_output_____Converting a cuDF or Dask-cuDF `DataFrame` to a numpy `ndarray`._____no_output_____

df.as_matrix()_____no_output_____ddf.compute().as_matrix()_____no_output_____

Converting a cuDF or Dask-cuDF `Series` to a numpy `ndarray`._____no_output_____

df['a'].to_array()_____no_output_____ddf['a'].compute().to_array()_____no_output_____

## Arrow_____no_output_____Converting a cuDF or Dask-cuDF `DataFrame` to a PyArrow `Table`._____no_output_____

df.to_arrow()_____no_output_____ddf.compute().to_arrow()_____no_output_____

Getting Data In/Out
------------------------
_____no_output_____## CSV_____no_output_____Writing to a CSV file._____no_output_____

if not os.path.exists('example_output'):
    os.mkdir('example_output')

df.to_csv('example_output/foo.csv', index=False)_____no_output_____ddf.compute().to_csv('example_output/foo_dask.csv', index=False)_____no_output_____

Reading from a csv file._____no_output_____

df = cudf.read_csv('example_output/foo.csv')
df_____no_output_____ddf = dask_cudf.read_csv('example_output/foo_dask.csv')
ddf.compute()_____no_output_____

Reading all CSV files in a directory into a single `dask_cudf.DataFrame`, using the star wildcard._____no_output_____

ddf = dask_cudf.read_csv('example_output/*.csv')
ddf.compute()_____no_output_____

## Parquet_____no_output_____Writing to parquet files, using the CPU via PyArrow._____no_output_____

df.to_parquet('example_output/temp_parquet')_____no_output_____

Reading parquet files with a GPU-accelerated parquet reader._____no_output_____

df = cudf.read_parquet('example_output/temp_parquet')
df_____no_output_____

Writing to parquet files from a `dask_cudf.DataFrame` using PyArrow under the hood._____no_output_____

ddf.to_parquet('example_files')  _____no_output_____

## ORC_____no_output_____Reading ORC files._____no_output_____

df2 = cudf.read_orc('/rapids/cudf/python/cudf/cudf/tests/data/orc/TestOrcFile.test1.orc')
df2_____no_output_____

Dask Performance Tips
--------------------------------

Like Apache Spark, Dask operations are [lazy](https://en.wikipedia.org/wiki/Lazy_evaluation). Instead of being executed at that moment, most operations are added to a task graph and the actual evaluation is delayed until the result is needed.

Sometimes, though, we want to force the execution of operations. Calling `persist` on a Dask collection fully computes it (or actively computes it in the background), persisting the result into memory. When we're using distributed systems, we may want to wait until `persist` is finished before beginning any downstream operations. We can enforce this contract by using `wait`. Wrapping an operation with `wait` will ensure it doesn't begin executing until all necessary upstream operations have finished.

The snippets below provide basic examples, using `LocalCUDACluster` to create one dask-worker per GPU on the local machine. For more detailed information about `persist` and `wait`, please see the Dask documentation for [persist](https://docs.dask.org/en/latest/api.html#dask.persist) and [wait](https://docs.dask.org/en/latest/futures.html#distributed.wait). Wait relies on the concept of Futures, which is beyond the scope of this tutorial. For more information on Futures, see the Dask [Futures](https://docs.dask.org/en/latest/futures.html) documentation. For more information about multi-GPU clusters, please see the [dask-cuda](https://github.com/rapidsai/dask-cuda) library (documentation is in progress)._____no_output_____First, we set up a GPU cluster. With our `client` set up, Dask-cuDF computation will be distributed across the GPUs in the cluster._____no_output_____

import time

from dask.distributed import Client, wait
from dask_cuda import LocalCUDACluster

cluster = LocalCUDACluster()
client = Client(cluster)
client_____no_output_____

### Persisting Data
Next, we create our Dask-cuDF DataFrame and apply a transformation, storing the result as a new column._____no_output_____

nrows = 10000000

df2 = cudf.DataFrame({'a': cp.arange(nrows), 'b': cp.arange(nrows)})
ddf2 = dask_cudf.from_cudf(df2, npartitions=5)
ddf2['c'] = ddf2['a'] + 5
ddf2_____no_output_____!nvidia-smiThu Jun  4 05:36:08 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00    Driver Version: 440.64.00    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:3B:00.0 Off |                    0 |
| N/A   38C    P0    27W /  70W |    743MiB / 15109MiB |      2%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla T4            On   | 00000000:5E:00.0 Off |                    0 |
| N/A   41C    P0    26W /  70W |    110MiB / 15109MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla T4            On   | 00000000:AF:00.0 Off |                    0 |
| N/A   35C    P0    27W /  70W |    110MiB / 15109MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla T4            On   | 00000000:D8:00.0 Off |                    0 |
| N/A   34C    P0    27W /  70W |    110MiB / 15109MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     57229      C   ...sets/pgali/envs/new_cudf_env/bin/python   633MiB |
|    0     57341      C   ...sets/pgali/envs/new_cudf_env/bin/python    99MiB |
|    1     57337      C   ...sets/pgali/envs/new_cudf_env/bin/python    99MiB |
|    2     57336      C   ...sets/pgali/envs/new_cudf_env/bin/python    99MiB |
|    3     57335      C   ...sets/pgali/envs/new_cudf_env/bin/python    99MiB |
+-----------------------------------------------------------------------------+

Because Dask is lazy, the computation has not yet occurred. We can see that there are twenty tasks in the task graph and we've used about 800 MB of memory. We can force computation by using `persist`. By forcing execution, the result is now explicitly in memory and our task graph only contains one task per partition (the baseline)._____no_output_____

ddf2 = ddf2.persist()
ddf2_____no_output_____!nvidia-smiThu Jun  4 05:36:09 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00    Driver Version: 440.64.00    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:3B:00.0 Off |                    0 |
| N/A   38C    P0    27W /  70W |    743MiB / 15109MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla T4            On   | 00000000:5E:00.0 Off |                    0 |
| N/A   42C    P0    26W /  70W |    110MiB / 15109MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla T4            On   | 00000000:AF:00.0 Off |                    0 |
| N/A   35C    P0    27W /  70W |    110MiB / 15109MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla T4            On   | 00000000:D8:00.0 Off |                    0 |
| N/A   34C    P0    27W /  70W |    110MiB / 15109MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     57229      C   ...sets/pgali/envs/new_cudf_env/bin/python   633MiB |
|    0     57341      C   ...sets/pgali/envs/new_cudf_env/bin/python    99MiB |
|    1     57337      C   ...sets/pgali/envs/new_cudf_env/bin/python    99MiB |
|    2     57336      C   ...sets/pgali/envs/new_cudf_env/bin/python    99MiB |
|    3     57335      C   ...sets/pgali/envs/new_cudf_env/bin/python    99MiB |
+-----------------------------------------------------------------------------+

Because we forced computation, we now have a larger object in distributed GPU memory._____no_output_____### Wait
Depending on our workflow or distributed computing setup, we may want to `wait` until all upstream tasks have finished before proceeding with a specific function. This section shows an example of this behavior, adapted from the Dask documentation.

First, we create a new Dask DataFrame and define a function that we'll map to every partition in the dataframe._____no_output_____

nrows = 10000000

df1 = cudf.DataFrame({'a': cp.arange(nrows), 'b': cp.arange(nrows)})
ddf1 = dask_cudf.from_cudf(df1, npartitions=100)

def func(df):
    time.sleep(cp.random.randint(1, 60))
    return (df + 5) * 3 - 11_____no_output_____

This function will do a basic transformation of every column in the dataframe, but the time spent in the function will vary due to the `time.sleep` statement randomly adding 1-60 seconds of time. We'll run this on every partition of our dataframe using `map_partitions`, which adds the task to our task-graph, and store the result. We can then call `persist` to force execution._____no_output_____

results_ddf = ddf2.map_partitions(func)
results_ddf = results_ddf.persist()_____no_output_____

However, some partitions will be done **much** sooner than others. If we had downstream processes that should wait for all partitions to be completed, we can enforce that behavior using `wait`._____no_output_____

wait(results_ddf)_____no_output_____

With `wait`, we can safely proceed on in our workflow._____no_output_____


# Notebook from djbsmith/dmu_products
Path: dmu4/dmu4_sm_AKARI-NEP/generate_holes.ipynb

# AKARI-NEP Generate hole region files

## I. Produce simple 10 arcsec holes.
First we produce the simplest possible hole regions. For every star brighter than 16 Mag it puts a 10 arcsec circle.

We then go on to produce varying size holes for individual pristine catalogues based on parameters computed by Seb Oliver's IDL code._____no_output_____

from herschelhelp_internal import git_version
print("This notebook was run with herschelhelp_internal version: \n{}".format(git_version()))This notebook was run with herschelhelp_internal version:
0246c5d (Thu Jan 25 17:01:47 2018 +0000) [with local modifications]
from herschelhelp_internal import starmask
from pymoc import MOC

import time
import os_____no_output_____OUT_DIR = os.environ.get('OUT_DIR', "./data")
SUFFIX = os.environ.get('SUFFIX', time.strftime("_%Y%m%d"))
SUFFIX = SUFFIX #+ '_WARNING-MADE-WITH-Lockman-SWIRE-PARAMS'

try:
    os.makedirs(OUT_DIR)
except FileExistsError:
    pass

field = 'AKARI-NEP'_____no_output_____

First we list all the fields including the ra and dec of the first star in the field. We do this because, due to a peculiarity of the pyregion code, we must supply an image header to produce mocs._____no_output_____

#Field names plus ra dec of first circle
fields= {
# Field                RA              DEC
'AKARI-NEP':          [274.654402036,  65.7962520276 ],
'AKARI-SEP':          [72.2316923316, -54.380443672  ],
'Bootes':             [216.431700722,  32.401081899  ],
'CDFS-SWIRE':         [51.0227099923, -29.8185285737 ],
'COSMOS':             [149.295925951,  1.08212668291 ],
'EGS':                [217.276981956,  53.6441519854 ],
'ELAIS-N1':           [247.096600963,  55.1757687739 ],
'ELAIS-N2':           [248.424493154,  39.1274077489 ],
'ELAIS-S1':           [7.10625839472, -43.8632559768 ],
'GAMA-09':            [129.076050945, -2.23171513025 ],
'GAMA-12':            [172.84437099,  -0.482115877707],
'GAMA-15':            [211.756497623, -2.28573712848 ],
'HDF-N':              [190.259734752,  62.205265532  ],
'Herschel-Stripe-82': [353.751913281, -7.10891111165 ],
'Lockman-SWIRE':      [161.942787703,  59.0563805825 ],
'NGP':                [192.899559129,  22.0990890388 ],
'SA13':               [197.895801254,  42.4400105492 ],
'SGP':                [334.297748942, -34.5037863499 ],
'SPIRE-NEP':          [266.334305546,  68.7904496043 ],
'SSDF':               [341.577544902, -59.1868365369 ],
'xFLS':               [261.387059958,  58.0184602211 ],
'XMM-13hr':           [203.318355937,  37.4745777866 ],
'XMM-LSS':            [32.9413834032, -6.02293494708 ]}

_____no_output_____

Then we generate a region file to define all the holes. At present this is very crude and simply puts a 10 arcsec hole over every star brighter than 16 Mag._____no_output_____

starmask.create_holes('../../dmu0/dmu0_GAIA/data/GAIA_{}.fits'.format(field),
                      'data/10_arcsec_holes_{}.reg'.format(field),
                      '../../dmu2/dmu2_field_coverages/{}_MOC.fits'.format(field))

There are 62607 GAIA stars in AKARI-NEP
Starmask written to data/10_arcsec_holes_AKARI-NEP.reg

Then we convert the region files to MOC format. We recomend against using the MOC since, even at this high order, it doesn't capture shape that well. This will become increasingly significant if we have smaller or more detailed shapes in the future._____no_output_____

ORDER=16
starmask.reg2moc('data/10_arcsec_holes_{}.reg'.format(field),
            '../../dmu2/dmu2_field_coverages/{}_MOC.fits'.format(field),
            'data/10_arcsec_holes_{}_O'.format(field) + str(ORDER) + '_MOC.fits',
            ra_typ=fields[field][0],
            dec_typ=fields[field][1],
            order=ORDER)WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:

WCSAXES =                    2 / Number of coordinate axes
CRPIX1  =            [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
     0.5 / Pixel coordinate of reference point
CRPIX2  =                0.5 / Pi [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
xel coordinate of reference point
CDELT1  =                0.675 / [deg] Coordin [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
ate increment at reference point
CDELT2  =                0.675 / [deg] Coordina [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
te increment at reference point
RADECSYSa= 'ICRS    '           / International  [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
Celestial Ref. System
CUNIT1  = 'deg     '                / Units of coordinate  [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
increment and value
CUNIT2  = 'deg     '                / Units of coordinate in [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
crement and value
CTYPE1  = 'RA---TAN'           /
CTYPE2  = 'DEC--TAN'         [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
   /
CRVAL1  =                  274.654402036 / [deg] Coordinate value at refer [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
ence point
CRVAL2  =                  65.7962520276 / [deg] Coordinate value at  [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
reference point
LONPOLE =                  0.0 / [deg] Native longitude of celes [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
tial pole
LATPOLE = 90.0 / [deg] Native latitude of celestial pole
              [astropy.io.fits.card]
WARNING: FITSFixedWarning:  0.5 / Pixel coordinate of reference point
a floating-point value was expected. [astropy.wcs.wcs]
WARNING: FITSFixedWarning:  /
the RADECSYS keyword is deprecated, use RADESYSa. [astropy.wcs.wcs]

## II Produce magnitude varying holes
Then we produce a varying hole parameterised by Seb's code. We define an annulus with a 1 arcsec circle at the centre and an outer radius r_50 (the radius at which the artefact density goes over 0.5 x background density) = 10^(A + B x mag)._____no_output_____

# Run with parameters from IDL code. The IDL output files are in ./data/

per_catalogue_params = [
#     Field       Band        A          B             magnitude limit

    [field, 'irac_i1',          2.33151,   -0.0905957,   16], # Using 18 as faint limit.

]

#We previously took a moc of the survey to only make holes on that area, which
#was a mistake as it led to missing holes for the XID+ run
_____no_output_____for pristine_cat in per_catalogue_params:
    starmask.create_holes('../../dmu0/dmu0_GAIA/data/GAIA_' + pristine_cat[0] + '.fits',    #GAIA catalogue
                          'data/holes_' + pristine_cat[0] + '_' + pristine_cat[1] + SUFFIX + '.reg', #output file
                          '../../dmu2/dmu2_field_coverages/' + field + '_MOC.fits',         #moc of region in which to produce holes
                          AB = [pristine_cat[2],pristine_cat[3]],                           #radius as f of mag params
                          mag_lim = pristine_cat[4])                                        #upper magnitude limitThere are 62607 GAIA stars in AKARI-NEP
Starmask written to data/holes_AKARI-NEP_irac_i1_20180423.reg

Again we convert the region files to MOC format._____no_output_____

ORDER=16
for pristine_cat in per_catalogue_params:
    starmask.reg2moc('data/holes_' + field + '_' + pristine_cat[1] + SUFFIX + '.reg',
            '../../dmu2/dmu2_field_coverages/' + field + '_MOC.fits',
            'data/holes_' + field + '_' + pristine_cat[1] + '_O' + str(ORDER) + SUFFIX + '.fits',
            ra_typ=fields[field][0],
            dec_typ=fields[field][1],
            order=ORDER)WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:

WCSAXES =                    2 / Number of coordinate axes
CRPIX1  =            [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
     0.5 / Pixel coordinate of reference point
CRPIX2  =                0.5 / Pi [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
xel coordinate of reference point
CDELT1  =                0.675 / [deg] Coordin [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
ate increment at reference point
CDELT2  =                0.675 / [deg] Coordina [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
te increment at reference point
RADECSYSa= 'ICRS    '           / International  [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
Celestial Ref. System
CUNIT1  = 'deg     '                / Units of coordinate  [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
increment and value
CUNIT2  = 'deg     '                / Units of coordinate in [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
crement and value
CTYPE1  = 'RA---TAN'           /
CTYPE2  = 'DEC--TAN'         [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
   /
CRVAL1  =                  274.654402036 / [deg] Coordinate value at refer [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
ence point
CRVAL2  =                  65.7962520276 / [deg] Coordinate value at  [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
reference point
LONPOLE =                  0.0 / [deg] Native longitude of celes [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
tial pole
LATPOLE = 90.0 / [deg] Native latitude of celestial pole
              [astropy.io.fits.card]
WARNING: FITSFixedWarning:  0.5 / Pixel coordinate of reference point
a floating-point value was expected. [astropy.wcs.wcs]
WARNING: FITSFixedWarning:  /
the RADECSYS keyword is deprecated, use RADESYSa. [astropy.wcs.wcs]


# Notebook from UWashington-Astro300/Astro300-A16
Path: FirstLast_HW3.ipynb

# First Last - Homework 3_____no_output_____* Use the `Astropy` units and constants packages to solve the following problems.
* Do not hardcode any constants!
* Unless asked, your units should be in the simplest SI units possible_____no_output_____

<code>
import numpy as np

from astropy import units as u
from astropy import constants as const
from astropy.units import imperial
imperial.enable()_____no_output_____
</code>

### Impulse is a change in momentum

$$ I = \Delta\ p\ =\ m\Delta v $$_____no_output_____**Problem 1** - Calculate the $\Delta$v that would be the result of an impuse of 700 (N * s) for M = 338 kg._____no_output_____**Problem 2** - Calculate the $\Delta$v that would be the result of an impuse of 700 (lbf * s) for M = 338 kg._____no_output_____This is the unit conversion error that doomed the [Mars Climate Orbiter](https://en.wikipedia.org/wiki/Mars_Climate_Orbiter)_____no_output_____### The range of a projectile launched with a velocity (v) at and angle ($\theta$) is

$$R\ =\  {v^2 \over g}\ sin(2\theta)$$_____no_output_____**Problem 3** - Find R for v = 123 mph and $\theta$ = 1000 arc minutes_____no_output_____**Problem 4** - How fast to you have to throw a football at 33.3 degrees so that is goes exactly 100 yards? Express your answer in mph_____no_output_____### Kepler's third law can be expressed as:

$$ T^2 = \left( {{4\pi^2} \over {GM}} \right)\ r^3 $$

Where **T** is the orbial period of an object at distance (**r**) around a central object of mass (**M**).

It assumes the mass of the orbiting object is small compared to the mass of the central object._____no_output_____**Problem 5** - Calculate the orbital period of HST. HST orbits 353 miles above the surface of the Earth. Expess your answer in minutes._____no_output_____** Problem 6 ** - An exoplanet orbits the star Epsilon Tauri in 595 days at a distance of 1.93 AU. Calculate the mass of Epsilon Tauri in terms of solar masses._____no_output_____### The velocity of an object in orbit is

$$ v=\sqrt{GM\over r} $$

Where the object is at a distance (**r**) around a central object of mass (**M**)._____no_output_____**Problem 7** - Calculate the velocity of HST. Expess your answer in km/s and mph._____no_output_____**Problem 8** - The Procliamer's song [500 miles](https://youtu.be/MJuyn0WAYNI?t=27s) has a duration of 3 minutes and 33 seconds. Calculate at what altitude, above the Earth's surface, you would have to orbit to go 1000 miles in this time. Express your answer in km and miles._____no_output_____### The Power being received by a solar panel in space can be expressed as:

$$ I\  =\ {{L_{\odot}} \over {4 \pi d^2}}\ \varepsilon$$

Where **I** is the power **per unit area** at a distance (**d**) from the Sun, and $\varepsilon$ is the efficiency of the solar panel.

The solar panels that power spacecraft have an efficiency of about 40%._____no_output_____** Problem 9 ** -  The New Horizons spacecraft requires 220 Watts of power.

Calculate the area of a solar panel that would be needed to power New Horizons at a distance of 1 AU from the Sun._____no_output_____** Problem 10 ** - Same question as above but now a d = 30 AU._____no_output_____** Problem 11 ** - The main part of the Oort cloud is thought to be at a distance of about 10,000 AU.

Calculate the size of the solar panel New Horizons would need to operate in the Oort cloud.

Express your answer in units of the area of an American football field (120 yd x 53.3 yd)._____no_output_____** Problem 12 ** - Calculate the maximum distance from the Sun can you be where a solar panel of 1 football field can power the New Horizons spacecraft. Express your answer in AU._____no_output_____### Due Tue Oct 18 - 5pm
- `Make sure to change the filename to your name!`
- `Make sure to change the Title to your name!`
- `File -> Download as -> HTML (.html)`
- `upload your .html and .ipynb file to the class Canvas page`  _____no_output_____


# Notebook from nhutnamhcmus/datacamp-playground
Path: introduction-to-data-visualization-in-python/1. Customizing plots/notebook_section_1.ipynb

import pandas as pd_____no_output_____percent_bachelors_degrees_women_usa = pd.read_csv('../datasets/percent-bachelors-degrees-women-usa.csv')
print(percent_bachelors_degrees_women_usa.head())   Year  Agriculture  Architecture  Art and Performance    Biology   Business  \
0  1970     4.229798     11.921005                 59.7  29.088363   9.064439
1  1971     5.452797     12.003106                 59.9  29.394403   9.503187
2  1972     7.420710     13.214594                 60.4  29.810221  10.558962
3  1973     9.653602     14.791613                 60.2  31.147915  12.804602
4  1974    14.074623     17.444688                 61.9  32.996183  16.204850

   Communications and Journalism  Computer Science  Education  Engineering  \
0                           35.3              13.6  74.535328          0.8
1                           35.5              13.6  74.149204          1.0
2                           36.6              14.9  73.554520          1.2
3                           38.4              16.4  73.501814          1.6
4                           40.5              18.9  73.336811          2.2

     English  Foreign Languages  Health Professions  Math and Statistics  \
0  65.570923               73.8                77.1                 38.0
1  64.556485               73.9                75.5                 39.0
2  63.664263               74.6                76.9                 40.2
3  62.941502               74.9                77.4                 40.9
4  62.413412               75.3                77.9                 41.8

   Physical Sciences  Psychology  Public Administration  \
0               13.8        44.4                   68.4
1               14.9        46.2                   65.5
2               14.8        47.6                   62.6
3               16.5        50.4                   64.3
4               18.2        52.6                   66.1

   Social Sciences and History
0                         36.8
1                         36.2
2                         36.1
3                         36.4
4                         37.3
year = percent_bachelors_degrees_women_usa['Year']
print(year.head())0    1970
1    1971
2    1972
3    1973
4    1974
Name: Year, dtype: int64
physical_sciences = percent_bachelors_degrees_women_usa['Physical Sciences']
print(physical_sciences.head())0    13.8
1    14.9
2    14.8
3    16.5
4    18.2
Name: Physical Sciences, dtype: float64
computer_science = percent_bachelors_degrees_women_usa['Computer Science']
print(computer_science.head())0    13.6
1    13.6
2    14.9
3    16.4
4    18.9
Name: Computer Science, dtype: float64

# Multiple plots on single axis
It is time now to put together some of what you have learned and combine line plots on a common set of axes. The data set here comes from records of undergraduate degrees awarded to women in a variety of fields from 1970 to 2011. You can compare trends in degrees most easily by viewing two curves on the same set of axes.

Here, three NumPy arrays have been pre-loaded for you: year (enumerating years from 1970 to 2011 inclusive), physical_sciences (representing the percentage of Physical Sciences degrees awarded to women each in corresponding year), and computer_science (representing the percentage of Computer Science degrees awarded to women in each corresponding year).

You will issue two plt.plot() commands to draw line plots of different colors on the same set of axes. Here, year represents the x-axis, while physical_sciences and computer_science are the y-axes._____no_output_____

# Import matplotlib.pyplot
import matplotlib.pyplot as plt

# Plot in blue the % of degrees awarded to women in the Physical Sciences
plt.plot(year, physical_sciences, color='blue')

# Plot in red the % of degrees awarded to women in Computer Science
plt.plot(year, computer_science, color='red')

# Display the plot
plt.show()
_____no_output_____

# Using axes()
Rather than overlaying line plots on common axes, you may prefer to plot different line plots on distinct axes. The command plt.axes() is one way to do this (but it requires specifying coordinates relative to the size of the figure).

Here, you have the same three arrays year, physical_sciences, and computer_science representing percentages of degrees awarded to women over a range of years. You will use plt.axes() to create separate sets of axes in which you will draw each line plot.

In calling plt.axes([xlo, ylo, width, height]), a set of axes is created and made active with lower corner at coordinates (xlo, ylo) of the specified width and height. Note that these coordinates can be passed to plt.axes() in the form of a list or a tuple.

The coordinates and lengths are values between 0 and 1 representing lengths relative to the dimensions of the figure. After issuing a plt.axes() command, plots generated are put in that set of axes._____no_output_____

# Create plot axes for the first line plot
plt.axes([0.05, 0.05, 0.425, 0.9])

# Plot in blue the % of degrees awarded to women in the Physical Sciences
plt.plot(year, physical_sciences, color='blue')

# Create plot axes for the second line plot
plt.axes([0.525, 0.05, 0.425, 0.9])

# Plot in red the % of degrees awarded to women in Computer Science
plt.plot(year, computer_science, color='red')

# Display the plot
plt.show()_____no_output_____

# Using subplot() (1)
The command plt.axes() requires a lot of effort to use well because the coordinates of the axes need to be set manually. A better alternative is to use plt.subplot() to determine the layout automatically.

In this exercise, you will continue working with the same arrays from the previous exercises: year, physical_sciences, and computer_science. Rather than using plt.axes() to explicitly lay out the axes, you will use plt.subplot(m, n, k) to make the subplot grid of dimensions m by n and to make the kth subplot active (subplots are numbered starting from 1 row-wise from the top left corner of the subplot grid)._____no_output_____

# Create a figure with 1x2 subplot and make the left subplot active
plt.subplot(1,2,1)

# Plot in blue the % of degrees awarded to women in the Physical Sciences
plt.plot(year, physical_sciences, color='blue')
plt.title('Physical Sciences')

# Make the right subplot active in the current 1x2 subplot grid
plt.subplot(1,2,2)

# Plot in red the % of degrees awarded to women in Computer Science
plt.plot(year, computer_science, color='red')
plt.title('Computer Science')

# Use plt.tight_layout() to improve the spacing between subplots
plt.tight_layout()
plt.show()
_____no_output_____

# Using subplot() (2)
Now you have some familiarity with plt.subplot(), you can use it to plot more plots in larger grids of subplots of the same figure.

Here, you will make a $2 \times 2$ grid of subplots and plot the percentage of degrees awarded to women in Physical Sciences (using physical_sciences), in Computer Science (using computer_science), in Health Professions (using health), and in Education (using education)._____no_output_____

health = percent_bachelors_degrees_women_usa['Health Professions']
print(health.head())0    77.1
1    75.5
2    76.9
3    77.4
4    77.9
Name: Health Professions, dtype: float64
education = percent_bachelors_degrees_women_usa['Education']
print(education.head())0    74.535328
1    74.149204
2    73.554520
3    73.501814
4    73.336811
Name: Education, dtype: float64
# Create a figure with 2x2 subplot layout and make the top left subplot active
plt.subplot(2,2,1)

# Plot in blue the % of degrees awarded to women in the Physical Sciences
plt.plot(year, physical_sciences, color='blue')
plt.title('Physical Sciences')

# Make the top right subplot active in the current 2x2 subplot grid
plt.subplot(2,2,2)

# Plot in red the % of degrees awarded to women in Computer Science
plt.plot(year, computer_science, color='red')
plt.title('Computer Science')

# Make the bottom left subplot active in the current 2x2 subplot grid
plt.subplot(2,2,3)

# Plot in green the % of degrees awarded to women in Health Professions
plt.plot(year, health, color='green')
plt.title('Health Professions')

# Make the bottom right subplot active in the current 2x2 subplot grid
plt.subplot(2,2,4)

# Plot in yellow the % of degrees awarded to women in Education
plt.plot(year, education, color='yellow')
plt.title('Education')

# Improve the spacing between subplots and display them
plt.tight_layout()
plt.show()
_____no_output_____

# Using xlim(), ylim()
In this exercise, you will work with the matplotlib.pyplot interface to quickly set the x- and y-limits of your plots.

You will now create the same figure as in the previous exercise using plt.plot(), this time setting the axis extents using plt.xlim() and plt.ylim(). These commands allow you to either zoom or expand the plot or to set the axis ranges to include important values (such as the origin).

In this exercise, as before, the percentage of women graduates in Computer Science and in the Physical Sciences are held in the variables computer_science and physical_sciences respectively over year.

After creating the plot, you will use plt.savefig() to export the image produced to a file._____no_output_____

# Plot the % of degrees awarded to women in Computer Science and the Physical Sciences
plt.plot(year,computer_science, color='red')
plt.plot(year, physical_sciences, color='blue')

# Add the axis labels
plt.xlabel('Year')
plt.ylabel('Degrees awarded to women (%)')

# Set the x-axis range
plt.xlim(1990, 2010)

# Set the y-axis range
plt.ylim(0, 50)

# Add a title and display the plot
plt.title('Degrees awarded to women (1990-2010)\nComputer Science (red)\nPhysical Sciences (blue)')
plt.show()

# Save the image as 'xlim_and_ylim.png'
plt.savefig('xlim_and_ylim.png')
_____no_output_____

# Using axis()
Using plt.xlim() and plt.ylim() are useful for setting the axis limits individually. In this exercise, you will see how you can pass a 4-tuple to plt.axis() to set limits for both axes at once. For example, plt.axis((1980,1990,0,75)) would set the extent of the x-axis to the period between 1980 and 1990, and would set the y-axis extent from 0 to 75% degrees award.

Once again, the percentage of women graduates in Computer Science and in the Physical Sciences are held in the variables computer_science and physical_sciences where each value was measured at the corresponding year held in the year variable._____no_output_____

# Plot in blue the % of degrees awarded to women in Computer Science
plt.plot(year,computer_science, color='blue')

# Plot in red the % of degrees awarded to women in the Physical Sciences
plt.plot(year, physical_sciences,color='red')

# Set the x-axis and y-axis limits
plt.axis((1990, 2010, 0, 50))

# Show the figure
plt.show()

# Save the figure as 'axis_limits.png'
plt.savefig('axis_limits.png')
_____no_output_____

# Using legend()
Legends are useful for distinguishing between multiple datasets displayed on common axes. The relevant data are created using specific line colors or markers in various plot commands. Using the keyword argument label in the plotting function associates a string to use in a legend.

For example, here, you will plot enrollment of women in the Physical Sciences and in Computer Science over time. You can label each curve by passing a label argument to the plotting call, and request a legend using plt.legend(). Specifying the keyword argument loc determines where the legend will be placed._____no_output_____

# Specify the label 'Computer Science'
plt.plot(year, computer_science, color='red', label='Computer Science')

# Specify the label 'Physical Sciences'
plt.plot(year, physical_sciences, color='blue', label='Physical Sciences')

# Add a legend at the lower center
plt.legend(loc='lower center')

# Add axis labels and title
plt.xlabel('Year')
plt.ylabel('Enrollment (%)')
plt.title('Undergraduate enrollment of women')
plt.show()
_____no_output_____

# Using annotate()
It is often useful to annotate a simple plot to provide context. This makes the plot more readable and can highlight specific aspects of the data. Annotations like text and arrows can be used to emphasize specific observations.

Here, you will once again plot enrollment of women in the Physical Sciences and Computer Science over time. The legend is set up as before. Additionally, you will mark the inflection point when enrollment of women in Computer Science reached a peak and started declining using plt.annotate().

To enable an arrow, set arrowprops=dict(facecolor='black'). The arrow will point to the location given by xy and the text will appear at the location given by xytext.

Computer Science enrollment and the years of enrollment have been preloaded for you as the arrays computer_science and year, respectively._____no_output_____

# Compute the maximum enrollment of women in Computer Science: cs_max
cs_max = computer_science.max()

# Calculate the year in which there was maximum enrollment of women in Computer Science: yr_max
yr_max = year[computer_science.argmax()]

# Plot with legend as before
plt.plot(year, computer_science, color='red', label='Computer Science')
plt.plot(year, physical_sciences, color='blue', label='Physical Sciences')
plt.legend(loc='lower right')

# Add a black arrow annotation
plt.annotate('Maximum', xy=(yr_max, cs_max), xytext=(yr_max+5, cs_max+5), arrowprops=dict(facecolor='black'))

# Add axis labels and title
plt.xlabel('Year')
plt.ylabel('Enrollment (%)')
plt.title('Undergraduate enrollment of women')
plt.show()_____no_output_____

# Modifying styles
Matplotlib comes with a number of different stylesheets to customize the overall look of different plots. To activate a particular stylesheet you can simply call plt.style.use() with the name of the style sheet you want. To list all the available style sheets you can execute: print(plt.style.available)._____no_output_____

# Import matplotlib.pyplot
import matplotlib.pyplot as plt

# Set the style to 'ggplot'
plt.style.use('ggplot')

# Create a figure with 2x2 subplot layout
plt.subplot(2, 2, 1)

# Plot the enrollment % of women in the Physical Sciences
plt.plot(year, physical_sciences, color='blue')
plt.title('Physical Sciences')

# Plot the enrollment % of women in Computer Science
plt.subplot(2, 2, 2)
plt.plot(year, computer_science, color='red')
plt.title('Computer Science')

# Add annotation
cs_max = computer_science.max()
yr_max = year[computer_science.argmax()]
plt.annotate('Maximum', xy=(yr_max, cs_max), xytext=(yr_max-1, cs_max-10), arrowprops=dict(facecolor='black'))

# Plot the enrollmment % of women in Health professions
plt.subplot(2, 2, 3)
plt.plot(year, health, color='green')
plt.title('Health Professions')

# Plot the enrollment % of women in Education
plt.subplot(2, 2, 4)
plt.plot(year, education, color='yellow')
plt.title('Education')

# Improve spacing between subplots and display them
plt.tight_layout()
plt.show()_____no_output_____


# Notebook from LuGeorgiev/Python-SoftUni
Path: DataScience/08.ExamPreparation/Exploring Music Genres.ipynb

%matplotlib inline_____no_output_____import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from scipy.stats import ttest_ind
from scipy.sparse import csr_matrix_____no_output_____

# Music Genres
## How does music of different genres differ? Which factors contribute to determining the genre of a song?_____no_output_____### Abstract
TODO: Describe the problem, procedures, and outcomes in short. Do this section last._____no_output_____### Introduction
TODO: Describe the task at hand, datasets, info, etc._____no_output_____### Track information: reading and exploring
#### FMA dataset_____no_output_____

tracks_data = pd.read_csv("data/fma/tracks.csv", header = [0, 1], index_col = 0, low_memory = False)_____no_output_____tracks_data.head()_____no_output_____tracks_data.shape_____no_output_____tracks_data.columns_____no_output_____tracks_data.dtypes_____no_output_____

### Number of songs per genre, all time_____no_output_____

counts_by_genre = tracks_data.groupby(("track","genre_top")).size().sort_values()_____no_output_____plt.barh(counts_by_genre.index, counts_by_genre)

plt.xlabel("Number of songs")
plt.title("Number of songs by genre")
plt.show()_____no_output_____

### Multiple genres
TODO: Describe what we did, why we decided to keep the genres in a separate table, numerically encoded vs. text categories, etc._____no_output_____

tracks_data[("track", "genres_all")].sample(20)_____no_output_____tracks_data[tracks_data.index.isin([140595, 135557, 135466])]_____no_output_____

It turns out that some of the songs don't have a genre at all. We found out this was not a problem._____no_output_____

missing_genres_filter = tracks_data[("track", "genres_all")].str.len() == len("[]")_____no_output_____tracks_data[missing_genres_filter]_____no_output_____all_genres_cleaned = tracks_data[("track", "genres_all")].str.slice(1, -1)_____no_output_____all_genres_cleaned = all_genres_cleaned.str.split(", ", expand = True)_____no_output_____all_genres_cleaned = all_genres_cleaned.replace("", np.nan)_____no_output_____all_genres_cleaned = all_genres_cleaned.fillna(-1)_____no_output_____all_genres_cleaned = all_genres_cleaned.astype(int)_____no_output_____all_genres_cleaned.columns = [("track", "genre_" + str(column)) for column in all_genres_cleaned.columns]_____no_output_____all_genres_cleaned.to_csv("data/genres_for_songs.csv")_____no_output_____tracks_data = tracks_data.merge(all_genres_cleaned, on = "track_id")C:\Users\Yordan\Anaconda3\envs\tf2\lib\site-packages\pandas\core\reshape\merge.py:618: UserWarning: merging between different levels can give an unintended result (2 levels on the left, 1 on the right)
  warnings.warn(msg, UserWarning)
tracks_data = tracks_data.drop(("track", "genres_all"), axis = 1)_____no_output_____tracks_data_____no_output_____

We bring back the genre names._____no_output_____

genre_info = pd.read_csv("data/fma/genres.csv")_____no_output_____genre_info_____no_output_____genre_replacement_values = {}
for row_id, row in genre_info.iterrows():
    genre_replacement_values[row.genre_id] = row.title
genre_replacement_values[-1] = ""_____no_output_____genre_columns_filter = [("track", concrete) for (general, concrete) in tracks_data.columns if concrete.startswith("genre_") and concrete != "genre_top"]_____no_output_____tracks_data[genre_columns_filter] = tracks_data[genre_columns_filter].replace(genre_replacement_values)_____no_output_____tracks_data_____no_output_____

### Album info cleanup
TODO: Describe that we don't need the album columns (at least initially)._____no_output_____

album_columns = [(general, concrete) for (general, concrete) in tracks_data.columns if general == "album"]_____no_output_____tracks_data = tracks_data.drop(album_columns, axis = 1)_____no_output_____tracks_data_____no_output_____

### Evolution of genres
TODO: Describe the goals and the process
#### Converting the relevant columns to their appropriate types_____no_output_____

def convert_track_column_to_datetime(dataframe, track_column):
    dataframe[("track", track_column)] = pd.to_datetime(tracks_data[("track", track_column)])_____no_output_____convert_track_column_to_datetime(tracks_data, "date_created")
convert_track_column_to_datetime(tracks_data, "date_recorded")_____no_output_____num_tracks_by_year = tracks_data.groupby(tracks_data[("track", "date_recorded")].dt.year).size()_____no_output_____num_tracks_by_year_____no_output_____

#### Time evolution: all songs_____no_output_____

def plot_song_counts_by_year(
    num_tracks_by_year,
    min_year = None,
    max_year = None,
    plot_type = "bar",
    legend_title = None,
    show_figure = True):
    if min_year is None:
        min_year = num_tracks_by_year.index.min()
    if max_year is None:
        max_year = num_tracks_by_year.index.max()

    num_tracks_by_year_filtered = num_tracks_by_year[
        (num_tracks_by_year.index >= min_year) & (num_tracks_by_year.index <= max_year)]
    if plot_type == "bar":
        plt.bar(num_tracks_by_year_filtered.index, num_tracks_by_year_filtered, label = legend_title)
    elif plot_type == "line":
        plt.plot(num_tracks_by_year_filtered.index, num_tracks_by_year_filtered, label = legend_title)
    else:
        raise ValueError("The plot type must be either \"bar\" or \"line\"")

    plt.xlabel("Year")
    plt.ylabel("Number of songs")
    plt.xlim((min_year, max_year))
    if legend_title:
        plt.legend()
    if show_figure:
        plt.show()_____no_output_____plot_song_counts_by_year(num_tracks_by_year)_____no_output_____plot_song_counts_by_year(num_tracks_by_year, plot_type = "line")_____no_output_____

TODO: Describe missing data (and how we found out that it was indeed missing, not skewed), comparison of plots and results._____no_output_____

num_tracks_by_year.idxmax(), num_tracks_by_year.max()_____no_output_____num_tracks_by_year[(num_tracks_by_year.index >= 1925) & (num_tracks_by_year.index <= 1980)]_____no_output_____plot_song_counts_by_year(num_tracks_by_year, max_year = 1925)_____no_output_____plot_song_counts_by_year(num_tracks_by_year, min_year = 1980)_____no_output_____plot_song_counts_by_year(num_tracks_by_year, min_year = 2000, max_year = 2010)_____no_output_____plot_song_counts_by_year(num_tracks_by_year, min_year = 2000, max_year = 2010, plot_type = "line")_____no_output_____

#### Time evolution: per-genre split_____no_output_____

min_year, max_year = num_tracks_by_year.idxmin(), num_tracks_by_year.idxmax()

for genre_name, genre_data in tracks_data.groupby(("track", "genre_top")):
    num_tracks_in_genre_by_year = genre_data.groupby(genre_data[("track", "date_recorded")].dt.year).size()
    plot_song_counts_by_year(
        num_tracks_in_genre_by_year,
        min_year = min_year,
        max_year = max_year,
        plot_type = "line",
        show_figure = False)_____no_output_____for genre_name, genre_data in tracks_data.groupby(("track", "genre_top")):
    num_tracks_in_genre_by_year = genre_data.groupby(genre_data[("track", "date_recorded")].dt.year).size()
    plot_song_counts_by_year(
        num_tracks_in_genre_by_year,
        min_year = min_year,
        max_year = max_year,
        plot_type = "line",
        legend_title = genre_name,
        show_figure = True)_____no_output_____

TODO: Describe results, possibly make prettier plots (e.g. subplots) for final presentation._____no_output_____### Song durations: general behavior
TODO: Describe the process (from general to concrete analysis); describe asymmetry and how we found out about it, etc._____no_output_____

plt.hist(tracks_data[("track", "duration")], bins = "fd")

plt.xlabel("Duration [s]")
plt.ylabel("Number of songs")
plt.show()_____no_output_____positive_song_durations = tracks_data[("track", "duration")] > 0
log_transformed_durations = np.log(tracks_data[positive_song_durations][("track", "duration")])

plt.hist(log_transformed_durations, bins = "fd")

plt.xlabel("ln Duration")
plt.ylabel("Number of songs")
plt.show()_____no_output_____plt.figure(figsize = (15, 10))
for genre_name, genre_data in tracks_data.groupby(("track", "genre_top")):
    plt.hist(genre_data[("track", "duration")], label = genre_name, bins = "fd", alpha = 0.3)

    plt.xlabel("Duration [s]")
    plt.ylabel("Number of songs")
    plt.legend()
plt.show()_____no_output_____

TODO: Make a graph where durations stand out more. A good idea is to plot the histogram without filling in the bars (use `step` as the histogram type)._____no_output_____### Durations vs. genre: statistical significance testing
TODO: Describe what we did, and why. Present the final results as a pivot table._____no_output_____

durations_by_genre = {}
for genre_name, genre_data in tracks_data.groupby(("track", "genre_top")):
    durations_by_genre[genre_name] = genre_data[("track", "duration")].tolist()_____no_output_____ttest_ind(durations_by_genre["Classical"], durations_by_genre["Hip-Hop"])_____no_output_____for genre1 in durations_by_genre.keys():
    for genre2 in durations_by_genre.keys():
        print(genre1, genre2, ttest_ind(durations_by_genre[genre1], durations_by_genre[genre2]).pvalue)Blues Blues 1.0
Blues Classical 0.16478099078229572
Blues Country 0.012086660643773377
Blues Easy Listening 0.31249703110001964
Blues Electronic 0.7916591496680272
Blues Experimental 0.005032869759071465
Blues Folk 0.13163814368611731
Blues Hip-Hop 1.375340465294909e-06
Blues Instrumental 0.6544020862348228
Blues International 0.14830290786435762
Blues Jazz 0.00011717538312733254
Blues Old-Time / Historic 6.412549424791028e-10
Blues Pop 2.7770184268800892e-06
Blues Rock 0.07217046844358146
Blues Soul-RnB 0.692998749501362
Blues Spoken 0.00011845044277681152
Classical Blues 0.16478099078229572
Classical Classical 1.0
Classical Country 0.0010241319232339521
Classical Easy Listening 0.7925137311141895
Classical Electronic 9.411783809073676e-10
Classical Experimental 9.947146283825229e-06
Classical Folk 5.5523732146671205e-18
Classical Hip-Hop 1.1152855700736453e-46
Classical Instrumental 2.595713846904216e-07
Classical International 0.44040238810651366
Classical Jazz 0.00018390762792358195
Classical Old-Time / Historic 2.2325937750275714e-13
Classical Pop 1.2519820753258653e-38
Classical Rock 8.583930723880975e-41
Classical Soul-RnB 0.12634492266491668
Classical Spoken 4.608604193570985e-18
Country Blues 0.012086660643773377
Country Classical 0.0010241319232339521
Country Country 1.0
Country Easy Listening 0.03903808035120408
Country Electronic 0.003604359976708276
Country Experimental 1.9002473593999841e-07
Country Folk 0.28327040287750016
Country Hip-Hop 0.1310259615541611
Country Instrumental 0.26701205464138317
Country International 0.0001562942643387115
Country Jazz 3.6044007521060434e-11
Country Old-Time / Historic 0.002319488457560584
Country Pop 0.016278936047032885
Country Rock 0.49528232604448774
Country Soul-RnB 0.00219564247301929
Country Spoken 1.2393870520080501e-08
Easy Listening Blues 0.31249703110001964
Easy Listening Classical 0.7925137311141895
Easy Listening Country 0.03903808035120408
Easy Listening Easy Listening 1.0
Easy Listening Electronic 0.5816683551219992
Easy Listening Experimental 0.35801843683637025
Easy Listening Folk 0.12945286620415689
Easy Listening Hip-Hop 0.0004063032706017037
Easy Listening Instrumental 0.5103389753866481
Easy Listening International 0.8628177018312349
Easy Listening Jazz 0.17615780290863564
Easy Listening Old-Time / Historic 3.115361200586645e-05
Easy Listening Pop 0.0014653193721558132
Easy Listening Rock 0.09979174735492279
Easy Listening Soul-RnB 0.4709830126362572
Easy Listening Spoken 0.10147409642291252
Electronic Blues 0.7916591496680272
Electronic Classical 9.411783809073676e-10
Electronic Country 0.003604359976708276
Electronic Easy Listening 0.5816683551219992
Electronic Electronic 1.0
Electronic Experimental 1.933844765853131e-101
Electronic Folk 2.906368892125869e-12
Electronic Hip-Hop 3.253473464791086e-52
Electronic Instrumental 0.0008712384234456001
Electronic International 1.168335474209728e-07
Electronic Jazz 1.6839100771503285e-31
Electronic Old-Time / Historic 2.873533322156643e-16
Electronic Pop 1.8286244559205233e-51
Electronic Rock 1.94184108996009e-44
Electronic Soul-RnB 0.9678994885866253
Electronic Spoken 2.0170233728417883e-111
Experimental Blues 0.005032869759071465
Experimental Classical 9.947146283825229e-06
Experimental Country 1.9002473593999841e-07
Experimental Easy Listening 0.35801843683637025
Experimental Electronic 1.933844765853131e-101
Experimental Experimental 1.0
Experimental Folk 1.6727287272069476e-66
Experimental Hip-Hop 6.260368920854745e-125
Experimental Instrumental 3.1044676570694396e-38
Experimental International 9.095677251085231e-09
Experimental Jazz 0.24828355513431094
Experimental Old-Time / Historic 1.1597443917411718e-26
Experimental Pop 1.4622172172906213e-97
Experimental Rock 2.405553385105103e-283
Experimental Soul-RnB 0.0009052552306839726
Experimental Spoken 1.3526488146351964e-27
Folk Blues 0.13163814368611731
Folk Classical 5.5523732146671205e-18
Folk Country 0.28327040287750016
Folk Easy Listening 0.12945286620415689
Folk Electronic 2.906368892125869e-12
Folk Experimental 1.6727287272069476e-66
Folk Folk 1.0
Folk Hip-Hop 3.1142991726611676e-13
Folk Instrumental 0.09895061906743115
Folk International 1.7096145435643873e-19
Folk Jazz 1.4197635645654451e-49
Folk Old-Time / Historic 1.665217795641671e-08
Folk Pop 3.1915035552722186e-18
Folk Rock 0.16444783734805782
Folk Soul-RnB 0.02022613747258099
Folk Spoken 2.9721462350784937e-74
Hip-Hop Blues 1.375340465294909e-06
Hip-Hop Classical 1.1152855700736453e-46
Hip-Hop Country 0.1310259615541611
Hip-Hop Easy Listening 0.0004063032706017037
Hip-Hop Electronic 3.253473464791086e-52
Hip-Hop Experimental 6.260368920854745e-125
Hip-Hop Folk 3.1142991726611676e-13
Hip-Hop Hip-Hop 1.0
Hip-Hop Instrumental 8.36155388537007e-11
Hip-Hop International 8.352576468342459e-58
Hip-Hop Jazz 9.617268932713515e-120
Hip-Hop Old-Time / Historic 0.0006036094518608089
Hip-Hop Pop 7.254169610603327e-05
Hip-Hop Rock 9.372144671793716e-12
Hip-Hop Soul-RnB 3.356535237211986e-11
Hip-Hop Spoken 8.808516769491866e-120
Instrumental Blues 0.6544020862348228
Instrumental Classical 2.595713846904216e-07
Instrumental Country 0.26701205464138317
Instrumental Easy Listening 0.5103389753866481
Instrumental Electronic 0.0008712384234456001
Instrumental Experimental 3.1044676570694396e-38
Instrumental Folk 0.09895061906743115
Instrumental Hip-Hop 8.36155388537007e-11
Instrumental Instrumental 1.0
Instrumental International 5.673849960307794e-07
Instrumental Jazz 7.604305831651586e-18
Instrumental Old-Time / Historic 5.450616381192818e-05
Instrumental Pop 2.2783456099111886e-12
Instrumental Rock 0.00035577094356259643
Instrumental Soul-RnB 0.4183448832001656
Instrumental Spoken 2.942974136631281e-40
International Blues 0.14830290786435762
International Classical 0.44040238810651366
International Country 0.0001562942643387115
International Easy Listening 0.8628177018312349
International Electronic 1.168335474209728e-07
International Experimental 9.095677251085231e-09
International Folk 1.7096145435643873e-19
International Hip-Hop 8.352576468342459e-58
International Instrumental 5.673849960307794e-07
International International 1.0
International Jazz 1.361891058092968e-07
International Old-Time / Historic 4.357022331248559e-18
International Pop 1.4580162202073883e-48
International Rock 5.129822104859941e-39
International Soul-RnB 0.1259139206338292
International Spoken 3.0118910466905096e-24
Jazz Blues 0.00011717538312733254
Jazz Classical 0.00018390762792358195
Jazz Country 3.6044007521060434e-11
Jazz Easy Listening 0.17615780290863564
Jazz Electronic 1.6839100771503285e-31
Jazz Experimental 0.24828355513431094
Jazz Folk 1.4197635645654451e-49
Jazz Hip-Hop 9.617268932713515e-120
Jazz Instrumental 7.604305831651586e-18
Jazz International 1.361891058092968e-07
Jazz Jazz 1.0
Jazz Old-Time / Historic 1.5449402024158902e-37
Jazz Pop 5.994773659761958e-88
Jazz Rock 2.4761830451914135e-76
Jazz Soul-RnB 6.300791395169849e-06
Jazz Spoken 3.835774904342071e-07
Old-Time / Historic Blues 6.412549424791028e-10
Old-Time / Historic Classical 2.2325937750275714e-13
Old-Time / Historic Country 0.002319488457560584
Old-Time / Historic Easy Listening 3.115361200586645e-05
Old-Time / Historic Electronic 2.873533322156643e-16
Old-Time / Historic Experimental 1.1597443917411718e-26
Old-Time / Historic Folk 1.665217795641671e-08
Old-Time / Historic Hip-Hop 0.0006036094518608089
Old-Time / Historic Instrumental 5.450616381192818e-05
Old-Time / Historic International 4.357022331248559e-18
Old-Time / Historic Jazz 1.5449402024158902e-37
Old-Time / Historic Old-Time / Historic 1.0
Old-Time / Historic Pop 0.528596356741645
Old-Time / Historic Rock 3.8788430631199813e-07
Old-Time / Historic Soul-RnB 1.141413548376462e-13
Old-Time / Historic Spoken 1.0128962846880063e-24
Pop Blues 2.7770184268800892e-06
Pop Classical 1.2519820753258653e-38
Pop Country 0.016278936047032885
Pop Easy Listening 0.0014653193721558132
Pop Electronic 1.8286244559205233e-51
Pop Experimental 1.4622172172906213e-97
Pop Folk 3.1915035552722186e-18
Pop Hip-Hop 7.254169610603327e-05
Pop Instrumental 2.2783456099111886e-12
Pop International 1.4580162202073883e-48
Pop Jazz 5.994773659761958e-88
Pop Old-Time / Historic 0.528596356741645
Pop Pop 1.0
Pop Rock 5.42918297446536e-19
Pop Soul-RnB 2.7947532683617145e-10
Pop Spoken 5.960728158932374e-84
Rock Blues 0.07217046844358146
Rock Classical 8.583930723880975e-41
Rock Country 0.49528232604448774
Rock Easy Listening 0.09979174735492279
Rock Electronic 1.94184108996009e-44
Rock Experimental 2.405553385105103e-283
Rock Folk 0.16444783734805782
Rock Hip-Hop 9.372144671793716e-12
Rock Instrumental 0.00035577094356259643
Rock International 5.129822104859941e-39
Rock Jazz 2.4761830451914135e-76
Rock Old-Time / Historic 3.8788430631199813e-07
Rock Pop 5.42918297446536e-19
Rock Rock 1.0
Rock Soul-RnB 0.006776627603215871
Rock Spoken 1.6883702972769212e-200
Soul-RnB Blues 0.692998749501362
Soul-RnB Classical 0.12634492266491668
Soul-RnB Country 0.00219564247301929
Soul-RnB Easy Listening 0.4709830126362572
Soul-RnB Electronic 0.9678994885866253
Soul-RnB Experimental 0.0009052552306839726
Soul-RnB Folk 0.02022613747258099
Soul-RnB Hip-Hop 3.356535237211986e-11
Soul-RnB Instrumental 0.4183448832001656
Soul-RnB International 0.1259139206338292
Soul-RnB Jazz 6.300791395169849e-06
Soul-RnB Old-Time / Historic 1.141413548376462e-13
Soul-RnB Pop 2.7947532683617145e-10
Soul-RnB Rock 0.006776627603215871
Soul-RnB Soul-RnB 1.0
Soul-RnB Spoken 2.197185313801792e-06
Spoken Blues 0.00011845044277681152
Spoken Classical 4.608604193570985e-18
Spoken Country 1.2393870520080501e-08
Spoken Easy Listening 0.10147409642291252
Spoken Electronic 2.0170233728417883e-111
Spoken Experimental 1.3526488146351964e-27
Spoken Folk 2.9721462350784937e-74
Spoken Hip-Hop 8.808516769491866e-120
Spoken Instrumental 2.942974136631281e-40
Spoken International 3.0118910466905096e-24
Spoken Jazz 3.835774904342071e-07
Spoken Old-Time / Historic 1.0128962846880063e-24
Spoken Pop 5.960728158932374e-84
Spoken Rock 1.6883702972769212e-200
Spoken Soul-RnB 2.197185313801792e-06
Spoken Spoken 1.0

### Song lyrics
TODO: Describe how the dataset was generated, how we want to use it, and what we could do to model it (e.g. Naïve Bayes)._____no_output_____

with open("data/musixmatch/mxm_dataset_train.txt", "r", encoding = "utf-8") as file:
    for i in range(17):
        file.readline()
    words = file.readline()_____no_output_____words = words[1:].split(",")_____no_output_____words_____no_output_____words_dictionary = {}
for i in list(range(1, 5001)):
    words_dictionary[i] = {}_____no_output_____N_SONGS = 1000

song_ids = []
with open("data/musixmatch/mxm_dataset_train.txt", "r", encoding = "utf-8") as file:
    for i in range(18):
        file.readline()

    current_song_info = file.readline().strip()
    song_counter = 1
    while current_song_info is not None:
        current_song_features = current_song_info.split(",")
        song_id = current_song_features[0]
        song_ids.append(song_id)
        word_counts = current_song_features[2:]
        word_counts = [word_count_pair.split(":") for word_count_pair in word_counts]
        for [word_id, count] in word_counts:
            word_id = int(word_id)
            count = int(count)
            words_dictionary[word_id][song_id] = count

        if song_counter > N_SONGS:
            break
        current_song_info = file.readline()
        song_counter += 1_____no_output_____for word_id in words_dictionary:
    for song_id in song_ids:
        if song_id not in words_dictionary[word_id]:
            words_dictionary[word_id][song_id] = 0_____no_output_____word_counts = pd.DataFrame(words_dictionary)
del words_dictionary_____no_output_____word_counts_____no_output_____

### References
TODO: Link papers, other articles, methods, images, etc. that you have used. I'm leaving links to the datasets only.

[1] Million Song Dataset: http://millionsongdataset.com/

[2] MSD: MusiXMatch text features: http://millionsongdataset.com/musixmatch/

[3] Free Music Archive: https://freemusicarchive.org/_____no_output_____


# Notebook from imsb-uke/rna_augment
Path: run_me_first.ipynb

<code>
!pip install .Processing /projectbig/jupyternotebook/rnaseq_augmentation/rna_augment
Building wheels for collected packages: rna-augment
  Building wheel for rna-augment (setup.py) ... done
  Created wheel for rna-augment: filename=rna_augment-1.0-py3-none-any.whl size=4849 sha256=7a014434644fb2e9913b8d96903d65658feaa4b7b76d3115ad365d28cd04185c
  Stored in directory: /tmp/pip-ephem-wheel-cache-wmumywkh/wheels/00/ee/c8/8924be0c972354d6a50a3f2237d39b9490c91d31fa5a4de69d
Successfully built rna-augment
Installing collected packages: rna-augment
Successfully installed rna-augment-1.0

</code>