# Notebook from astropatel/SED
Path: Twinkle Tutorial.ipynb

# TWINKLE : THE SED GENERATOR_____no_output_____#### This tool, although a little rusty, will generate a stellar spectral energy distribution (SED) using empirical broad-band photometric data, fit to stellar models.

#### Right now, the two free parameters are the stellar radius and temperature. The gravity and metallicty are held fixed._____no_output_____-----------_____no_output_____# 1. Input File Structure_____no_output_____## 1.1 JSON Parameter File_____no_output_____The program works by feeding it a JSON parameter file that contains the names of folders and files that will be used by the code, boolean values that will dictate some fit hacks, plot commands and which photometry from the input file that will be used.

For instance, the example JSON file is located and titled $\textbf{./sed_paramfile.json}$_____no_output_____### 1.1.1 File set up:_____no_output_____In the JSON file, the "folders" section contains the following keys:

    > "topdir" - Top directory name.
    > "codedir" - directory under "topdir" that contains the code for twinkle.
    > "supportdir" - Directory where the empirical standard colors, stellar models, and response functions are kept.

and the "files" section contains the following keys:

    > "starfile" - The file that contains empirical stellar data (photometry, parallax, etc).
    > "bv_colorfile" - textfile that contains the empirical standard colors to use a reference.

If all the files and folders are located in a directory called $\textbf{stinfo_topdir}$, then the following files should be structured in the following way:

    > topdir/codedir/starfile
    > topdir/codedir/sed_paramfile.json
    > topdir/codedir/plot_sed.py
    > topdir/codedir/sed.py

    > topdir/codedir/supportdir/EMamajek_MSColors.txt
    > topdir/codedir/supportdir/Models/
    > topdir/codedir/supportdir/RSR/
_____no_output_____### 1.1.2 Photometric Bands:_____no_output_____You can use the JSON file to indicate which photometric band data to use for different purposes. There are  six main keys which are lists. Each key is an array of strings, where each string entry is the identifier for a specific photometric band.

    > "mags2use0_original"
        - All the bands to include at ANY POINT in the calculation.

The rest are subsets of "mags2use0_original"

    > "mags4Phot0_original"
        - Bands that will be fit to the stellar model

    > "mags4scale0_original"
        - Bands that are used to calculate a first order constant to scale the raw model flux to observed         surface flux.

    > "scaleSEDbands"
        - Bands that will be used to re-scale the fitted photosphere. To be used with the boolean key "scalephot".

    > "Remove_RedStars"

    > "mags4Dust0"
        - Bands at which to calculate the excess flux and eventually one or two blackbodies.
_____no_output_____For example:
    > "mags2use0_original" : ["BJ","VJ","J2M","H2M","Ks2M","W1","W2","W3","W4","HPACS160_flux","HPACS100_flux"],

    Will incorporate the Johnson B,V, 2MASS J,H,Ks, and WISE W1,W2,W3,W4, and Herschel 100, 160  photometric bands. The suffix "_flux" indicates that the input data is flux in jansky and not in units of magnitude.

    > "mags4Phot0_original"  : ["BJ","VJ","J2M","H2M","Ks2M","W1","W2"],

    This indicates that only the Johnson B,V, 2MASS J,H,Ks, and WISE W1, W2 bands to use to fit the stellar photosphere. _____no_output_____##### For all available filters, check out the $\textbf{/Docs/available_filters.txt}$ file_____no_output_____## 1.1 Stellar Photometric Data File_____no_output_____The text file listed as "topdir/codedir/starfile" contains information for all the stars for which you want to create an SED. Each row contains information for a single star, where a column is the meta data for that star.

The folder can be tab or space separated.

The first row is the header. There are required and optional header names. They can be found in the file $\textbf{/Docs/Starfile_ColumnDescription.txt}$_____no_output_____You can add as many unique photometric columns as you wish. As described in the $\textbf{Starfile_ColumnDescription.txt}$, any column header key with the suffix $\_flux$, should be in units of Jy. Otherwise, it will be assumed the units are in magnitudes._____no_output_____-----------_____no_output_____# 2. Running the Code_____no_output_____## 2.1 Set up_____no_output_____

from __future__ import print_function
import sys, os
import json
import numpy as np
import matplotlib.pyplot as plt
from twinkle import twinkle

%load_ext autoreload
%autoreload 2_____no_output_____

#### INITIATE INTERACTIVE MATPLOTLIB PLOTTING_____no_output_____##### If this doesn't work, change to "%matplotlib inline" and update both matplotlib and jupyter notebook to make nbagg backend to work_____no_output_____

%matplotlib nbagg_____no_output_____

## 2.2 Load JSON file
##### Change path to where you store your paramfile_____no_output_____

jfile = os.path.join(os.path.expanduser('~'),'Dropbox','Research','twinkle','twinkle','sed_paramfile.json')
#jfile = 'Users/rpatel/sed_paramfile.json'_____no_output_____script = open(jfile).read()
specs = json.loads(script)_____no_output_____

## 2.3  Select a star and feed JSON object into twinkle_____no_output_____

# select a name in the stellar text file
# check out the example file "sample_stardata.txt" as an example
sname = 'Star_ID1'
tauA = twinkle.Star(jfile, starname=sname)
stoo = twinkle.STools> /Users/rpatel/Dropbox (Personal)/Research/twinkle/twinkle/sed.py(1322)__init__()
-> workingdir = directories.WorkingDir(specs['folders']['topdir'])
(Pdb) c
-------------------------------
      Loading All Gridmodels
Loaded NextGen of g=40, met=0
       Done Loading Models
-------------------------------
Bands used to fit photosphere: [u'BJ' u'H2M' u'J2M' u'Ks2M' u'VJ' u'W1' u'W2']
Bands used to scale photosphere: [u'BJ' u'H2M' u'J2M' u'Ks2M' u'VJ']

#### AND THAT'S IT FOLKS! Your SED is lit!

In the output you'll see which bands were used for the photospheric model fits, the associated chi2 value, and the optimal values for the stellar radius and temperature from the fit
_____no_output_____### Now Let's do some more, shall we?_____no_output_____-----------_____no_output_____# 3. What Can I Access?_____no_output_____Once a twinkle object has been instantiated, you can access various information for that star, and the functions in both twinkle.py and sed.py._____no_output_____

print('Star:',tauA.starname)
print('distance:',tauA.disti)

# model used and fitted temperature (Kelvin)
print(tauA.modeli, tauA.StarTemp)Star: Star_ID1
distance: 64.1025641026
NextGen 8923.73162413
# All the bands included in the simulation.
# fluxes are also calculated for each of these
print(tauA.mags2use)[u'BJ', u'H2M', u'HPACS100', u'HPACS160', u'J2M', u'Ks2M', u'VJ', u'W1', u'W2', u'W3', u'W4']

-----------_____no_output_____# 4. Plot Stellar SED_____no_output_____### You can either plot the data with the built in "plot_photosphere" and "plot_observedData" functions, or access the global data._____no_output_____

import matplotlib.ticker as mtick

# SET UP AXIS
ax = plt.figure(figsize=(6,6)).add_subplot(111)

# PLOT PHOTOSPHERIC AND LITERATURE DATA
tauA.plot_photosphere(ax, pointsize=4,lcolor='blue',pcolor='blue',lw=.5)
tauA.plot_observedData(ax, pointsize=3,lw=.8)

# ADDITIONAL PLOT STUFF
plt.loglog()
#ax.set_xlim(.2,100)
#ax.set_ylim(1e-13,1e-6)
ax.set_ylabel(r'$\lambda F_{\lambda}\ [erg\,\ s^{-1}\ cm^{-2}] $',fontsize=20, family='sans-serif')
ax.set_xlabel(r'$\lambda\ [\mu m]$', fontsize=20, family='sans-serif')
plt.subplots_adjust(left=.19, right=.96, bottom=.15, top=.93)
ax.xaxis.set_major_formatter(mtick.FormatStrFormatter('%d'))

ax.annotate(r'{}'.format(tauA.starname), xy=(0.73, 0.53),xycoords='axes fraction',
            fontsize=20, family='Times New Roman')

ax.annotate(r'$T_* = {:.2f}$ K'.format(tauA.StarTemp), xy=(0.73, 0.47),
            xycoords='axes fraction',fontsize=10, family='Times New Roman', color='blue')

plt.show()_____no_output_____

# 4. Calculate Excess Flux_____no_output_____

# There's a function that will calculate the excess flux at bands listed in mags4Dust
tauA.calc_excessflux()HPACS100  excess flux  2.38746427586e-17 erg/s/cm2/ang
HPACS160  excess flux  4.66817105066e-18 erg/s/cm2/ang
W3  excess flux  2.14361397755e-16 erg/s/cm2/ang
W4  excess flux  3.97925224579e-17 erg/s/cm2/ang
# SET UP AXIS
waveEx = tauA.waveEx
exFlux = tauA.fluxEx
exFlux_e = tauA.efluxEx

ax3 = plt.figure(figsize=(6,6)).add_subplot(111)
ax3.set_title('Excess Flux')
ax3.errorbar(waveEx * twinkle._ANG2MICRON, waveEx * exFlux,
             yerr=waveEx * exFlux_e, fmt='mD',ms=6)
ax3.set_ylabel(r'$\lambda F_{\lambda}\ [erg\,\ s^{-1}\ cm^{-2}] $',fontsize=20, family='sans-serif')
ax3.set_xlabel(r'$\lambda\ [\mu m]$', fontsize=20, family='sans-serif')
ax3.xaxis.set_major_formatter(mtick.FormatStrFormatter('%d'))
plt.show()_____no_output_____

## 5.1 Plot New SED With FAR - IR Data_____no_output_____# 6. FIT ONE TO TWO BLACKBODIES_____no_output_____## 6.1 Try fitting 1 blackbody first_____no_output_____

# RESET FULL SPECTRUM - bbdust_flux
tauA.fullspectrum = tauA.StarPhotosphere[1][:]_____no_output_____# FOR NOW, USING OLD FITTING ROUTINES. WILL EVENTUALLY SWITCH TO PANDAS
# OR SCIPY
import mosaic_tools as mt
ft = mt.FittingTools()

# ==============================================
# INITIAL GUESS
tempdust = 100.
SEDx_ang = tauA.StarPhotosphere[0][:]
bbdust = {}
# ==============================================
# CALCULATE BROADBAND PHOTOMETRY ACROSS IR EXCESS WAVELENGTHS
for i,band in enumerate(tauA.mags4Dust):
    bbdust[band] = stoo.blackbody(SEDx_ang,np.array([tempdust]),1, np.array([band]))[0]

bbdust_flux = np.array(zip(*sorted(bbdust.items()))[1]).flatten()

FluxNorm_dust = np.average(exFlux / bbdust_flux, weights=1. / exFlux_e)

# INITIAL GUESS FOR DUST RADIUS
Rad_dust = np.sqrt(FluxNorm_dust / tauA.su2ea_dust)
print('Initial guess for dust radius: {} AU'.format(Rad_dust))
_____no_output_____

from scipy.optimize import curve_fit_____no_output_____p0_dust = np.array([tempdust, 30])
popt, pcov = curve_fit(stoo.blackbody, SEDx_ang, exFlux, p0=p0_dust, sigma=exFlux_e,**)_____no_output_____

# FIT BLACKBODY
#p0_dust = np.array([tempdust, Rad_dust])
p0_dust = np.array([tempdust, Rad_dust])
nparams = len(p0_dust)
fa_Dust = {'x': SEDx_ang, 'y': exFlux, 'err': exFlux_e,
           'func': stoo.blackbody, 'su2ea1': tauA.su2ea_dust, 'bands': tauA.mags4Dust,'bulk':True}

#parinfo_dust = [{'value': 0., 'step': 0, 'limits': [0, 0], 'limited': [0, 0], 'fixed': 0} for m in
#                range(nparams)]
parinfo_dust = [{'value': 0., 'step': 0, 'limits': [0, 0], 'limited': [0, 0], 'fixed': 0} for m in
                range(nparams)]

for k in range(nparams): parinfo_dust[k]['value'] = p0_dust[k]
parinfo_dust[0]['relstep'] = .2
parinfo_dust[1]['relstep'] = .2
parinfo_dust[0]['limited'] = [1,1]
parinfo_dust[0]['limits'] = [10,1000]

m_dust = mt.mpfit(ft.deviates_from_model, parinfo=parinfo_dust, functkw=fa_Dust)#,quiet=1)

# from scipy.optimize import curve_fit
# popt, pcov =
_____no_output_____

### Print out Chi2 and dust temperatures_____no_output_____

# CALCULATE SINGLE TEMP BLACKBODY FIT
p0_dust,errors = m_dust.params, m_dust.perror
chi2 = m_dust.fnorm/m_dust.dof
print('Chi2={:.2f}, T={:.1f} K , Radius = {:.1f} AU'.format(chi2,*p0_dust)) # , p0_dust, errors
#p0_dust[0] = p0_dust[0] * 1.2
dust_flux = stoo.blackbody(SEDx_ang, p0_dust,tauA.su2ea_dust)_____no_output_____

## 6.2 Plot New SEDs With Fitted Blackbody_____no_output_____

# SAVE THE FULL SPECTRUM IN CASE YOU NEED TO RESET
fullspectrum_0 = tauA.StarPhotosphere[1][:]_____no_output_____ax5 = plt.figure(figsize=(8,5)).add_subplot(111)

tauA.plot_photosphere(ax5, pointsize=8,lcolor='blue',pcolor='blue',lw=0.5)

tauA.fullspectrum = tauA.StarPhotosphere[1] + dust_flux

plt.plot(SEDx_ang * twinkle._ANG2MICRON, dust_flux * SEDx_ang, 'm-', ls='--',lw=2)

ax5.errorbar(waveEx * twinkle._ANG2MICRON, waveEx * exFlux,
             yerr=waveEx * exFlux_e, fmt='mD',ms=6)
tauA.plot_observedData(ax5,pointsize=8,lw=1.5)
ax5.set_title('Excess Flux + BLACKBODY')
ax5.set_xlim(.2,300)
ax5.set_ylim(1e-15,1e-6)
ax5.set_ylabel(r'$\lambda F_{\lambda}\ [erg\,\ s^{-1}\ cm^{-2}] $',fontsize=20, family='sans-serif')
ax5.set_xlabel(r'$\lambda\ [\mu m]$', fontsize=20, family='sans-serif')
plt.tight_layout()
ax5.xaxis.set_major_formatter(mtick.FormatStrFormatter('%d'))
ax5.annotate(r'{}'.format(tauA.starname), xy=(0.73, 0.67),xycoords='axes fraction',
            fontsize=20, family='Times New Roman')
ax5.annotate(r'$T_* = {:.2f}$ K'.format(tauA.StarTemp), xy=(0.73, 0.63),
            xycoords='axes fraction',fontsize=10, family='Times New Roman', color='blue')
ax5.xaxis.set_major_formatter(mtick.FormatStrFormatter('%d'))
plt.loglog()

# ===============================================================================================================
ax4 = plt.figure(figsize=(8,5)).add_subplot(111)
ax4.set_title('Excess Flux + BLACKBODY')
ax4.errorbar(waveEx * twinkle._ANG2MICRON, waveEx * exFlux,
             yerr=waveEx * exFlux_e, fmt='mD',ms=6)
ax4.plot(SEDx_ang * twinkle._ANG2MICRON, dust_flux * SEDx_ang,'m-',lw=2)
ax4.set_ylabel(r'$\lambda F_{\lambda}\ [erg\,\ s^{-1}\ cm^{-2}] $',fontsize=20, family='sans-serif')
ax4.set_xlabel(r'$\lambda\ [\mu m]$', fontsize=20, family='sans-serif')
plt.tight_layout()
ax4.xaxis.set_major_formatter(mtick.FormatStrFormatter('%d'))
ax4.set_xlim(8,300)
ax4.set_ylim(1e-13,1e-9)

plt.loglog()
plt.show()_____no_output_____

## 6.3 Fitting 2 blackbody's to excess flux_____no_output_____

# FOR NOW, USING OLD FITTING ROUTINES. WILL EVENTUALLY SWITCH TO PANDAS
# OR SCIPY

# ==============================================
# INITIAL GUESS
td1, td2, td3 = 500, 100., 40
SEDx_ang = tauA.StarPhotosphere[0]
bbdust = {}
# ==============================================
# CALCULATE BROADBAND PHOTOMETRY ACROSS IR EXCESS WAVELENGTHS
for band in tauA.mags4Dust:
    bbdust[band] = stoo.NBlackBody(SEDx_ang,np.array([td1,1,td2,1,td3,1]),1, np.array([band]))[0]

bbdust_flux = np.array(zip(*sorted(bbdust.items()))[1]).flatten()

FluxNorm_dust = np.average(exFlux / bbdust_flux,
                           weights=1. / exFlux_e)

# INITIAL GUESS FOR DUST RADIUS
Rad_dust = np.sqrt(FluxNorm_dust / tauA.su2ea_dust)
print(Rad_dust)_____no_output_____# FIT BLACKBODY
p0_dust = np.array([td1,Rad_dust,td2, Rad_dust*2,td3,Rad_dust**3])
nparams = len(p0_dust)
# fa_Dust = {'x': SEDx_ang, 'y': exFlux, 'err': exFlux_e,
#            'func': stoo.doubleBB, 'su2ea1': tauA.su2ea_dust, 'bands': tauA.mags4Dust,'bulk':True}
fa_Dust = {'x': SEDx_ang, 'y': exFlux, 'err': exFlux_e,
           'func': stoo.NBlackBody, 'su2ea1': tauA.su2ea_dust, 'bands': tauA.mags4Dust,'bulk':True}

parinfo_dust = [{'value': 0., 'relstep': 0, 'limits': [0, 0], 'limited': [0, 0], 'fixed': 0}
                for m in range(nparams)]

for k in range(nparams): parinfo_dust[k]['value'] = p0_dust[k]
parinfo_dust[0]['relstep'] = 0.3
parinfo_dust[1]['relstep'] = 0.2
parinfo_dust[0]['limited'] = [1,1]
parinfo_dust[0]['limits'] = [200,1000]

parinfo_dust[2]['relstep'] = 0.3
parinfo_dust[3]['relstep'] = 0.2
parinfo_dust[2]['limited'] = [1,1]
parinfo_dust[2]['limits'] = [50,100]

parinfo_dust[4]['relstep'] = 0.3
parinfo_dust[5]['relstep'] = 0.2
parinfo_dust[4]['limited'] = [1,1]
parinfo_dust[4]['limits'] = [2,80]

m_dust = mt.mpfit(ft.deviates_from_model, parinfo=parinfo_dust, functkw=fa_Dust)#,quiet=1)
_____no_output_____

### Print out Chi2 and dust temperatures_____no_output_____

# CALCULATE SINGLE TEMP BLACKBODY FIT
p0_dust,errors = m_dust.params, m_dust.perror
#chi2 = m_dust.fnorm/m_dust.dof
print('Chi2={:.2f}'.format(chi2))# , p0_dust, errors
print(p0_dust)
dust_flux = stoo.NBlackBody(SEDx_ang, p0_dust,tauA.su2ea_dust)_____no_output_____

## 6.2 Plot New SEDs With Fitted Blackbody_____no_output_____

# RESET FULL SPECTRUM -
tauA.resetFullSpectrum()_____no_output_____ax5 = plt.figure(figsize=(8,5)).add_subplot(111)
tauA.plot_photosphere(ax5, pointsize=8,lcolor='blue',pcolor='blue',lw=0.5)
tauA.fullspectrum = tauA.StarPhotosphere[1] + dust_flux

plt.plot(SEDx_ang * twinkle._ANG2MICRON, dust_flux * SEDx_ang, 'm-', ls='--',lw=2)
ax5.errorbar(waveEx * twinkle._ANG2MICRON, waveEx * exFlux,
             yerr=waveEx * exFlux_e, fmt='mD',ms=6)
tauA.plot_observedData(ax5,pointsize=8,lw=1.5)
ax5.set_title('Full Spectrum')
ax5.set_xlim(.2,300)
ax5.set_ylim(1e-15,1e-6)
ax5.set_ylabel(r'$\lambda F_{\lambda}\ [erg\,\ s^{-1}\ cm^{-2}] $',fontsize=20, family='sans-serif')
ax5.set_xlabel(r'$\lambda\ [\mu m]$', fontsize=20, family='sans-serif')
plt.subplots_adjust(left=.19, right=.96, bottom=.15, top=.93)
ax5.xaxis.set_major_formatter(mtick.FormatStrFormatter('%d'))
ax5.annotate(r'{}'.format(tauA.starname), xy=(0.73, 0.67),xycoords='axes fraction',
            fontsize=20, family='Times New Roman')
ax5.annotate(r'$T_* = {:.2f}$ K'.format(tauA.StarTemp), xy=(0.73, 0.63),
            xycoords='axes fraction',fontsize=10, family='Times New Roman', color='blue')
ax5.xaxis.set_major_formatter(mtick.FormatStrFormatter('%d'))
plt.loglog()

# ===============================================================================================================
ax4 = plt.figure(figsize=(8,5)).add_subplot(111)
ax4.set_title('Excess Flux')
ax4.errorbar(waveEx * twinkle._ANG2MICRON, waveEx * exFlux,
             yerr=waveEx * exFlux_e, fmt='mD',ms=6)
ax4.plot(SEDx_ang * twinkle._ANG2MICRON, dust_flux * SEDx_ang,'m-',lw=2)
ax4.set_ylabel(r'$\lambda F_{\lambda}\ [erg\,\ s^{-1}\ cm^{-2}] $',fontsize=20, family='sans-serif')
ax4.set_xlabel(r'$\lambda\ [\mu m]$', fontsize=20, family='sans-serif')
plt.subplots_adjust(left=.19, right=.96, bottom=.15, top=.93)
ax4.xaxis.set_major_formatter(mtick.FormatStrFormatter('%d'))
ax4.set_xlim(8,300)
ax4.set_ylim(1e-13,1e-9)

plt.loglog()
plt.show()_____no_output_____


# Notebook from ds-connectors/EPS-88-FA21
Path: week10_inclass/Week10_InClass_Glacial_Cycles.ipynb

# Drivers of glacial-interglacial cycles
_____no_output_____### Import the scientific python packages we will need_____no_output_____

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd_____no_output_____

The following video briefly describes how Earth's orbital fluccuations influence planetary climate:

https://youtu.be/iA788usYNWA

The Earth's orbital parameters of ellipticity, obliquity and precession vary in predictable ways.  One commonly used model for variations in them over the last few hundred million years was published by Laskar et al. (2004; http://dx.doi.org/10.1051/0004-6361:20041335).

> This solution has been improved...by using a direct integration of the gravitational equations for the orbital motion, and by improving the dissipative contributions, in particular in the evolution of the Earth–Moon System. The orbital solution has been used for the calibration of the Neogene period (Lourens et al.  [CITE]), and is expected to be used for age calibrations of paleoclimatic data over 40 to 50 Myr, eventually over the full Palaeogene period (65 Myr) with caution. Beyond this time span, the chaotic evolution of the orbits prevents a precise determination of the Earth's motion. However, the most regular components of the orbital solution could still be used over a much longer time span. - Laskar et al. (2004)

Let's take a look for the behavior of the last few million years using the data file from the Laskar et al. (2004) paper.

## Import the Lasker orbital solution_____no_output_____

orbital_cycles = pd.read_csv('./data/INSOLN.LA2004.BTL.100.csv')
orbital_cycles.head()_____no_output_____

**Code for you to write**

Define a variable called `orbital_cycles_1Myr` that is filtered to only have data for the past 1 million years (1000 ka), and plot the three orbital parameters._____no_output_____

plt.figure(figsize=(8,8))

plt.subplot(3,1,1)

plt.subplot(3,1,2)

plt.subplot(3,1,3)

plt.tight_layout()
plt.show()_____no_output_____

## Relating these cycles to incoming insolation

You can see a lot of cycles on different time scales. The question is how this relates to the amount of insolation.  In the literature, there are many attempts to convert the orbital parameters, like those in the plot above, to the amount of insolation received by the Earth's atmosphere as a function of latitude and age. You can get such estimates here: https://www.ncdc.noaa.gov/paleo-search/study/5792 associated with this paper (Huybers, P. 2006, https://dash.harvard.edu/bitstream/handle/1/3382981/Huybers_EarlyPleistoceneSummer.pdf).

It is traditional to consider the amount of insolation received at 65$^{\circ}$N.  So let's take a look._____no_output_____

insolation = pd.read_csv('./data/j_65north.csv')
insolation.head()_____no_output_____

**Code for you to write**

Make a plot using `plt.plot` of the insolation curve. Use `plt.xlim()` to make it so that time is limited and goes from 1000 ka to 0 ka (1,000 thousand years ago [same as 1 million] to present-day)._____no_output_____

plt.figure(figsize=(8,5))

plt.tight_layout()
plt.show()_____no_output_____

## How big are these insolation changes?

These changes in insolation are interpretted to be important in driving glacial-interglacial cycles. How big are the changes in insolation?

**Code for you to write**

*Calculate the mean, minimum, and maximum insolation from the data. Then use these values to calculate the percentage difference between the maximum insolation and the mean. Do the same between the maximum and the minimum.*

**Does this change seem like a big change sufficient to drive these cycles?**_____no_output_____## Spectral analysis

Both insolation and CO$_2$ have a lot of wiggles over the last million years, but are the wiggles related to each other?  One way to look at this is using _time series analysis_ to answer the basic question: *Do the two data sets have wiggles with the same frequencies?*

The analysis boils down to this:

- According to Fourier, any periodic function $f(t)$ can be represented as the sum of a series of sines and cosines:

$$f(t)=a_0+ \sum_{k=1}^{\infty} \bigr[a_k    \cos  \bigr(  \frac{k \pi t}{L}  \bigr)  + b_k    \sin  \bigr( \frac{k \pi t}{L} \bigl) \bigl]  $$

Where $a_k$ and $b_k$ are the Fourier coefficients and control the amplitude of the sinusoidal wave, $-L$ to $L$ is the domain of the periodic function, $k$ is the index of the sinusoidal wave, and $\frac{k \pi}{L}$ gives the angular frequency of each wave. Each $k$th-wave has a different amplitude and frequency, higher $k$ waves have higher frequency.

- You can represent data as a series in time (in the time-domain) as we have been doing OR you can represent the data in terms of frequency, looking for the _power_ in the data as a function of frequency.  This is known as the _power spectrum_.

Let us take advantage of a `signal.periodogram` function in the `scipy` package.  That module has functions that allow us to  calculate the _power spectral density_ for a time series.  As a result we will be able to generate a _periodogram_, which is a plot of power versus frequency.

We will also use a _window_ in the periodogram calculation.  What a _window_ does is multiply the time series by a function (called a taper) that weights information, suppressing data at the edges of the window and focussing on the center of the window.  The simplest window is a _box car_ which gives equal weight to everything inside the window.  In the following, we will use a _Hann window_ which looks more like a bell curve. You can check it out here: https://en.wikipedia.org/wiki/Window_function_____no_output_____## What is the dominant periodic signal in the insolation curve?
_____no_output_____

from scipy import signal_____no_output_____insolation_freqs,insolation_power = signal.periodogram(insolation['Insolation (W/m^2)'],window='hann')

plt.plot(insolation_freqs,insolation_power,label='Insolation power spectra',linewidth=2)
plt.xlim(.001,.06)
plt.xlabel('Frequency, 1/ka')
plt.ylabel('Power')
plt.legend()
plt.show()_____no_output_____

## What does this dominant peak correspond to?

We know that eccentricity is supposed to have a dominant period at 100 kyr, obliquity at 41 kyr and precession at 23 and 19 kyr. Remember that these numbers are expressed in terms of the period, which is the inverse of the frequency so the frequency of them with be the inverse of the period._____no_output_____

eccentricity_frequency =
obliquity_frequency =
precession_frequency_1 =
precession_frequency_2 = _____no_output_____

**Code for you to write**

Plot the periodogram and plot the eccentricity and obliquity lines as well._____no_output_____

insolation_freqs,insolation_power = signal.periodogram(insolation['Insolation (W/m^2)'],window='hann')
plt.plot()
plt.axvline()

plt.legend()
plt.show()

#make a semilogy plot - a plot with a log-y axis and linear x-axis
_____no_output_____

## Let's have a look at the ice core CO$_2$ data. Does the signal match with the insolation signal?_____no_output_____

ice_core_co2 = pd.read_csv('./data/antarctica2015co2composite.txt',header=137,sep='\t')
ice_core_co2['age_ka'] = ice_core_co2['age_gas_calBP']/1000
ice_core_co2.tail()_____no_output_____

To analyze these data, we need to have an evenly spaced timeseries. We can use the scipy interpolation capabilities to do so. Here is an simple example from the scipy documentation:_____no_output_____

from scipy.interpolate import interp1d

x = np.linspace(0, 10, 11)
y = np.cos(-x**2/9.0)
f = interp1d(x, y) #linear interpolation
f2 = interp1d(x, y, kind='cubic') #cublic spline interpolation

xnew = np.linspace(0, 10, 41)
plt.plot(x, y, 'o')
plt.plot(xnew, f(xnew), '-')
plt.plot(xnew, f2(xnew), '--')
plt.legend(['data', 'linear', 'cubic'], loc='best')
plt.show()_____no_output_____

We can apply such an interpolation to the ice core data and resample it at evenly spaced 1 kyr intervals:_____no_output_____

ice_interp =

xnew = np.linspace()
co2_interp = ice_interp(xnew)

plt.figure(figsize=(8,5))
plt.plot(ice_core_co2['age_ka'],ice_core_co2['co2_ppm'], 'o')
plt.plot(xnew, co2_interp, '-')
plt.legend(['data', 'interpolation'], loc='best')
plt.xlim(800,0)
plt.xlabel('Age (ka)')
plt.ylabel('CO$_2$ (ppm)')
plt.show()_____no_output_____#Compare the Ice Core CO2 to Insolation

plt.figure(figsize=(8,6))

plt.subplot(2,1,1)

plt.subplot(2,1,2)

plt.show()_____no_output_____

## What is the dominant signal in the ice sheet CO$_2$ data?

**Code for you to write**

Take the same approach above and use the `signal.periodogram()` function to calculate the power spectral density of the CO$_2$ data._____no_output_____

#Write code here_____no_output_____

### What is the dominant signal in the 65ºN insolation curve and how does it compare to dominant period in the ice sheet CO$_2$ data?

*Write your answer here*_____no_output_____## Longer timescale paleoclimate records

The ice core record is a very impressive one, but unfortunately, it only goes back ~800,000 years. While these seems like a long time, it is only 0.02% of the history of the Earth._____no_output_____One way that we can go further back with a paleoclimate record is using marine fossils such as formainifera. Foraminifera are made of calcium carbonate: CaCO$_3$

The oxygen isotopes ($^{16}$O vs $^{18}$O) in the CaCO$_3$ forms a record of both of temperature and ice volume. $\delta^{18}$O is a way to express the ratio between $^{16}$O and $^{18}$O. These oxygen isotopes have different mass and therefore for have different evaporation rates from the oceans. This leads to fractionation of the oxygen isotopes and a changing ratio, $\delta^{18}$O, depending on temperature.

$$\delta ^{18}O={\Biggl (}{\frac  {{\bigl (}{\frac  {^{{18}}O}{^{{16}}O}}{\bigr )}_{{sample}}}{{\bigl (}{\frac  {^{{18}}O}{^{{16}}O}}{\bigr )}_{{standard}}}}-1{\Biggr )}\times 1000\ ^{{o}}\!/\!_{{oo}}$$

When water is warm $\delta ^{18}$O is low and when water is cold $\delta ^{18}$O is high. There is a similar relationship with ice volume: when there is less ice the $\delta ^{18}$O of water is lower and when there is more ice (cold) $\delta ^{18}$O of water is higher which ends up contributing to the value of the fossil foraminifera.

So higher $\delta^{18}$O values are associated with a colder planet and lower $\delta^{18}$O values with a warmer planet. This video is provides a good explanation of this https://youtu.be/YfRDNyB1XOY

A compilation of these data was published by Lisecki and Raymo (2005, http://dx.doi.org/10.1029/2004PA001071) called the LR04 stack.  This is a stack of 58 records of oxygen isotopic variations, several of which were independently dated using magnetostratigraphy, from all over the world's oceans.

Let's import the data and take a look at the record: _____no_output_____

LR04_d18O = pd.read_csv('./data/LR04stack.csv')
LR04_d18O.tail()_____no_output_____plt.plot(LR04_d18O['Age (ka)'],LR04_d18O['d18O'])
plt.xlabel('Age (ka)')
plt.ylabel('$\delta ^{18}$O')
plt.xlim(5320,0)
plt.show()_____no_output_____d18O_1Ma=LR04_d18O[LR04_d18O['Age (ka)']<1000] # filter data for last 1Ma

plt.plot(d18O_1Ma['Age (ka)'],d18O_1Ma['d18O'])
plt.title('LR04 oxygen isotopic variation stack', fontsize=14)
plt.xlabel('Age (ka)', fontsize=14)
plt.ylabel('$\delta ^{18}$O', fontsize=14)
plt.ylim(5,3)
plt.xlim(1000,0)
plt.show()_____no_output_____plt.subplot(2,1,1)
plt.plot(xnew, co2_interp)
plt.ylabel('$CO_2$ ppm')
plt.xlim(800,0)

plt.subplot(2,1,2)
plt.plot(d18O_1Ma['Age (ka)'],d18O_1Ma['d18O'])
plt.title('LR04 oxygen isotopic variation stack', fontsize=14)
plt.xlabel('Age (ka)', fontsize=14)
plt.ylabel('$\delta ^{18}$O', fontsize=14)
#plt.ylim(1,0)
plt.xlim(800,0)
plt.tight_layout()
plt.show()_____no_output_____

Let's do the same signal analysis in order to analyze the power spectra, and plot both linear and semilogy._____no_output_____

#Write code here_____no_output_____

#### The correlation between two data series is a good way to test to see if they are similar, possibly resulting from the same underlying cause.

We will use the Pearson Product-Moment Correlation defined as:

r(x,y)=$\frac{\sum (x_i-\bar{x})(y_i-\bar{y})}{\sqrt{\sum (x_i-\bar{x})^2 \sum (y_i-\bar{y})^2}}$, the values range between -1 to 1, where -1 and 1 are perfect anti-correlationa and correlation, zero is no correlation.

The function we can use is `np.corrcoef()`_____no_output_____

# Lets explore the correlation between the CO2 and the dO18 data sets.
#To do this we use the np.corrcoef() function

#interpolate the d18O data the same way as CO2

corout=np.corrcoef(co2_interp, d18O_interp)
datacor=corout[0][1]
print(f'Correlation between CO2 and dO18 is {datacor:.3f}')

#How significant is this correlation? - use a bootstrap randomly sampling the d180_interp data
#as was done in a previous assignment, plotting the historgram, the 95% confidence

cc=np.zeros(10000)
for i in np.arange(0,10000,1):

_____no_output_____

### Interpretation

What orbital frequencies do you observe spectral peaks at in the $\delta ^{18}$O power spectra? How does that compare to what we saw in the CO$_2$ data? Are there peaks at other frequencies?

How strong is the correlation between the CO2 and the dO18 observations? How does it compare to random?

How does the sign of the correlation relate in terms of inferred climate?_____no_output______Write your answers here.______no_output_____### Turn in the Notebook

**Export as pdf and upload to bCourses.**_____no_output_____


# Notebook from kazewong/nrpytutorial
Path: notebook/scalarf_bssn.ipynb

# BSSN scalar field

## This module implements a basic numerical relativity code to evolve a scalar field profile in *spherical coordinates*

### This is a spherically symmetric problem, so we can set the symmetry about the $\theta$- and $\phi$-axes. Not sampling in the $\theta$ and $\phi$  directions greatly speeds up the simulation.

### NRPy+ Source Code for this module:
1. [BSSN_SF/ID_array_psi.py](BSSN_SF/ID_array_psi.py): Initial data to be input later; sets all ADM variables in Cartesian basis:
1. [BSSN_SF/CartesianADMID_to_BSSNCurvilinearID.py](BSSN_SF/CartesianADMID_to_BSSNCurvilinearID.py); [\[**tutorial**\]](Tutorial-ADM_Initial_Data-Converting_ADMCartesian_to_BSSNCurvilinear.ipynb): Cartesian ADM$\to$Curvilinear BSSN converter function
1. [BSSN_SF/BSSN_ID_function_string.py](BSSN_SF/BSSN_ID_function_string.py): Sets up the C code string enabling initial data be set up in a point-by-point fashion
1. [BSSN_SF/BSSN_Constraints.py](BSSN_SF/BSSNConstraints.py); [\[**tutorial**\]](Tutorial-BSSNConstraints.ipynb): Hamiltonian constraint in BSSN curvilinear basis/coordinates
1. [BSSN_SF/BSSN_RHSs.py](BSSN_SF/BSSN_RHSs.py); [\[**tutorial**\]](Tutorial-BSSNCurvilinear.ipynb): Generates the right-hand sides for the BSSN evolution equations and the evolution of the scalar field in singular, curvilinear coordinates

Here we use NRPy+ to generate the C source code necessary to set up arbitrary initial data, to be input from an array in the C code later. Then we use it to generate the RHS expressions for [Method of Lines](https://reference.wolfram.com/language/tutorial/NDSolveMethodOfLines.html) time integration based on the [explicit Runge-Kutta fourth-order scheme](https://en.wikipedia.org/wiki/Runge%E2%80%93Kutta_methods) (RK4)._____no_output_____

cd ..//mnt/c/Users/Thoma/Desktop/KazeML_ScalarField/nrpytutorial
# First we import needed core NRPy+ modules
from outputC import *
import NRPy_param_funcs as par
import grid as gri
import loop as lp
import indexedexp as ixp
import finite_difference as fin
import reference_metric as rfm

# Set spatial dimension (must be 3 for BSSN)
DIM = 3
par.set_parval_from_str("grid::DIM",DIM)

# Then we set the coordinate system for the numerical grid
par.set_parval_from_str("reference_metric::CoordSystem","Spherical")
rfm.reference_metric() # Create ReU, ReDD needed for rescaling B-L initial data, generating BSSN RHSs, etc.

# Then we set theta and phi to be the symmetry axes; i.e., axis "12", corresponding to the i1 and i2 directions.
#      This sets all spatial derivatives in the phi direction to zero.
par.set_parval_from_str("indexedexp::symmetry_axes","12")_____no_output_____

## Step 2a: First import arbitrary ADM initial data to be input from a file from the BSSN.ID_array_psi NRPy+ module

These initial data are expressed in the Cartesian coordinate basis (i.e., vectors and tensors are expressed in Cartesian components)._____no_output_____

import BSSN_SF.ID_array_psi as idp
idp.ID_array_psi()_____no_output_____

## Step 2b: Next transform the initial data ADM quantities in Cartesian coordinates to BSSN quantities in curvilinear coordinates, using the BSSN.CartesianADMID_to_BSSNCurvilinearID NRPy+ module and output result to file._____no_output_____

import BSSN_SF.CartesianADMID_to_BSSNCurvilinearID as ctob
cf,hDD,lambdaU,aDD,trK,alpha,vetU,betU = \
    ctob.Convert_Cartesian_ADM_to_BSSN_curvilinear(idp.Cartxyz, idp.gammaCartDD,
                                                   idp.KCartDD, idp.alphaCart, idp.betaCartU, idp.BCartU)
# Now output the initial data to file:
with open("BSSN_SF/ID_array_psi.h","w") as file:
    file.write(idp.returnfunction)_____no_output_____

## Step 3A: Output BSSN RHS expressions_____no_output_____

import BSSN_SF.BSSN_RHSs as rhs
# Set the *covariant*, second-order Gamma-driving shift condition
par.set_parval_from_str("BSSN_SF.BSSN_RHSs::ShiftEvolutionOption", "GammaDriving2ndOrder_Covariant", )
rhs.BSSN_RHSs()

import time
print("Generating C code for BSSN RHSs in "+par.parval_from_str("reference_metric::CoordSystem")+" coordinates.")
start = time.time()

BSSN_evol_rhss = [ \
                  lhrh(lhs=gri.gfaccess("rhs_gfs","aDD00"),rhs=rhs.a_rhsDD[0][0]),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","aDD01"),rhs=rhs.a_rhsDD[0][1]),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","aDD02"),rhs=rhs.a_rhsDD[0][2]),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","aDD11"),rhs=rhs.a_rhsDD[1][1]),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","aDD12"),rhs=rhs.a_rhsDD[1][2]),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","aDD22"),rhs=rhs.a_rhsDD[2][2]),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","alpha"),rhs=rhs.alpha_rhs),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","betU0"),rhs=rhs.bet_rhsU[0]),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","betU1"),rhs=rhs.bet_rhsU[1]),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","betU2"),rhs=rhs.bet_rhsU[2]),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","cf"),   rhs=rhs.cf_rhs),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","hDD00"),rhs=rhs.h_rhsDD[0][0]),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","hDD01"),rhs=rhs.h_rhsDD[0][1]),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","hDD02"),rhs=rhs.h_rhsDD[0][2]),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","hDD11"),rhs=rhs.h_rhsDD[1][1]),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","hDD12"),rhs=rhs.h_rhsDD[1][2]),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","hDD22"),rhs=rhs.h_rhsDD[2][2]),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","lambdaU0"),rhs=rhs.lambda_rhsU[0]),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","lambdaU1"),rhs=rhs.lambda_rhsU[1]),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","lambdaU2"),rhs=rhs.lambda_rhsU[2]),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","trK"),  rhs=rhs.trK_rhs),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","vetU0"),rhs=rhs.vet_rhsU[0]),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","vetU1"),rhs=rhs.vet_rhsU[1]),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","vetU2"),rhs=rhs.vet_rhsU[2]),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","vv"),rhs=rhs.vv_rhs),
                  lhrh(lhs=gri.gfaccess("rhs_gfs","uu"),rhs=rhs.uu_rhs)]

BSSN_RHSs_string = fin.FD_outputC("returnstring",BSSN_evol_rhss, params="outCverbose=False",upwindcontrolvec=rhs.betaU)
end = time.time()
print("Finished in "+str(end-start)+" seconds.")

with open("BSSN_SF/BSSN_RHSs.h", "w") as file:
    file.write(lp.loop(["i2","i1","i0"],["NGHOSTS","NGHOSTS","NGHOSTS"],
                       ["NGHOSTS+Nxx[2]","NGHOSTS+Nxx[1]","NGHOSTS+Nxx[0]"],
                       ["1","1","1"],["const REAL invdx0 = 1.0/dxx[0];\n"+
                                      "const REAL invdx1 = 1.0/dxx[1];\n"+
                                      "const REAL invdx2 = 1.0/dxx[2];\n"+
                                      "#pragma omp parallel for",
                                      "    const REAL xx2 = xx[2][i2];",
                                      "        const REAL xx1 = xx[1][i1];"],"",
                                     "const REAL xx0 = xx[0][i0];\n"+BSSN_RHSs_string))Generating C code for BSSN RHSs in Spherical coordinates.
Finished in 76.0874760151 seconds.

## Step 3B: Output the Hamiltonian constraint_____no_output_____

import BSSN_SF.BSSNConstraints as bssncon
bssncon.BSSNConstraints()

H = gri.register_gridfunctions("AUX","H")
import time
print("Generating C code for Hamiltonian constraint")
start = time.time()
Hamiltonianstring = fin.FD_outputC("returnstring",lhrh(lhs=gri.gfaccess("aux_gfs","H"),rhs=bssncon.H),
                                   params="outCverbose=False")
end = time.time()
print("Finished in "+str(end-start)+" seconds.")

with open("BSSN_SF/Hamiltonian.h", "w") as file:
    file.write(lp.loop(["i2","i1","i0"],["NGHOSTS","NGHOSTS","NGHOSTS"],
                       ["NGHOSTS+Nxx[2]","NGHOSTS+Nxx[1]","NGHOSTS+Nxx[0]"],
                       ["1","1","1"],["const REAL invdx0 = 1.0/dxx[0];\n"+
                                      "const REAL invdx1 = 1.0/dxx[1];\n"+
                                      "const REAL invdx2 = 1.0/dxx[2];\n"+
                                      "#pragma omp parallel for",
                                      "    const REAL xx2 = xx[2][i2];",
                                      "        const REAL xx1 = xx[1][i1];"],"",
                                     "const REAL xx0 = xx[0][i0];\n"+Hamiltonianstring))Generating C code for Hamiltonian constraint
Finished in 13.9935438633 seconds.
from BSSN_SF.BSSNCurvilinear_to_CartesianADMID import Convert_BSSN_curvilinear_Cartesian_ADM
import BSSN_SF.BSSN_RHSs as bssnrhs
import BSSN_SF.ADM_ID_function_string as ADIF

Cartxyz = ixp.declarerank1("Cartxyz")

gammaCartDD,KCartDD,alphaCart,betaCartU,BCartU = Convert_BSSN_curvilinear_Cartesian_ADM(Cartxyz,
                                                                                bssnrhs.cf,
                                                                                bssnrhs.hDD,
                                                                                None,
                                                                                bssnrhs.aDD,
                                                                                bssnrhs.trK,
                                                                                bssnrhs.alpha,
                                                                                bssnrhs.vetU,
                                                                                bssnrhs.betU)
returnfunction = ADIF.ADM_ID_function_string(gammaCartDD,KCartDD,alphaCart,betaCartU,BCartU)
with open("BSSN_SF/ID_array_ADM.h","w") as file:
    file.write(returnfunction)    _____no_output_____

## Step 4: Apply singular, curvilinear coordinate boundary conditions [*a la* Section IV.E of the SENR/NRPy+ paper](https://arxiv.org/abs/1712.07658)

The procedure is the same as in the [Start-to-Finish Example: Head-On Black Hole Collision](../../Unused_tutorials/Tutorial-Start_to_Finish-BSSNCurvilinear-Two_BHs_Collide.ipynb):
##### Step 4A: Find locations to where outer ghost zone gridpoints map
##### Step 4B: At each ghost zone gridpoint, find and store the correct parity condition type for each gridfunction up to rank 2 (the highest rank in the BSSN RHSs)_____no_output_____

# Generic coordinate NRPy+ file output, Part 1: output the coordinate bounds xxmin[] and xxmax[]:
with open("BSSN_SF/xxminmax.h", "w") as file:
    file.write("const REAL xxmin[3] = {"+str(rfm.xxmin[0])+","+str(rfm.xxmin[1])+","+str(rfm.xxmin[2])+"};\n")
    file.write("const REAL xxmax[3] = {"+str(rfm.xxmax[0])+","+str(rfm.xxmax[1])+","+str(rfm.xxmax[2])+"};\n")
# Generic coordinate NRPy+ file output, Part 2: output the conversion from (x0,x1,x2) to Cartesian (x,y,z)
outputC([rfm.xxCart[0],rfm.xxCart[1],rfm.xxCart[2]],["xCart[0]","xCart[1]","xCart[2]"],
        "BSSN_SF/xxCart.h")
# Generic coordinate NRPy+ file output, Part 3: output the conversion from Cartesian (x,y,z) to interior/OB (x0,x1,x2)
outputC([rfm.Cart_to_xx[0],rfm.Cart_to_xx[1],rfm.Cart_to_xx[2]],
        ["Cart_to_xx0_inbounds","Cart_to_xx1_inbounds","Cart_to_xx2_inbounds"],
        "BSSN_SF/Cart_to_xx.h")
# Generic coordinate NRPy+ file output, Part 4: output the proper distance between gridpoints in given coordinate system
dxx     = ixp.declarerank1("dxx",DIM=3)
ds_dirn = rfm.ds_dirn(dxx)
outputC([ds_dirn[0],ds_dirn[1],ds_dirn[2]],["ds_dirn0","ds_dirn1","ds_dirn2"],"BSSN_SF/ds_dirn.h")Wrote to file "BSSN_SF/xxCart.h"
Wrote to file "BSSN_SF/Cart_to_xx.h"
Wrote to file "BSSN_SF/ds_dirn.h"

####   Step 4B.i: Set the parity condition for each of the 10 parity condition types

First we fill in the parity condition arrays. These take as input $(x_0,x_1,x_2)_{\rm in}$ and $(x_0,x_1,x_2)_{\rm IB}$, and output the necessary dot products for each parity type. To wit, there are 10 parity types for BSSN evolved variables, which include tensors up to and including rank-2:

0. Scalar (Rank-0 tensor)
1. Rank-1 tensor in **i0** direction
1. Rank-1 tensor in **i1** direction
1. Rank-1 tensor in **i2** direction
1. Rank-2 tensor in **i0-i0** direction
1. Rank-2 tensor in **i0-i1** direction
1. Rank-2 tensor in **i0-i2** direction
1. Rank-2 tensor in **i1-i1** direction
1. Rank-2 tensor in **i1-i2** direction
1. Rank-2 tensor in **i2-i2** direction_____no_output_____

parity = ixp.zerorank1(DIM=10)
UnitVectors_inner = ixp.zerorank2()
xx0_inbounds,xx1_inbounds,xx2_inbounds = sp.symbols("xx0_inbounds xx1_inbounds xx2_inbounds", real=True)
for i in range(3):
    for j in range(3):
        UnitVectors_inner[i][j] = rfm.UnitVectors[i][j].subs(rfm.xx[0],xx0_inbounds).subs(rfm.xx[1],xx1_inbounds).subs(rfm.xx[2],xx2_inbounds)
# Type 0: scalar
parity[0] = sp.sympify(1)
# Type 1: i0-direction vector or one-form
# Type 2: i1-direction vector or one-form
# Type 3: i2-direction vector or one-form
for i in range(3):
    for Type in range(1,4):
        parity[Type] += rfm.UnitVectors[Type-1][i]*UnitVectors_inner[Type-1][i]
# Type 4: i0i0-direction rank-2 tensor
# parity[4] = parity[1]*parity[1]
# Type 5: i0i1-direction rank-2 tensor
# Type 6: i0i2-direction rank-2 tensor
# Type 7: i1i1-direction rank-2 tensor
# Type 8: i1i2-direction rank-2 tensor
# Type 9: i2i2-direction rank-2 tensor
count = 4
for i in range(3):
    for j in range(i,3):
        parity[count] = parity[i+1]*parity[j+1]
        count = count + 1

lhs_strings = []
for i in range(10):
    lhs_strings.append("parity["+str(i)+"]")
outputC(parity,lhs_strings, "BSSN_SF/set_parity_conditions.h")Wrote to file "BSSN_SF/set_parity_conditions.h"

####   Step 4B.ii: Implement the boundary condition ghost zone mapping routine, difining the parity conditions_____no_output_____

%%writefile BSSN_SF/curvilinear_parity_and_outer_boundary_conditions.h

// First we define the struct that will be used to store the 10 parity conditions at all gridpoints:
// We store the 10 parity conditions in a struct consisting of 10 integers, one for each condition.
// Note that these conditions can only take one of two values: +1 or -1.
typedef struct parity_conditions {
  int8_t parity[10];
} parity_condition;

typedef struct ghostzone_map {
  short i0,i1,i2;
} gz_map;

void set_bc_parity_conditions(REAL parity[10], const REAL xx0,const REAL xx1,const REAL xx2,
                              const REAL xx0_inbounds,const REAL xx1_inbounds,const REAL xx2_inbounds) {
    #include "set_parity_conditions.h"
}

void set_up_bc_gz_map_and_parity_conditions(const int Nxx_plus_2NGHOSTS[3], REAL *xx[3],
                                            const REAL dxx[3], const REAL xxmin[3], const REAL xxmax[3],
                                            gz_map *bc_gz_map, parity_condition *bc_parity_conditions) {
  LOOP_REGION(0,Nxx_plus_2NGHOSTS[0],0,Nxx_plus_2NGHOSTS[1],0,Nxx_plus_2NGHOSTS[2]) {
    REAL xCart[3];
    xxCart(xx, i0,i1,i2, xCart);
    REAL Cartx = xCart[0];
    REAL Carty = xCart[1];
    REAL Cartz = xCart[2];

    REAL Cart_to_xx0_inbounds,Cart_to_xx1_inbounds,Cart_to_xx2_inbounds;
#include "Cart_to_xx.h"
    int i0_inbounds = (int)( (Cart_to_xx0_inbounds - xxmin[0] - (1.0/2.0)*dxx[0] + ((REAL)NGHOSTS)*dxx[0])/dxx[0] + 0.5 );
    int i1_inbounds = (int)( (Cart_to_xx1_inbounds - xxmin[1] - (1.0/2.0)*dxx[1] + ((REAL)NGHOSTS)*dxx[1])/dxx[1] + 0.5 );
    int i2_inbounds = (int)( (Cart_to_xx2_inbounds - xxmin[2] - (1.0/2.0)*dxx[2] + ((REAL)NGHOSTS)*dxx[2])/dxx[2] + 0.5 );

    REAL xCart_orig[3]; for(int ii=0;ii<3;ii++) xCart_orig[ii] = xCart[ii];
    xxCart(xx, i0_inbounds,i1_inbounds,i2_inbounds, xCart);

#define EPS_ABS 1e-8
    if(fabs( (double)(xCart_orig[0] - xCart[0]) ) > EPS_ABS ||
       fabs( (double)(xCart_orig[1] - xCart[1]) ) > EPS_ABS ||
       fabs( (double)(xCart_orig[2] - xCart[2]) ) > EPS_ABS) {
      printf("Error. Cartesian disagreement: ( %.15e %.15e %.15e ) != ( %.15e %.15e %.15e )\n",
             (double)xCart_orig[0],(double)xCart_orig[1],(double)xCart_orig[2],
             (double)xCart[0],(double)xCart[1],(double)xCart[2]);
      exit(1);
    }

    if(i0_inbounds-i0 == 0 && i1_inbounds-i1 == 0 && i2_inbounds-i2 == 0) {
      bc_gz_map[IDX3(i0,i1,i2)].i0=-1;
      bc_gz_map[IDX3(i0,i1,i2)].i1=-1;
      bc_gz_map[IDX3(i0,i1,i2)].i2=-1;
      for(int which_parity=0; which_parity<10; which_parity++) {
        bc_parity_conditions[IDX3(i0,i1,i2)].parity[which_parity] = 1;
      }
    } else {
      bc_gz_map[IDX3(i0,i1,i2)].i0=i0_inbounds;
      bc_gz_map[IDX3(i0,i1,i2)].i1=i1_inbounds;
      bc_gz_map[IDX3(i0,i1,i2)].i2=i2_inbounds;
      const REAL xx0 = xx[0][i0];
      const REAL xx1 = xx[1][i1];
      const REAL xx2 = xx[2][i2];
      const REAL xx0_inbounds = xx[0][i0_inbounds];
      const REAL xx1_inbounds = xx[1][i1_inbounds];
      const REAL xx2_inbounds = xx[2][i2_inbounds];
      REAL REAL_parity_array[10];
      set_bc_parity_conditions(REAL_parity_array,  xx0,xx1,xx2, xx0_inbounds,xx1_inbounds,xx2_inbounds);
      for(int whichparity=0;whichparity<10;whichparity++) {
          //printf("Good? Parity %d evaluated to %e\n",whichparity,REAL_parity_array[whichparity]);
          // Perform sanity check on parity array output: should be +1 or -1 to within 8 significant digits:
          if( (REAL_parity_array[whichparity]  > 0 && fabs(REAL_parity_array[whichparity] - (+1)) > 1e-8) ||
              (REAL_parity_array[whichparity] <= 0 && fabs(REAL_parity_array[whichparity] - (-1)) > 1e-8) ) {
              printf("Error. Parity evaluated to %e , which is not within 8 significant digits of +1 or -1.",REAL_parity_array[whichparity]);
              exit(1);
          }
          if(REAL_parity_array[whichparity] < 0.0) bc_parity_conditions[IDX3(i0,i1,i2)].parity[whichparity] = -1;
          if(REAL_parity_array[whichparity] > 0.0) bc_parity_conditions[IDX3(i0,i1,i2)].parity[whichparity] = +1;
      }
    }
  }
}

// Part P6: Declare boundary condition OB_UPDATE macro,
//          which updates a single face of the 3D grid cube
//          using quadratic polynomial extrapolation.
const int MAXFACE = -1;
const int NUL     = +0;
const int MINFACE = +1;

#define OB_UPDATE(inner,which_gf, bc_gz_map,bc_parity_conditions, i0min,i0max, i1min,i1max, i2min,i2max, FACEX0,FACEX1,FACEX2) \
  LOOP_REGION(i0min,i0max, i1min,i1max, i2min,i2max) {                  \
    const int idx3 = IDX3(i0,i1,i2);                                    \
    if(bc_gz_map[idx3].i0 == -1 && inner==0) {                          \
      gfs[IDX4(which_gf,i0,i1,i2)] =                                    \
        +3.0*gfs[IDX4(which_gf,i0+1*FACEX0,i1+1*FACEX1,i2+1*FACEX2)]    \
        -3.0*gfs[IDX4(which_gf,i0+2*FACEX0,i1+2*FACEX1,i2+2*FACEX2)]    \
        +1.0*gfs[IDX4(which_gf,i0+3*FACEX0,i1+3*FACEX1,i2+3*FACEX2)];   \
    } else if(bc_gz_map[idx3].i0 != -1 && inner==1) {                   \
     gfs[IDX4(which_gf,i0,i1,i2)] =                                    \
        ( (REAL)bc_parity_conditions[idx3].parity[evol_gf_parity[which_gf]] )* \
                                             gfs[IDX4(which_gf,           \
                                                    bc_gz_map[idx3].i0, \
                                                    bc_gz_map[idx3].i1, \
                                                    bc_gz_map[idx3].i2)]; \
    }                                                                   \
  }

// Part P7: Boundary condition driver routine: Apply BCs to all six
//          boundary faces of the cube, filling in the innermost
//          ghost zone first, and moving outward.
void apply_bcs(const int Nxx[3],const int Nxx_plus_2NGHOSTS[3],
               gz_map *bc_gz_map,parity_condition *bc_parity_conditions,REAL *gfs) {
#pragma omp parallel for
  for(int which_gf=0;which_gf<NUM_EVOL_GFS;which_gf++) {
    int imin[3] = { NGHOSTS, NGHOSTS, NGHOSTS };
    int imax[3] = { Nxx_plus_2NGHOSTS[0]-NGHOSTS, Nxx_plus_2NGHOSTS[1]-NGHOSTS, Nxx_plus_2NGHOSTS[2]-NGHOSTS };
    for(int which_gz = 0; which_gz < NGHOSTS; which_gz++) {
      for(int inner=0;inner<2;inner++) {
        // After updating each face, adjust imin[] and imax[]
        //   to reflect the newly-updated face extents.
        OB_UPDATE(inner,which_gf, bc_gz_map,bc_parity_conditions, imin[0]-1,imin[0], imin[1],imax[1], imin[2],imax[2], MINFACE,NUL,NUL); imin[0]--;
        OB_UPDATE(inner,which_gf, bc_gz_map,bc_parity_conditions, imax[0],imax[0]+1, imin[1],imax[1], imin[2],imax[2], MAXFACE,NUL,NUL); imax[0]++;

        OB_UPDATE(inner,which_gf, bc_gz_map,bc_parity_conditions, imin[0],imax[0], imin[1]-1,imin[1], imin[2],imax[2], NUL,MINFACE,NUL); imin[1]--;
        OB_UPDATE(inner,which_gf, bc_gz_map,bc_parity_conditions, imin[0],imax[0], imax[1],imax[1]+1, imin[2],imax[2], NUL,MAXFACE,NUL); imax[1]++;

        OB_UPDATE(inner,which_gf, bc_gz_map,bc_parity_conditions, imin[0],imax[0], imin[1],imax[1], imin[2]-1,imin[2], NUL,NUL,MINFACE); imin[2]--;
        OB_UPDATE(inner,which_gf, bc_gz_map,bc_parity_conditions, imin[0],imax[0], imin[1],imax[1], imax[2],imax[2]+1, NUL,NUL,MAXFACE); imax[2]++;
        if(inner==0) { for(int ii=0;ii<3;ii++) {imin[ii]++; imax[ii]--;} }
      }
    }
  }
}Overwriting BSSN_SF/curvilinear_parity_and_outer_boundary_conditions.h

####   Step 4B.iii: Set the #define list for all gridfunctions_____no_output_____

# First we set up the evolved and auxiliary variables lists
evolved_variables_list   = []
auxiliary_variables_list = []
for i in range(len(gri.glb_gridfcs_list)):
    if gri.glb_gridfcs_list[i].gftype == "EVOL":
        evolved_variables_list.append(gri.glb_gridfcs_list[i].name)
    if gri.glb_gridfcs_list[i].gftype == "AUX":
        auxiliary_variables_list.append(gri.glb_gridfcs_list[i].name)

# Next we alphabetize the lists
evolved_variables_list.sort()
auxiliary_variables_list.sort()

# Finally we set up the #define statements:
with open("BSSN_SF/gridfunction_defines.h", "w") as file:
    file.write("/* This file is automatically generated by NRPy+. Do not edit. */\n\n")
    file.write("/* EVOLVED VARIABLES: */\n")
    file.write("#define NUM_EVOL_GFS "+str(len(evolved_variables_list))+"\n")
    for i in range(len(evolved_variables_list)):
        file.write("#define "+evolved_variables_list[i].upper()+"GF\t"+str(i)+"\n")
    file.write("\n\n /* AUXILIARY VARIABLES: */\n")
    file.write("#define NUM_AUX_GFS "+str(len(auxiliary_variables_list))+"\n")
    for i in range(len(auxiliary_variables_list)):
        file.write("#define "+auxiliary_variables_list[i].upper()+"GF\t"+str(i)+"\n")_____no_output_____

####   Step 4B.iv: Assign the correct parity condition for each BSSN gridfunction_____no_output_____

# Next we set the parity conditions on all evolved gridfunctions, based on how many digits are at the end of their names
parity_type = []
for i in range(len(evolved_variables_list)):
    varname = evolved_variables_list[i]
    parity_type__orig_len = len(parity_type)
    if  len(varname)>2:
        if   varname[len(varname)-2] == "0" and varname[len(varname)-1] == "0":
            parity_type.append(4)
        elif varname[len(varname)-2] == "0" and varname[len(varname)-1] == "1":
            parity_type.append(5)
        elif varname[len(varname)-2] == "0" and varname[len(varname)-1] == "2":
            parity_type.append(6)
        elif varname[len(varname)-2] == "1" and varname[len(varname)-1] == "1":
            parity_type.append(7)
        elif varname[len(varname)-2] == "1" and varname[len(varname)-1] == "2":
            parity_type.append(8)
        elif varname[len(varname)-2] == "2" and varname[len(varname)-1] == "2":
            parity_type.append(9)
    if len(varname)>1 and len(parity_type) == parity_type__orig_len:
        if   varname[len(varname)-1] == "0":
            parity_type.append(1)
        elif varname[len(varname)-1] == "1":
            parity_type.append(2)
        elif varname[len(varname)-1] == "2":
            parity_type.append(3)
    if varname[len(varname)-1].isdigit() == False:
        parity_type.append(0)

    if len(parity_type) == parity_type__orig_len:
        print("Error: Could not figure out parity type for evolved variable: "+varname)
        exit(1)

with open("BSSN_SF/gridfunction_defines.h", "a") as file:
    file.write("\n\n/* PARITY TYPES FOR EVOLVED VARIABLES.\n")
    file.write("   SEE \"Tutorial-Start_to_Finish-BSSNCurvilinear-Two_BHs_Collide.ipynb\" FOR DEFINITIONS. */\n")
    file.write("const int8_t evol_gf_parity["+str(len(evolved_variables_list))+"] = { ")
    for i in range(len(evolved_variables_list)-1):
        file.write(str(parity_type[i])+", ")
    file.write(str(parity_type[len(evolved_variables_list)-1])+" };\n")

for i in range(len(evolved_variables_list)):
    print("Evolved gridfunction \""+evolved_variables_list[i]+"\" has parity type "+str(parity_type[i])+".")Evolved gridfunction "Rtrace" has parity type 0.
Evolved gridfunction "aDD00" has parity type 4.
Evolved gridfunction "aDD01" has parity type 5.
Evolved gridfunction "aDD02" has parity type 6.
Evolved gridfunction "aDD11" has parity type 7.
Evolved gridfunction "aDD12" has parity type 8.
Evolved gridfunction "aDD22" has parity type 9.
Evolved gridfunction "alpha" has parity type 0.
Evolved gridfunction "betU0" has parity type 1.
Evolved gridfunction "betU1" has parity type 2.
Evolved gridfunction "betU2" has parity type 3.
Evolved gridfunction "cf" has parity type 0.
Evolved gridfunction "hDD00" has parity type 4.
Evolved gridfunction "hDD01" has parity type 5.
Evolved gridfunction "hDD02" has parity type 6.
Evolved gridfunction "hDD11" has parity type 7.
Evolved gridfunction "hDD12" has parity type 8.
Evolved gridfunction "hDD22" has parity type 9.
Evolved gridfunction "lambdaU0" has parity type 1.
Evolved gridfunction "lambdaU1" has parity type 2.
Evolved gridfunction "lambdaU2" has parity type 3.
Evolved gridfunction "trK" has parity type 0.
Evolved gridfunction "uu" has parity type 0.
Evolved gridfunction "vetU0" has parity type 1.
Evolved gridfunction "vetU1" has parity type 2.
Evolved gridfunction "vetU2" has parity type 3.
Evolved gridfunction "vv" has parity type 0.

## Step 5: Enforce conformal 3-metric $\det{\bar{\gamma}_{ij}}=\det{\hat{\gamma}_{ij}}$ constraint (Eq. 53 of [Ruchlin, Etienne, and Baumgarte (2018)](https://arxiv.org/abs/1712.07658))

Again as in [Start-to-Finish Example: Head-On Black Hole Collision](../../Unused_tutorials/Tutorial-Start_to_Finish-BSSNCurvilinear-Two_BHs_Collide.ipynb)._____no_output_____

# First define the Kronecker delta:
KroneckerDeltaDD = ixp.zerorank2()
for i in range(DIM):
    KroneckerDeltaDD[i][i] = sp.sympify(1)

# The detgammabar in BSSN_RHSs is set to detgammahat when BSSN_RHSs::detgbarOverdetghat_equals_one=True (default),
#    so we manually compute it here:
dummygammabarUU, detgammabar = ixp.symm_matrix_inverter3x3(rhs.gammabarDD)

# Next apply the constraint enforcement equation above.
hprimeDD = ixp.zerorank2()
for i in range(DIM):
    for j in range(DIM):
        hprimeDD[i][j] = \
        (sp.Abs(rfm.detgammahat)/detgammabar)**(sp.Rational(1,3)) * (KroneckerDeltaDD[i][j] + rhs.hDD[i][j]) \
        - KroneckerDeltaDD[i][j]

enforce_detg_constraint_vars = [ \
                                lhrh(lhs=gri.gfaccess("in_gfs","hDD00"),rhs=hprimeDD[0][0]),
                                lhrh(lhs=gri.gfaccess("in_gfs","hDD01"),rhs=hprimeDD[0][1]),
                                lhrh(lhs=gri.gfaccess("in_gfs","hDD02"),rhs=hprimeDD[0][2]),
                                lhrh(lhs=gri.gfaccess("in_gfs","hDD11"),rhs=hprimeDD[1][1]),
                                lhrh(lhs=gri.gfaccess("in_gfs","hDD12"),rhs=hprimeDD[1][2]),
                                lhrh(lhs=gri.gfaccess("in_gfs","hDD22"),rhs=hprimeDD[2][2]) ]

enforce_gammadet_string = fin.FD_outputC("returnstring",enforce_detg_constraint_vars,
                                         params="outCverbose=False,preindent=0,includebraces=False")

with open("BSSN_SF/enforce_detgammabar_constraint.h", "w") as file:
    indent = "   "
    file.write("void enforce_detgammabar_constraint(const int Nxx_plus_2NGHOSTS[3],REAL *xx[3], REAL *in_gfs) {\n\n")
    file.write(lp.loop(["i2","i1","i0"],["0","0","0"],
                       ["Nxx_plus_2NGHOSTS[2]","Nxx_plus_2NGHOSTS[1]","Nxx_plus_2NGHOSTS[0]"],
                       ["1","1","1"],["#pragma omp parallel for",
                                      "    const REAL xx2 = xx[2][i2];",
                                      "        const REAL xx1 = xx[1][i1];"],"",
                                     "const REAL xx0 = xx[0][i0];\n"+enforce_gammadet_string))
    file.write("}\n")_____no_output_____

# BSSNScalar_Field.c: The Main C Code_____no_output_____

!rm BSSN_SF/BSSNScalar_Field.c
!mkdir BSSN_SF-evolution
!mkdir BSSN_SF-output2Dmkdir: cannot create directory ‘BSSN_SF-evolution’: File exists
mkdir: cannot create directory ‘BSSN_SF-output2D’: File exists
# Part P0: Set the number of ghost cells, from NRPy+'s FD_CENTDERIVS_ORDER
with open("BSSN_SF/BSSNScalar_Field.c", "w") as file:
    file.write("// Part P0: Set the number of ghost cells, from NRPy+'s FD_CENTDERIVS_ORDER\n")
    # Upwinding in BSSN requires that NGHOSTS = FD_CENTDERIVS_ORDER/2 + 1 <- Notice the +1.
    file.write("#define NGHOSTS "+str(int(par.parval_from_str("finite_difference::FD_CENTDERIVS_ORDER")/2)+1)+"\n")_____no_output_____%%writefile -a BSSN_SF/BSSNScalar_Field.c

// Step P1a: Import needed header files
#include "stdio.h"
#include "stdlib.h"
#include <stdint.h>
#include "math.h"
#include "time.h"
#ifndef M_PI
    #define M_PI 3.14159265358979323846
#endif

// Step P1b: Import necessary gsl libraries for interpolating the initial data onto the grid

#include "gsl/gsl_spline.h"
#include "gsl/gsl_errno.h"
#include "gsl/gsl_interp.h"

// Step P2: Add needed #define's to set data type, the IDX4() macro, and the gridfunctions
// Step P2a: set REAL=double, so that all floating point numbers are stored to at least ~16 significant digits.
#define REAL double

// Step P3: Set free parameters
// Step P3a: Free parameters for the numerical grid

// Spherical coordinates parameter
const REAL RMAX    = 30.; /* Set to approximately the time you wish to evolve for,
                            * so that at t=t_final data at the origin is not
                            * affected by the boundary conditions */
// Time coordinate parameters
const REAL t_final =  50.;
const REAL CFL_FACTOR = 0.5; // Set the CFL Factor

// Step P3b: Free parameters for the spacetime evolution
const REAL eta = 2.; // Gamma-driving shift condition parameter.

// Step P4: Implement the algorithm for upwinding.
//          *NOTE*: This upwinding is backwards from
//          usual upwinding algorithms, because the
//          upwinding control vector in BSSN (the shift)
//          acts like a *negative* velocity.
#define UPWIND_ALG(UpwindVecU) UpwindVecU > 0.0 ? 1.0 : 0.0

// Step P5: Set free parameters for Psi initial data
const REAL psi_posn_x = 0.0,psi_posn_y = 0.0,psi_posn_z = 0.0;

// Step P5b: Set free parameters for the scalar field
const REAL scalar_posn_x = 0.0;
const REAL scalar_posn_y = 0.0;
const REAL scalar_posn_z = 0.0;
const REAL br_on = 1.; // Turn on(1.)/off(0.) scalar field backreaction on the metric
const REAL pot1_on = 0.; // Turn on(1.)/off(0.) quadratic potential
const REAL pot2_on = 0.; // Turn on(1.)/off(0.) self-interactiong potential
// Make sure only one potential is on at a time
// Variables for the scalar field potential
const REAL scalarmass = 1.; // Scalar mass, \mu = c/\hbar m
const REAL fa = 0.05; // Decay constant, only relevant for the self-interacting potential

//Step P5c: Declare vars for initial data arrays
//          We use initial data profiles for the scalar
//          and the conformal factor that is known to
//          lead to stable scalar field evolution
REAL uu_in;
REAL vv_in;
REAL psi_in;
REAL alpha_in;
REAL r_scalar;
REAL r_psi;

// Step P6: Declare the IDX4(gf,i,j,k) macro, which enables us to store 4-dimensions of
//          data in a 1D array. In this case, consecutive values of "i"
//          (all other indices held to a fixed value) are consecutive in memory, where
//          consecutive values of "j" (fixing all other indices) are separated by
//          Nxx_plus_2NGHOSTS[0] elements in memory. Similarly, consecutive values of
//          "k" are separated by Nxx_plus_2NGHOSTS[0]*Nxx_plus_2NGHOSTS[1] in memory, etc.
#define IDX4(g,i,j,k) \
( (i) + Nxx_plus_2NGHOSTS[0] * ( (j) + Nxx_plus_2NGHOSTS[1] * ( (k) + Nxx_plus_2NGHOSTS[2] * (g) ) ) )
#define IDX3(i,j,k) ( (i) + Nxx_plus_2NGHOSTS[0] * ( (j) + Nxx_plus_2NGHOSTS[1] * (k) ) )
// Assuming idx = IDX3(i,j,k). Much faster if idx can be reused over and over:
#define IDX4pt(g,idx)   ( (idx) + (Nxx_plus_2NGHOSTS[0]*Nxx_plus_2NGHOSTS[1]*Nxx_plus_2NGHOSTS[2]) * (g) )

// Step P7: Set #define's for BSSN gridfunctions. C code generated above
#include "gridfunction_defines.h"

#define LOOP_REGION(i0min,i0max, i1min,i1max, i2min,i2max) \
  for(int i2=i2min;i2<i2max;i2++) for(int i1=i1min;i1<i1max;i1++) for(int i0=i0min;i0<i0max;i0++)

void xxCart(REAL *xx[3],const int i0,const int i1,const int i2, REAL xCart[3]) {
    REAL xx0 = xx[0][i0];
    REAL xx1 = xx[1][i1];
    REAL xx2 = xx[2][i2];
#include "xxCart.h"
}

// Step P8: Include basic functions needed to impose curvilinear
//          parity and boundary conditions.
#include "curvilinear_parity_and_outer_boundary_conditions.h"

#include "enforce_detgammabar_constraint.h"

// Step P9: Find the CFL-constrained timestep
REAL find_timestep(const int Nxx_plus_2NGHOSTS[3],const REAL dxx[3],REAL *xx[3], const REAL CFL_FACTOR) {
  const REAL dxx0 = dxx[0], dxx1 = dxx[1], dxx2 = dxx[2];
  REAL dsmin = 1e38; // Start with a crazy high value... close to the largest number in single precision.
  LOOP_REGION(NGHOSTS,Nxx_plus_2NGHOSTS[0]-NGHOSTS, NGHOSTS,Nxx_plus_2NGHOSTS[1]-NGHOSTS, NGHOSTS,Nxx_plus_2NGHOSTS[2]-NGHOSTS) {
    const REAL xx0 = xx[0][i0], xx1 = xx[1][i1], xx2 = xx[2][i2];
    REAL ds_dirn0, ds_dirn1, ds_dirn2;
#include "ds_dirn.h"
#define MIN(A, B) ( ((A) < (B)) ? (A) : (B) )
    // Set dsmin = MIN(dsmin, ds_dirn0, ds_dirn1, ds_dirn2);
    dsmin = MIN(dsmin,MIN(ds_dirn0,MIN(ds_dirn1,ds_dirn2)));
  }
  return dsmin*CFL_FACTOR;
}

// Contains BSSN_ID() for arbitrary initial data array
#include "ID_array_psi.h"

// Step P10: Declare the function for the exact solution. time==0 corresponds to the initial data.
void initial_data(const int Nxx_plus_2NGHOSTS[3],REAL *xx[3], REAL *in_gfs) {

// Step P11a: Declare initial data arrays
FILE *uu_file = fopen("BSSN_SF/InitialData/phiCC9.csv", "r");
FILE *vv_file = fopen("BSSN_SF/InitialData/PiCC9.csv", "r");
FILE *psi_file = fopen("BSSN_SF/InitialData/psiCC9.csv", "r");
FILE *alpha_file = fopen("BSSN_SF/InitialData/alphaCC9.csv", "r");

int temp;
int alen = 0;
while(fscanf(uu_file,"%lf\n",&temp)==1){
    alen++;
}
double r_arr[alen];
double uu_in_arr[alen];
double vv_in_arr[alen];
double psi_in_arr[alen];
double alpha_in_arr[alen];

rewind(uu_file);
for(int i=0;i<alen;i++){
    r_arr[i] = 0.01*i;
    fscanf(uu_file, "%lf\n", &uu_in_arr[i]);
    fscanf(vv_file, "%lf\n", &vv_in_arr[i]);
    fscanf(psi_file, "%lf\n", &psi_in_arr[i]);
    fscanf(alpha_file, "%lf\n", &alpha_in_arr[i]);
}

// Step P11b: Declare splines to interpolate onto the cartesian grid

gsl_interp_accel *acc = gsl_interp_accel_alloc ();

gsl_spline *spline_u = gsl_spline_alloc (gsl_interp_cspline, alen);
gsl_spline_init(spline_u, r_arr, uu_in_arr, alen);

gsl_spline *spline_v = gsl_spline_alloc (gsl_interp_cspline, alen);
gsl_spline_init(spline_v, r_arr, vv_in_arr, alen);

gsl_spline *spline_psi = gsl_spline_alloc (gsl_interp_cspline, alen);
gsl_spline_init(spline_psi, r_arr, psi_in_arr, alen);

gsl_spline *spline_alpha = gsl_spline_alloc (gsl_interp_cspline, alen);
gsl_spline_init(spline_alpha, r_arr, alpha_in_arr, alen);

#pragma omp parallel for
  LOOP_REGION(0,Nxx_plus_2NGHOSTS[0], 0,Nxx_plus_2NGHOSTS[1], 0,Nxx_plus_2NGHOSTS[2]) {
    const int idx = IDX3(i0,i1,i2);
    REAL xCart[3];
    xxCart(xx, i0,i1,i2, xCart);
    {
        r_psi = sqrt(pow(-psi_posn_x + xCart[0], 2) + pow(-psi_posn_y + xCart[1], 2) + pow(-psi_posn_z + xCart[2], 2));
        psi_in = gsl_spline_eval (spline_psi, r_psi, acc);
        alpha_in = gsl_spline_eval (spline_alpha, r_psi, acc);
    }
    BSSN_ID(xx[0][i0],xx[1][i1],xx[2][i2],xCart[0],xCart[1],xCart[2],
            &in_gfs[IDX4pt(HDD00GF,idx)],&in_gfs[IDX4pt(HDD01GF,idx)],&in_gfs[IDX4pt(HDD02GF,idx)],
            &in_gfs[IDX4pt(HDD11GF,idx)],&in_gfs[IDX4pt(HDD12GF,idx)],&in_gfs[IDX4pt(HDD22GF,idx)],
            &in_gfs[IDX4pt(TRKGF,idx)],
            &in_gfs[IDX4pt(ADD00GF,idx)],&in_gfs[IDX4pt(ADD01GF,idx)],&in_gfs[IDX4pt(ADD02GF,idx)],
            &in_gfs[IDX4pt(ADD11GF,idx)],&in_gfs[IDX4pt(ADD12GF,idx)],&in_gfs[IDX4pt(ADD22GF,idx)],
            &in_gfs[IDX4pt(LAMBDAU0GF,idx)],&in_gfs[IDX4pt(LAMBDAU1GF,idx)],&in_gfs[IDX4pt(LAMBDAU2GF,idx)],
            &in_gfs[IDX4pt(VETU0GF,idx)],&in_gfs[IDX4pt(VETU1GF,idx)],&in_gfs[IDX4pt(VETU2GF,idx)],
            &in_gfs[IDX4pt(BETU0GF,idx)],&in_gfs[IDX4pt(BETU1GF,idx)],&in_gfs[IDX4pt(BETU2GF,idx)],
            &in_gfs[IDX4pt(ALPHAGF,idx)],&in_gfs[IDX4pt(CFGF,idx)]);
    REAL xx0 = xCart[0];
    REAL xx1 = xCart[1];
    REAL xx2 = xCart[2];
    {
        r_scalar = sqrt(pow(-scalar_posn_x + xx0, 2) + pow(-scalar_posn_y + xx1, 2) + pow(-scalar_posn_z + xx2, 2));
        in_gfs[IDX4(UUGF, i0, i1, i2)] = gsl_spline_eval (spline_u, r_scalar, acc);
        in_gfs[IDX4(VVGF, i0, i1, i2)] = gsl_spline_eval (spline_v, r_scalar, acc);
    }
  }
}

// Step P12: Implement Hamiltonian constraint diagnostic
void Hamiltonian_constraint(const int Nxx[3],const int Nxx_plus_2NGHOSTS[3],const REAL dxx[3], REAL *xx[3],
                            REAL *in_gfs, REAL *aux_gfs) {
#include "Hamiltonian.h"
}

// Step P13: Declare the function to evaluate the BSSN RHSs
void rhs_eval(const int Nxx[3],const int Nxx_plus_2NGHOSTS[3],const REAL dxx[3], REAL *xx[3], const REAL *in_gfs,REAL *rhs_gfs) {
#include "BSSN_RHSs.h"
}

#include "ID_array_ADM.h"

// main() function:
// Step 0: Read command-line input, set up grid structure, allocate memory for gridfunctions, set up coordinates
// Step 1: Set up scalar wave initial data
// Step 2: Evolve scalar wave initial data forward in time using Method of Lines with RK4 algorithm,
//         applying quadratic extrapolation outer boundary conditions.
// Step 3: Output relative error between numerical and exact solution.
// Step 4: Free all allocated memory
int main(int argc, const char *argv[]) {

  // Step 0a: Read command-line input, error out if nonconformant
  if(argc != 4 || atoi(argv[1]) < NGHOSTS) {
      printf("Error: Expected one command-line argument: ./BSSNCurvilinear_Playground Nx0 Nx1 Nx2,\n");
      printf("where Nx[0,1,2] is the number of grid points in the 0, 1, and 2 directions.\n");
      printf("Nx[] MUST BE larger than NGHOSTS (= %d)\n",NGHOSTS);
      exit(1);
  }
  // Step 0b: Set up numerical grid structure, first in space...
  const int Nx0 = atoi(argv[1]);
  const int Nx1 = atoi(argv[2]);
  const int Nx2 = atoi(argv[3]);
  if(Nx0%2 != 0 || Nx1%2 != 0 || Nx2%2 != 0) {
    printf("Error: Cannot guarantee a proper cell-centered grid if number of grid cells not set to even number.\n");
    printf("       For example, in case of angular directions, proper symmetry zones will not exist.\n");
    exit(1);
  }
  const int Nxx[3] = { Nx0, Nx1, Nx2 };
  const int Nxx_plus_2NGHOSTS[3] = { Nxx[0]+2*NGHOSTS, Nxx[1]+2*NGHOSTS, Nxx[2]+2*NGHOSTS };
  const int Nxx_plus_2NGHOSTS_tot = Nxx_plus_2NGHOSTS[0]*Nxx_plus_2NGHOSTS[1]*Nxx_plus_2NGHOSTS[2];
#include "xxminmax.h"

  // Step 0c: Allocate memory for gridfunctions
  REAL *evol_gfs = (REAL *)malloc(sizeof(REAL) * NUM_EVOL_GFS * Nxx_plus_2NGHOSTS_tot);
  REAL *next_in_gfs = (REAL *)malloc(sizeof(REAL) * NUM_EVOL_GFS * Nxx_plus_2NGHOSTS_tot);
  REAL *aux_gfs  = (REAL *)malloc(sizeof(REAL) * NUM_AUX_GFS * Nxx_plus_2NGHOSTS_tot);
  REAL *k1_gfs = (REAL *)malloc(sizeof(REAL) * NUM_EVOL_GFS * Nxx_plus_2NGHOSTS_tot);
  REAL *k2_gfs = (REAL *)malloc(sizeof(REAL) * NUM_EVOL_GFS * Nxx_plus_2NGHOSTS_tot);
  REAL *k3_gfs = (REAL *)malloc(sizeof(REAL) * NUM_EVOL_GFS * Nxx_plus_2NGHOSTS_tot);
  REAL *k4_gfs = (REAL *)malloc(sizeof(REAL) * NUM_EVOL_GFS * Nxx_plus_2NGHOSTS_tot);

  // Step 0d: Set up space and time coordinates
  // Step 0d.i: Set \Delta x^i on uniform grids.
  REAL dxx[3];
  for(int i=0;i<3;i++) dxx[i] = (xxmax[i] - xxmin[i]) / ((REAL)Nxx[i]);

  // Step 0d.ii: Set up uniform coordinate grids
  REAL *xx[3];
  for(int i=0;i<3;i++) {
    xx[i] = (REAL *)malloc(sizeof(REAL)*Nxx_plus_2NGHOSTS[i]);
    for(int j=0;j<Nxx_plus_2NGHOSTS[i];j++) {
      xx[i][j] = xxmin[i] + ((REAL)(j-NGHOSTS) + (1.0/2.0))*dxx[i]; // Cell-centered grid.
    }
  }

  // Step 0d.iii: Set timestep based on smallest proper distance between gridpoints and CFL factor
  REAL dt = find_timestep(Nxx_plus_2NGHOSTS, dxx,xx, CFL_FACTOR);
  //printf("# Timestep set to = %e\n",(double)dt);
  int N_final = (int)(t_final / dt + 0.5); // The number of iterations in time.
                                           //Add 0.5 to account for C rounding down integers.

  // Step 0e: Find ghostzone mappings and parities:
  gz_map *bc_gz_map = (gz_map *)malloc(sizeof(gz_map)*Nxx_plus_2NGHOSTS_tot);
  parity_condition *bc_parity_conditions = (parity_condition *)malloc(sizeof(parity_condition)*Nxx_plus_2NGHOSTS_tot);
  set_up_bc_gz_map_and_parity_conditions(Nxx_plus_2NGHOSTS,xx,dxx,xxmin,xxmax,  bc_gz_map, bc_parity_conditions);

  // Step 1: Set up initial data to be exact solution at time=0:
  initial_data(Nxx_plus_2NGHOSTS, xx, evol_gfs);

  // Step 1b: Apply boundary conditions *FOR VALIDATION PURPOSES*
  apply_bcs(Nxx, Nxx_plus_2NGHOSTS, bc_gz_map,bc_parity_conditions, evol_gfs);
  enforce_detgammabar_constraint(Nxx_plus_2NGHOSTS, xx, evol_gfs);

  // Step 2: Evaluate Hamiltonian constraint violation
  //Hamiltonian_constraint(Nxx,Nxx_plus_2NGHOSTS,dxx, xx, evol_gfs, aux_gfs);

  // Step 3: Start the timer, for keeping track of how fast the simulation is progressing.
  struct timespec start, end;
  //clock_gettime(CLOCK_REALTIME, &start);

  // Step 4: Integrate the initial data forward in time using the Method of Lines and RK4

  char filename2[100];
  sprintf(filename2,"BSSN_SF-evolution/quad_pot_uu_vv_cf.txt");
  FILE *evol = fopen(filename2, "w");
  for(int n=0;n<=N_final;n++) { // Main loop to progress forward in time.
    /***************************************************/
    /* Implement RK4 for Method of Lines timestepping: */
    /***************************************************/
    /* -= RK4: Step 1 of 4 =- */
    /* First evaluate k1 = RHSs expression             */
    rhs_eval(Nxx,Nxx_plus_2NGHOSTS,dxx, xx,evol_gfs, k1_gfs);
    /* Next k1 -> k1*dt, and then set the input for    */
    /*    the next RHS eval call to y_n+k1/2           */
#pragma omp parallel for
    for(int i=0;i<Nxx_plus_2NGHOSTS_tot*NUM_EVOL_GFS;i++) {
      k1_gfs[i] *= dt;
      next_in_gfs[i] = evol_gfs[i] + k1_gfs[i]*0.5;
    }
    /* Finally, apply boundary conditions to           */
    /* next_in_gfs, so its data are set everywhere.    */
    apply_bcs(Nxx, Nxx_plus_2NGHOSTS, bc_gz_map,bc_parity_conditions, next_in_gfs);
    enforce_detgammabar_constraint(Nxx_plus_2NGHOSTS, xx, next_in_gfs);

    /* -= RK4: Step 2 of 4 =- */
    rhs_eval(Nxx,Nxx_plus_2NGHOSTS,dxx, xx,next_in_gfs, k2_gfs);
#pragma omp parallel for
    for(int i=0;i<Nxx_plus_2NGHOSTS_tot*NUM_EVOL_GFS;i++) {
      k2_gfs[i] *= dt;
      next_in_gfs[i] = evol_gfs[i] + k2_gfs[i]*0.5;
    }
    apply_bcs(Nxx, Nxx_plus_2NGHOSTS, bc_gz_map,bc_parity_conditions, next_in_gfs);
    enforce_detgammabar_constraint(Nxx_plus_2NGHOSTS, xx, next_in_gfs);

    /* -= RK4: Step 3 of 4 =- */
    rhs_eval(Nxx,Nxx_plus_2NGHOSTS,dxx, xx,next_in_gfs, k3_gfs);
#pragma omp parallel for
    for(int i=0;i<Nxx_plus_2NGHOSTS_tot*NUM_EVOL_GFS;i++) {
      k3_gfs[i] *= dt;
      next_in_gfs[i] = evol_gfs[i] + k3_gfs[i];
    }
    apply_bcs(Nxx, Nxx_plus_2NGHOSTS, bc_gz_map,bc_parity_conditions, next_in_gfs);
    enforce_detgammabar_constraint(Nxx_plus_2NGHOSTS, xx, next_in_gfs);

    /* -= RK4: Step 4 of 4 =- */
    rhs_eval(Nxx,Nxx_plus_2NGHOSTS,dxx, xx,next_in_gfs, k4_gfs);
#pragma omp parallel for
    for(int i=0;i<Nxx_plus_2NGHOSTS_tot*NUM_EVOL_GFS;i++) {
      k4_gfs[i] *= dt;
      evol_gfs[i] += (1.0/6.0)*(k1_gfs[i] + 2.0*k2_gfs[i] + 2.0*k3_gfs[i] + k4_gfs[i]);
    }
    Hamiltonian_constraint(Nxx,Nxx_plus_2NGHOSTS,dxx, xx, evol_gfs, aux_gfs);
    apply_bcs(Nxx, Nxx_plus_2NGHOSTS, bc_gz_map,bc_parity_conditions, evol_gfs);
    enforce_detgammabar_constraint(Nxx_plus_2NGHOSTS, xx, evol_gfs);
    /* Output the solution of the scalar field and the conformal factor at diffrent time slices on a 2D grid */
    if(n%10 == 0) {
        char filename[100];
        sprintf(filename,"BSSN_SF-output2D/quad_pot_2d_t-%08d.txt",n);
        FILE *out2D = fopen(filename, "w");
        const int i0MIN=NGHOSTS; // In spherical, r=Delta r/2.
        const int i1mid=Nxx_plus_2NGHOSTS[1]/2;
        const int i2mid=Nxx_plus_2NGHOSTS[2]/2;
        LOOP_REGION(NGHOSTS,Nxx_plus_2NGHOSTS[0]-NGHOSTS, NGHOSTS,Nxx_plus_2NGHOSTS[1]-NGHOSTS, NGHOSTS,Nxx_plus_2NGHOSTS[2]-NGHOSTS) {
           REAL xx0 = xx[0][i0];
           REAL xx1 = xx[1][i1];
           REAL xx2 = xx[2][i2];
           REAL xCart[3];
#include "xxCart.h"
           int idx = IDX3(i0,i1,i2);
             ADMCart_ID(out2D, n*(double)dt , xx0, xx1, xx2, xCart[0], xCart[1], xCart[2],
                        evol_gfs[IDX4pt(HDD00GF,idx)], evol_gfs[IDX4pt(HDD01GF,idx)], evol_gfs[IDX4pt(HDD02GF,idx)],
                        evol_gfs[IDX4pt(HDD11GF,idx)], evol_gfs[IDX4pt(HDD12GF,idx)], evol_gfs[IDX4pt(HDD22GF,idx)],
                        evol_gfs[IDX4pt(ADD00GF,idx)], evol_gfs[IDX4pt(ADD01GF,idx)], evol_gfs[IDX4pt(ADD02GF,idx)],
                        evol_gfs[IDX4pt(ADD11GF,idx)], evol_gfs[IDX4pt(ADD12GF,idx)], evol_gfs[IDX4pt(ADD22GF,idx)],
                        evol_gfs[IDX4pt(TRKGF,idx)],
                        evol_gfs[IDX4pt(LAMBDAU0GF,idx)],evol_gfs[IDX4pt(LAMBDAU1GF,idx)],evol_gfs[IDX4pt(LAMBDAU2GF,idx)],
                        evol_gfs[IDX4pt(VETU0GF,idx)],evol_gfs[IDX4pt(VETU1GF,idx)],evol_gfs[IDX4pt(VETU2GF,idx)],
                        evol_gfs[IDX4pt(BETU0GF,idx)],evol_gfs[IDX4pt(BETU1GF,idx)],evol_gfs[IDX4pt(BETU2GF,idx)],
                        evol_gfs[IDX4pt(ALPHAGF,idx)],evol_gfs[IDX4pt(CFGF,idx)],
                        evol_gfs[IDX4pt(UUGF,idx)],evol_gfs[IDX4pt(VVGF,idx)]);

         }
         fclose(out2D);
    }

    // Output time evolution at r=0
    int idx0 = IDX3(0,0,0);
    fprintf(evol,"%e %e %e %e\n", n*dt, evol_gfs[IDX4pt(UUGF,idx0)],evol_gfs[IDX4pt(VVGF,idx0)],evol_gfs[IDX4pt(CFGF,idx0)]);

    // Progress indicator printing to stdout
    // Measure average time per iteration
    //clock_gettime(CLOCK_REALTIME, &end);
    const long long unsigned int time_in_ns = 1000000000L * (end.tv_sec - start.tv_sec) + end.tv_nsec - start.tv_nsec;
    const REAL s_per_iteration_avg = ((REAL)time_in_ns / (REAL)n) / 1.0e9;

    const int iterations_remaining = N_final - n;
    const REAL time_remaining_in_mins = s_per_iteration_avg * (REAL)iterations_remaining / 60.0;

    const REAL num_RHS_pt_evals = (REAL)(Nxx[0]*Nxx[1]*Nxx[2]) * 4.0 * (REAL)n; // 4 RHS evals per gridpoint for RK4
    const REAL RHS_pt_evals_per_sec = num_RHS_pt_evals / ((REAL)time_in_ns / 1.0e9);

    // Progress indicator printing to stdout
    printf("%c[2K", 27); // Clear the line
    printf("It: %d t=%.2f | %.1f%%; ETA %.0f s | t/h %.2f | gp/s %.2e\r",  // \r is carriage return, move cursor to the beginning of the line
           n, n * (double)dt, (double)(100.0 * (REAL)n / (REAL)N_final),
           (double)time_remaining_in_mins*60, (double)(dt * 3600.0 / s_per_iteration_avg), (double)RHS_pt_evals_per_sec);
    fflush(stdout); // Flush the stdout buffer
  } // End main loop to progress forward in time.
  printf("\n"); // Clear the line.
  fclose(evol);

  /* Step 4: Free all allocated memory */
  free(bc_parity_conditions);
  free(bc_gz_map);
  free(k4_gfs);
  free(k3_gfs);
  free(k2_gfs);
  free(k1_gfs);
  free(aux_gfs);
  free(next_in_gfs);
  free(evol_gfs);
  for(int i=0;i<3;i++) free(xx[i]);
  return 0;
}Appending to BSSN_SF/BSSNScalar_Field.c
#!cd BSSN_SF/
import time
print("Now compiling, should take ~5 seconds...\n")
start = time.time()
#!gcc -Ofast -march=native -ftree-parallelize-loops=2 -fopenmp BSSN/BSSNCurvilinear_Playground.c -o BSSNCurvilinear_Playground -lm
!gcc -std=c11 -Ofast -march=native BSSN_SF/BSSNScalar_Field.c -o BSSN-SField -L/usr/local/lib -lgsl -lgslcblas -lm
end = time.time()
print("Finished in "+str(end-start)+" seconds. Running should take take ~40 seconds...\n")

start = time.time()
!./BSSN-SField 512 2 2
end = time.time()
print("Finished in "+str(end-start)+" seconds.")Now compiling, should take ~5 seconds...

BSSN_SF/BSSNScalar_Field.c: In function ‘initial_data’:
BSSN_SF/BSSNScalar_Field.c:132:25: warning: format ‘%lf’ expects argument of type ‘double *’, but argument 3 has type ‘int *’ [-Wformat=]
 while(fscanf(uu_file,"%lf\n",&temp)==1){
                       ~~^    ~~~~~
                       %d
BSSN_SF/BSSNScalar_Field.c:144:5: warning: ignoring return value of ‘fscanf’, declared with attribute warn_unused_result [-Wunused-result]
     fscanf(uu_file, "%lf\n", &uu_in_arr[i]);
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
BSSN_SF/BSSNScalar_Field.c:145:5: warning: ignoring return value of ‘fscanf’, declared with attribute warn_unused_result [-Wunused-result]
     fscanf(vv_file, "%lf\n", &vv_in_arr[i]);
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
BSSN_SF/BSSNScalar_Field.c:146:5: warning: ignoring return value of ‘fscanf’, declared with attribute warn_unused_result [-Wunused-result]
     fscanf(psi_file, "%lf\n", &psi_in_arr[i]);
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
BSSN_SF/BSSNScalar_Field.c:147:5: warning: ignoring return value of ‘fscanf’, declared with attribute warn_unused_result [-Wunused-result]
     fscanf(alpha_file, "%lf\n", &alpha_in_arr[i]);
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/bin/ld: cannot find -lgsl
/usr/bin/ld: cannot find -lgslcblas
collect2: error: ld returned 1 exit status
Finished in 6.16143703461 seconds. Running should take take ~40 seconds...

/bin/sh: 1: ./BSSN-SField: not found
Finished in 0.135982990265 seconds.
import os
import numpy as np
tag = 'ID_CC_9'
filelist = np.sort(os.listdir('./BSSN_SF-output2D/'))
keys = ['t','r','phi','pi','cf','Hcon','hDD00','hDD01','hDD02','hDD11','hDD12','hDD22','trK','ADD00','ADD01','ADD02','ADD11','ADD12','ADD22','alpha','beta1','beta2','beta3']
resolution = 512
t = np.array([])
r = np.array([])
phi = np.array([])
pi = np.array([])
cf = np.array([])
Hcon = np.array([])
hDD = []
trK = np.array([])
ADD = []
alpha = np.array([])
beta1 = np.array([])
beta2 = np.array([])
beta3 = np.array([])

for i in filelist:
    if 'quad' in i:
        data = np.genfromtxt('./BSSN_SF-output2D/'+i)
        t = np.append(t,data[:resolution,0],axis=0)
        r = np.append(r,data[:resolution,1],axis=0)
        phi = np.append(phi,data[:resolution,2],axis=0)
        pi = np.append(pi,data[:resolution,3],axis=0)
        cf = np.append(cf,data[:resolution,4],axis=0)
        Hcon = np.append(Hcon,data[:resolution,5],axis=0)
        hDD.append(data[:resolution,6:12],)
        trK = np.append(trK,data[:resolution,12],axis=0)
        ADD.append(data[:resolution,13:19])
        beta1 = np.append(beta1,data[:resolution,19],axis=0)
        beta2 = np.append(beta2,data[:resolution,20],axis=0)
        beta3 = np.append(beta3,data[:resolution,21],axis=0)
        alpha = np.append(alpha,data[:resolution,22],axis=0)

hDD = np.array(hDD)
ADD = np.array(ADD)

np.savez('./BSSN_SF-output2D/'+tag,t=t,r=r,phi=phi,pi=pi,cf=cf,Hcon=Hcon,hDD=hDD,trK=trK,ADD=ADD,alpha=alpha,beta1=beta1,beta2=beta2,beta3=beta3)_____no_output_____data.shape_____no_output_____import matplotlib.pyplot as plt
%matplotlib inline
plt.plot(phi.reshape(-1,512)[600].T)_____no_output_____N = 4
lines = [plt.plot([], [])[0] for _ in range(N)] #lines to animate

rectangles = plt.bar([0.5,1,1.5],[50,40,90],width=0.1) #rectangles to animate

patches = lines + list(rectangles) #things to animate
print(line)_____no_output_____

### Plot the scalar field time evolution_____no_output_____

import numpy as np
from scipy.interpolate import griddata
from pylab import savefig
import matplotlib.pyplot as plt
import matplotlib.cm as cm
from IPython.display import Image
from IPython.display import HTML
import matplotlib.image as mgimg
import glob
import sys
from matplotlib import animation
plt.rcParams.update(plt.rcParamsDefault)
plt.rc('xtick', labelsize=16)
plt.rc('ytick', labelsize=16)
plt.rcParams.update({'font.size': 16})
plt.rcParams.update({'font.family' : 'sans-serif'})
plt.rcParams.update({'font.sans-serif' : 'Helvetica'})
plt.rcParams.update({'figure.figsize' : '8., 2.9'})
plt.rcParams.update({'figure.autolayout': True})
evol = np.loadtxt("BSSN_SF-evolution/quad_pot_uu_vv_cf.txt")
t = evol[:,0]
uu = evol[:,1]
vv = evol[:,2]
cf = evol[:,3]
plt.plot(t,uu, color=cm.BuGn(8/10.,1))
plt.xlabel("time")
#plt.xticks([])
plt.ylabel(r"$\varphi(r=0)$")
#plt.ylabel(r"$e^{-4\phi}(r=0)$")
#plt.ylim(0.9,0.95)
#plt.ylim(-0.3,0.3)
plt.ylim(-0.049,0.049)
plt.xlim(-0.1,300)
plt.savefig("BSSN_SF-evolution/quad_pot_uu.png")
plt.close()
Image("BSSN_SF-evolution/quad_pot_uu.png")_____no_output_____

## Generate images of the scalar field on 2D grid for animation _____no_output_____

## VISUALIZATION ANIMATION, PART 1: Generate PNGs, one per frame of movie ##
import numpy as np
from scipy.interpolate import griddata
from pylab import savefig
import matplotlib.pyplot as plt
import matplotlib.cm as cm
from IPython.display import Image
from IPython.display import HTML
import matplotlib.image as mgimg
import glob
import sys
from matplotlib import animation

plt.rcParams.update(plt.rcParamsDefault)
plt.rc('xtick', labelsize=12)
plt.rc('ytick', labelsize=12)
plt.rcParams.update({'font.size': 12})
plt.rcParams.update({'font.family'       : 'sans-serif'})
plt.rcParams.update({'font.sans-serif'   : 'Helvetica'})
plt.rcParams.update({'figure.figsize'    :  '5, 3.5'})
plt.rcParams.update({'figure.autolayout': True})

globby = glob.glob('BSSN_SF-output2D/quad_pot_2d_t-*.txt')
file_list = []
for x in sorted(globby):
    file_list.append(x)

bound = 1000.
pl_xmin = -bound
pl_xmax = +bound
pl_ymin = -bound
pl_ymax = +bound

N_interp_pts  = 300
N_interp_ptsj = 300j
j = 0
maxtime = 300
numf = len(file_list)
for filename in file_list:
    j = j + 1
    time = j*maxtime/numf
    fig = plt.figure()
    x,y,sf,cf = np.loadtxt(filename).T #Transposed for easier unpacking

    plotquantity = sf
    plotdescription = "Scalar field"
    plt.title("time = %.2f"%time)
    plt.xlabel("y/M")
    plt.ylabel("z/M")

    grid_x, grid_y = np.mgrid[pl_xmin:pl_xmax:N_interp_ptsj, pl_ymin:pl_ymax:N_interp_ptsj]
    points = np.zeros((len(x), 2))
    for i in range(len(x)):
        points[i][0] = x[i]
        points[i][1] = y[i]

    grid = griddata(points, plotquantity, (grid_x, grid_y), method='nearest')
    gridcub = griddata(points, plotquantity, (grid_x, grid_y), method='cubic')
    im = plt.imshow(gridcub.T, extent=(pl_xmin,pl_xmax, pl_ymin,pl_ymax),cmap='YlGn')
    ax = plt.colorbar()
    ax.set_label(plotdescription)
    savefig(filename+"_sf.png",dpi=150)
    plt.close(fig)
    sys.stdout.write("%c[2K" % 27)
    sys.stdout.write("Processing file %.2f "%time+filename+"\r")
    sys.stdout.flush()_____no_output_____

## And for the conformal factor_____no_output_____

for filename in file_list:
    j = j + 1
    time = j*maxtime/numf
    fig = plt.figure()
    x,y,sf,cf = np.loadtxt(filename).T #Transposed for easier unpacking

    plotquantity = cf
    plotdescription = "Conformal factor"
    plt.title("time = %.2f"%time)
    plt.xlabel("y/M")
    plt.ylabel("z/M")

    grid_x, grid_y = np.mgrid[pl_xmin:pl_xmax:N_interp_ptsj, pl_ymin:pl_ymax:N_interp_ptsj]
    points = np.zeros((len(x), 2))
    for i in range(len(x)):
        # Zach says: No idea why x and y get flipped...
        points[i][0] = x[i]
        points[i][1] = y[i]

    grid = griddata(points, plotquantity, (grid_x, grid_y), method='nearest')
    gridcub = griddata(points, plotquantity, (grid_x, grid_y), method='cubic')
    #im = plt.imshow(grid, extent=(pl_xmin,pl_xmax, pl_ymin,pl_ymax))
    im = plt.imshow(gridcub.T, extent=(pl_xmin,pl_xmax, pl_ymin,pl_ymax),cmap='YlGn')
    #plt.pcolormesh(grid_y,grid_x, grid, vmin=-8, vmax=0) # Set colorbar range from -8 to 0
    ax = plt.colorbar(format='%.3f')
    ax.set_label(plotdescription)
    savefig(filename+"_cf.png",dpi=150)
    plt.close(fig)
    sys.stdout.write("%c[2K" % 27)
    sys.stdout.write("Processing file %.2f "%time+filename+"\r")
    sys.stdout.flush()_____no_output_____

## Generate animation of the time evolution of the scalar field and the conformal factor_____no_output_____

globby = glob.glob('BSSN_SF-output2D/quad_pot_2d_t-*.txt')
file_list = []
for x in sorted(globby):
    file_list.append(x)
## VISUALIZATION ANIMATION, PART 2: Combine PNGs to generate movie ##

# https://stackoverflow.com/questions/14908576/how-to-remove-frame-from-matplotlib-pyplot-figure-vs-matplotlib-figure-frame
# https://stackoverflow.com/questions/23176161/animating-pngs-in-matplotlib-using-artistanimation
# Set up formatting for the movie files
#Writer = animation.writers['ffmpeg']
#writer = Writer(fps=15, metadata=dict(artist='Me'), bitrate=1800)
import numpy as np
from scipy.interpolate import griddata
from pylab import savefig
import matplotlib.pyplot as plt
import matplotlib.cm as cm
from IPython.display import Image
from IPython.display import HTML
import matplotlib.image as mgimg
import glob
import sys
from matplotlib import animation
plt.rcParams['animation.ffmpeg_path'] = '/usr/local/bin/ffmpeg'
Writer = animation.writers['ffmpeg']
fig = plt.figure(frameon=False)
ax = fig.add_axes([0, 0, 1, 1])
ax.axis('off')

myimages = []

for i in range(len(file_list)-1):
    img = mgimg.imread(file_list[i]+"_sf.png")
    imgplot = plt.imshow(img)
    myimages.append([imgplot])

ani = animation.ArtistAnimation(fig, myimages, interval=100,  repeat_delay=1000)
plt.close()
ani.save('BSSN_SF-movies/quad_pot_sf.mp4',dpi=150, fps=8,writer='ffmpeg')_____no_output_____fig = plt.figure(frameon=False)
ax = fig.add_axes([0, 0, 1, 1])
ax.axis('off')

myimages = []

for i in range(len(file_list)-1):
    img = mgimg.imread(file_list[i]+"_cf.png")
    imgplot = plt.imshow(img)
    myimages.append([imgplot])

ani = animation.ArtistAnimation(fig, myimages, interval=100,  repeat_delay=1000)
plt.close()
ani.save('BSSN_SF-movies/quad_pot_cf.mp4',dpi=150, fps=8,writer='ffmpeg')_____no_output_____


# Notebook from chambox/chambox.github.io
Path: notebooks/validating_prediction_models.ipynb

# Validating prediction models

Nowadays machine learning (ML) and artificial intelligence (AI) are integral to most businesses. Decision-makers in many sectors, e.g., banking and finance, have employed ML-algorithms to do their heavy lifting. Though it sounds like a smart move, it is imperative to make sure these models are indeed doing what is expected of them. As employees of a bank, models are prone to mistakes and there is always a price to pay when we use ML-models. Hence, there is a need to validate models.

Model validation is a broad field and may cover several notions. In this article, we narrow down to prediction models. A model may be considered valid if:

1. it performs well, i.e., based on some mathematical metrics such as “a small miss-classification error” in classification prediction models.
2. the model is fair, i.e., not racist, sexist, homophobes, xenophobe, etc. These are cultural or legal validity aspects
3. the model is interpret-able, some experts may argue that black-box models are invalid. In such cases understanding how the model makes predictions is central

The three validity aspects above are not exhaustive; model validation may mean different things depending on who is the validator.

In this article, we will discuss model validation from the viewpoint of 1. Most data scientist when talking about model validation will default to point 1. Hereunder, we give models details on model validation based on prediction errors.

## Validating prediction models based on errors in prediction

Before making any progress, we will introduce some notations here:

$Y$: will represent the outcome we want to predict, let's say something like stock prices on a given day. We will denote the predicted $Y$ with $\hat Y $

$x:$  will represent the characteristics of the outcome — we will always know $x$ at the time prediction.  For our stock example,
$x$ can be a date, open and closing prices for that date and so on.

$m$:  will represent a prediction model, in practice, this model will often contain parameters which is our job to estimate them. Once we estimate these parameters, we then denote the model with estimated parameters with $\hat m$ — this will differentiate $\hat m$ from $m$, the model with the true parameters.

$\beta:$  will represent parameters of the model $m$, similarly $\hat \beta$ will represent the parameters of $\hat m$.

We calculate predictions as follows:

$$
\hat Y(x) = \hat m (x) = x^t \hat \beta
$$

and want the prediction error to be as small as possible. The prediction error for a prediction at predictor $x$ is given by

$$
\hat Y(x)-Y^{\star}
$$

$Y^\star$ is the outcome we want to predict that has $x$ as characteristics. Since a prediction model is typically used to predict an outcome before it is observed, the outcome $Y^\star$ is unknown at the time of prediction. Hence, the prediction error cannot be computed.

*Recall that a prediction model is estimated by using data from the training data set $(X, Y)$ and that $Y^\star$ is an outcome at $x$ which is assumed to be independent of the training data. The idea is that the prediction model is intended for use in predicting a future observation $Y^\star$, i.e., an observation that still has to be realized/observed (otherwise prediction seems rather useless). Hence, $Y^\star$  can never be part of the training data set.*

## Evaluating prediction performance

Here we provide definitions and we show how the prediction performance of a prediction model can be evaluated from data.

Let $T= (Y; X)$ denote the training data, from which the prediction model is built. The building process typically involves feature (characteristic) selection and parameter estimation. Below we define different types of errors used for model validation.

### Test or Generalisation, outsample Error

The test or generalization error for prediction model is given by

$$
\text{Err}_T = \text{E}_{Y^\star,X^\star}\big\{ (\hat m(X^\star)-Y^\star)^2|T\big\}
$$

where $(Y^\star, X^\star)$ is independent of the training data.

The test error is conditional on the training data $T$. Hence, the test error evaluates the performance of the single model built from the observed training data. This is the ultimate target of the model assessment because it is exactly this prediction model that will be used in practice and applied to future predictors $X^\star$ to predict $Y^\star$. The test error is defined as an average over all such future observations $(Y^\star, X^\star)$. The test error is the most interesting error for model validation according to point .1 above.

### Conditional Test Error in $x$

The conditional test error in $x$ for a prediction model $\hat m$ is given by

$$
\text{Err}_T(x) = \text{E}_{Y^\star}\big\{ (\hat m(x)-Y^\star)^2|T,x\big\}
$$

where $Y^\star$ is an outcome at predictor $x$, independent of the training data.

### In-simple Error

The in-sample error for a prediction model is given by

$$
\text{Err}_{\text{in}T} = \frac{1}{n}\sum_{i=1}^n \text{Err}_{T}(x_i),
$$

_____no_output_____i.e., the in-sample error is the sample average of the conditional test errors evaluated in the n training dataset observations.

### Estimation of the in-sample error

We start with introducing the training error rate, which is closely related to the mean squared error in linear models

#### Training error
The training error is given by
$$
\bar {\text{err}} = \frac{1}{n}\sum_{i=1}^n(Y_i-\hat m (x_i))^2,
$$

where the $(Y, x)$ form the training dataset which is also used for training the models.
* The training error is an overly optimistic estimate of the test error
* The training error never increase when the model becomes more complex — cannot be used directly as a model selection criterion

*Model parameters are often estimated by minimising the training error (cfr. mean squared error). Hence the fitted model adapts to the training data, and hence the training error will be an overly optimistic estimate of the test error*

Other estimators of the in-sample error are:

* The Akaike information criterion (AIC)
* Bayesian information criterion (BIC)
* Mallow’s Cp

## Expected Prediction Errors and Cross-Validation

The test or generalization error was defined condition on the training data. By averaging over the distribution of training datasets, the expected test error arises.

### Expected Test Error

$$
\text{E}_T\text{Err}_T = \text{E}_T\big\{\text{E}_{Y^\star,X^\star}\big\{ (\hat m(X^\star)-Y^\star)^2|T\big\}\}
$$
$$
= \text{E}_{Y^\star,X^\star,T}\big\{ (\hat m(X^\star)-Y^\star)^2\}
$$

*The expected test error may not be of direct interest when the goal is to assess the prediction performance of a single prediction model. The expected test error averages the test errors of all models that can be build from all training datasets, and hence this may be less relevant when the interest is in evaluating one particular model that resulted from a single observed training dataset.*

Also, note that building a prediction model involves both parameter estimation and feature selection. Hence the expected test error also evaluates the feature selection procedure (on average). If the expected test error is small, it is an indication that the model building
process gives good predictions for future observations $(Y^\star; X^\star)$ on average.

#### Estimation of the expected test error

*Cross-validation*

Randomly divide the training dataset into $k$ approximately equal subsets. Train your model on $k-1$ of them and compute the prediction error on the $k$th subset. Do this in such a way that all $k$ subsets have been used for training and for computing the prediction error. Average the prediction errors from all $k$ subset. This is the cross-validation error.

*The process of cross-validation is clearly mimicking an expected test error and not the test error. Clearly, cross-validation is estimating the expected test error and not the test error in which we are interested in.*

## Conclusion

Validating prediction models based on prediction errors can get complicated very quickly due to the different types of prediction errors that exist. A data scientist that uses any of these errors during model validation should be conscious of it. Ideally, the model validation error of interest is the **test error**. Proxies like the in-sample and generalized test error are often used in practice — when used their shortcomings should be clearly stated and the reason for using them should be also outlined.
If there is enough data, you could, in fact, partition your data into training, validation( for hyper-parameter tuning) and perhaps multiple testing sets. This way good estimates of the test error are possible, leading to proper model validation.
_____no_output_____