# Notebook from stevenrbrandt/nrpytutorial
Path: in_progress/Tutorial-Start_to_Finish-GiRaFFE_NRPy-3D_tests-unstaggered.ipynb

# Start-to-Finish Example: `GiRaFFE_NRPy` 3D tests

### Author: Patrick Nelson

### Adapted from [Start-to-Finish Example: Head-On Black Hole Collision](../Tutorial-Start_to_Finish-BSSNCurvilinear-Two_BHs_Collide.ipynb)

## This module implements a basic GRFFE code to evolve one-dimensional GRFFE waves.

### NRPy+ Source Code for this module:
* [GiRaFFEfood_NRPy/GiRaFFEfood_NRPy_Exact_Wald.py](../../edit/in_progress/GiRaFFEfood_NRPy/GiRaFFEfood_NRPy_Exact_Wald.py) [\[**tutorial**\]](Tutorial-GiRaFFEfood_NRPy_Exact_Wald.ipynb) Generates Exact Wald initial data
* [GiRaFFEfood_NRPy/GiRaFFEfood_NRPy_Aligned_Rotator.py](../../edit/in_progress/GiRaFFEfood_NRPy/GiRaFFEfood_NRPy_Aligned_Rotator.py) [\[**tutorial**\]](Tutorial-GiRaFFEfood_NRPy_Aligned_Rotator.ipynb) Generates Aligned Rotator initial data
* [GiRaFFEfood_NRPy/GiRaFFEfood_NRPy_1D_tests.py](../../edit/in_progress/GiRaFFEfood_NRPy/GiRaFFEfood_NRPy_1D_tests.py) [\[**tutorial**\]](Tutorial-GiRaFFEfood_NRPy_1D_tests.ipynb) Generates Alfvén Wave initial data.
* [GiRaFFE_NRPy/Afield_flux.py](../../edit/in_progress/GiRaFFE_NRPy/Afield_flux.py) [\[**tutorial**\]](Tutorial-GiRaFFE_NRPy-Afield_flux.ipynb) Generates the expressions to find the flux term of the induction equation.
* [GiRaFFE_NRPy/GiRaFFE_NRPy_A2B.py](../../edit/in_progress/GiRaFFE_NRPy/GiRaFFE_NRPy_A2B.py) [\[**tutorial**\]](Tutorial-GiRaFFE_NRPy-Afield_flux.ipynb) Generates the driver to compute the magnetic field from the vector potential/
* [GiRaFFE_NRPy/GiRaFFE_NRPy_BCs.py](../../edit/in_progress/GiRaFFE_NRPy/GiRaFFE_NRPy_BCs.py) [\[**tutorial**\]](Tutorial-GiRaFFE_NRPy-BCs.ipynb) Generates the code to apply boundary conditions to the vector potential, scalar potential, and three-velocity.
* [GiRaFFE_NRPy/GiRaFFE_NRPy_C2P_P2C.py](../../edit/in_progress/GiRaFFE_NRPy/GiRaFFE_NRPy_C2P_P2C.py) [\[**tutorial**\]](Tutorial-GiRaFFE_NRPy-C2P_P2C.ipynb) Generates the conservative-to-primitive and primitive-to-conservative solvers.
* [GiRaFFE_NRPy/GiRaFFE_NRPy_Metric_Face_Values.py](../../edit/in_progress/GiRaFFE_NRPy/GiRaFFE_NRPy_Metric_Face_Values.py) [\[**tutorial**\]](Tutorial-GiRaFFE_NRPy-Metric_Face_Values.ipynb) Generates code to interpolate metric gridfunctions to cell faces.
* [GiRaFFE_NRPy/GiRaFFE_NRPy_PPM.py](../../edit/in_progress/GiRaFFE_NRPy/GiRaFFE_NRPy_PPM.py) [\[**tutorial**\]](Tutorial-GiRaFFE_NRPy-PPM.ipynb) Genearates code to reconstruct primitive variables on cell faces.
* [GiRaFFE_NRPy/GiRaFFE_NRPy_Source_Terms.py](../../edit/in_progress/GiRaFFE_NRPy/GiRaFFE_NRPy_Source_Terms.py) [\[**tutorial**\]](Tutorial-GiRaFFE_NRPy-Source_Terms.ipynb) Generates the expressions to find the flux term of the Poynting flux evolution equation.
* [GiRaFFE_NRPy/Stilde_flux.py](../../edit/in_progress/GiRaFFE_NRPy/Stilde_flux.py) [\[**tutorial**\]](Tutorial-GiRaFFE_NRPy-Stilde_flux.ipynb) Generates the expressions to find the flux term of the Poynting flux evolution equation.
* [../GRFFE/equations.py](../../edit/GRFFE/equations.py) [\[**tutorial**\]](../Tutorial-GRFFE_Equations-Cartesian.ipynb) Generates code necessary to compute the source terms.
* [../GRHD/equations.py](../../edit/GRHD/equations.py) [\[**tutorial**\]](../Tutorial-GRHD_Equations-Cartesian.ipynb) Generates code necessary to compute the source terms.

Here we use NRPy+ to generate the C source code necessary to set up initial data for an Alfvén wave (see [the original GiRaFFE paper](https://arxiv.org/pdf/1704.00599.pdf)). Then we use it to generate the RHS expressions for [Method of Lines](https://reference.wolfram.com/language/tutorial/NDSolveMethodOfLines.html) time integration based on the [explicit Runge-Kutta fourth-order scheme](https://en.wikipedia.org/wiki/Runge%E2%80%93Kutta_methods) (RK4)._____no_output_____

# Table of Contents
$$\label{toc}$$

This notebook is organized as follows

1. [Step 1](#initializenrpy): Set core NRPy+ parameters for numerical grids
1. [Step 2](#grffe): Output C code for GRFFE evolution
    1. [Step 2.a](#mol): Output macros for Method of Lines timestepping
1. [Step 3](#gf_id): Import `GiRaFFEfood_NRPy` initial data modules
1. [Step 4](#cparams): Output C codes needed for declaring and setting Cparameters; also set `free_parameters.h`
1. [Step 5](#mainc): `GiRaFFE_NRPy_standalone.c`: The Main C Code_____no_output_____

# Step 1: Set up core functions and parameters for solving  GRFFE equations \[Back to [top](#toc)\]
$$\label{setup}$$
_____no_output_____

import shutil, os, sys           # Standard Python modules for multiplatform OS-level functions
# First, we'll add the parent directory to the list of directories Python will check for modules.
nrpy_dir_path = os.path.join("..")
if nrpy_dir_path not in sys.path:
    sys.path.append(nrpy_dir_path)

# Step P1: Import needed NRPy+ core modules:
from outputC import outCfunction, lhrh # NRPy+: Core C code output module
import sympy as sp               # SymPy: The Python computer algebra package upon which NRPy+ depends
import finite_difference as fin  # NRPy+: Finite difference C code generation module
import NRPy_param_funcs as par   # NRPy+: Parameter interface
import grid as gri               # NRPy+: Functions having to do with numerical grids
import indexedexp as ixp         # NRPy+: Symbolic indexed expression (e.g., tensors, vectors, etc.) support
import cmdline_helper as cmd     # NRPy+: Multi-platform Python command-line interface

# Step P2: Create C code output directory:
Ccodesdir = os.path.join("GiRaFFE_unstaggered_standalone_Ccodes/")
# First remove C code output directory if it exists
# Courtesy https://stackoverflow.com/questions/303200/how-do-i-remove-delete-a-folder-that-is-not-empty
# !rm -r ScalarWaveCurvilinear_Playground_Ccodes
shutil.rmtree(Ccodesdir, ignore_errors=True)
# Then create a fresh directory
cmd.mkdir(Ccodesdir)

# Step P3: Create executable output directory:
outdir = os.path.join(Ccodesdir,"output/")
cmd.mkdir(Ccodesdir)
cmd.mkdir(outdir)

# Step P5: Set timestepping algorithm (we adopt the Method of Lines)
REAL      = "double"    # Best to use double here.
default_CFL_FACTOR= 0.5 # (GETS OVERWRITTEN WHEN EXECUTED.) In pure axisymmetry (symmetry_axes = 2 below) 1.0 works fine. Otherwise 0.5 or lower.

# Step P6: Set the finite differencing order to 2.
par.set_parval_from_str("finite_difference::FD_CENTDERIVS_ORDER",2)

thismodule = "Start_to_Finish-GiRaFFE_NRPy-1D_tests"
TINYDOUBLE = par.Cparameters("REAL", thismodule, "TINYDOUBLE", 1e-100)

import GiRaFFE_NRPy.GiRaFFE_NRPy_Main_Driver as md
# par.set_paramsvals_value("GiRaFFE_NRPy.GiRaFFE_NRPy_C2P_P2C::enforce_speed_limit_StildeD = False")
par.set_paramsvals_value("GiRaFFE_NRPy.GiRaFFE_NRPy_C2P_P2C::enforce_current_sheet_prescription = False")_____no_output_____

# Step 2: Output C code for GRFFE evolution \[Back to [top](#toc)\]
$$\label{grffe}$$

We will first write the C codes needed for GRFFE evolution. We have already written a module to generate all these codes and call the functions in the appropriate order, so we will import that here. We will take the slightly unusual step of doing this before we generate the initial data functions because the main driver module will register all the gridfunctions we need. It will also generate functions that, in addition to their normal spot in the MoL timestepping, will need to be called during the initial data step to make sure all the variables are appropriately filled in.

All of this is handled with a single call to `GiRaFFE_NRPy_Main_Driver_generate_all()`, which will register gridfunctions, write all the C code kernels, and write the C code functions to call those._____no_output_____

md.GiRaFFE_NRPy_Main_Driver_generate_all(Ccodesdir)Output C function calculate_AD_gauge_term_psi6Phi_flux_term_for_RHSs() to file GiRaFFE_unstaggered_standalone_Ccodes/RHSs/calculate_AD_gauge_term_psi6Phi_flux_term_for_RHSs.h
Output C function calculate_StildeD0_source_term() to file GiRaFFE_unstaggered_standalone_Ccodes/RHSs/calculate_StildeD0_source_term.h
Output C function calculate_StildeD1_source_term() to file GiRaFFE_unstaggered_standalone_Ccodes/RHSs/calculate_StildeD1_source_term.h
Output C function calculate_StildeD2_source_term() to file GiRaFFE_unstaggered_standalone_Ccodes/RHSs/calculate_StildeD2_source_term.h
Output C function calculate_Stilde_rhsD() to file GiRaFFE_unstaggered_standalone_Ccodes/RHSs/calculate_Stilde_rhsD.h
Output C function GiRaFFE_NRPy_cons_to_prims() to file GiRaFFE_unstaggered_standalone_Ccodes/C2P/GiRaFFE_NRPy_cons_to_prims.h
Output C function GiRaFFE_NRPy_prims_to_cons() to file GiRaFFE_unstaggered_standalone_Ccodes/C2P/GiRaFFE_NRPy_prims_to_cons.h

## Step 2.a: Output macros for Method of Lines timestepping \[Back to [top](#toc)\]
$$\label{mol}$$

Now, we generate the code to implement the method of lines using the fourth-order Runge-Kutta algorithm._____no_output_____

RK_method = "RK4"

# Step 3: Generate Runge-Kutta-based (RK-based) timestepping code.
#       As described above the Table of Contents, this is a 3-step process:
#       3.A: Evaluate RHSs (RHS_string)
#       3.B: Apply boundary conditions (post_RHS_string, pt 1)
import MoLtimestepping.C_Code_Generation as MoL
from MoLtimestepping.RK_Butcher_Table_Dictionary import Butcher_dict
RK_order  = Butcher_dict[RK_method][1]
cmd.mkdir(os.path.join(Ccodesdir,"MoLtimestepping/"))
MoL.MoL_C_Code_Generation(RK_method,
    RHS_string      = """
GiRaFFE_NRPy_RHSs(&params,auxevol_gfs,RK_INPUT_GFS,RK_OUTPUT_GFS);""",
    post_RHS_string = """
GiRaFFE_NRPy_post_step(&params,xx,auxevol_gfs,RK_OUTPUT_GFS,n+1);\n""",
    outdir = os.path.join(Ccodesdir,"MoLtimestepping/"))
_____no_output_____

# Step 3: Import `GiRaFFEfood_NRPy` initial data modules \[Back to [top](#toc)\]
$$\label{gf_id}$$

With the preliminaries out of the way, we will write the C functions to set up initial data. There are two categories of initial data that must be set: the spacetime metric variables, and the GRFFE plasma variables. We will set up the spacetime first._____no_output_____

# There are several initial data routines we need to test. We'll control which one we use with a string option
initial_data = "ExactWald" # Valid options: "ExactWald", "AlignedRotator"
spacetime = "ShiftedKerrSchild" # Valid options: "ShiftedKerrSchild", "flat"

if spacetime == "ShiftedKerrSchild":
    # Exact Wald is more complicated. We'll need the Shifted Kerr Schild metric in Cartesian coordinates.
    import BSSN.ShiftedKerrSchild as sks
    sks.ShiftedKerrSchild(True)
    import reference_metric as rfm
    par.set_parval_from_str("reference_metric::CoordSystem","Cartesian")
    rfm.reference_metric()
    # Use the Jacobian matrix to transform the vectors to Cartesian coordinates.
    par.set_parval_from_str("reference_metric::CoordSystem","Spherical")
    rfm.reference_metric()
    Jac_dUCart_dDrfmUD,Jac_dUrfm_dDCartUD = rfm.compute_Jacobian_and_inverseJacobian_tofrom_Cartesian()
    # Transform the coordinates of the Jacobian matrix from spherical to Cartesian:
    par.set_parval_from_str("reference_metric::CoordSystem","Cartesian")
    rfm.reference_metric()
    tmpa,tmpb,tmpc = sp.symbols("tmpa,tmpb,tmpc")
    for i in range(3):
        for j in range(3):
            Jac_dUCart_dDrfmUD[i][j] = Jac_dUCart_dDrfmUD[i][j].subs([(rfm.xx[0],tmpa),(rfm.xx[1],tmpb),(rfm.xx[2],tmpc)])
            Jac_dUCart_dDrfmUD[i][j] = Jac_dUCart_dDrfmUD[i][j].subs([(tmpa,rfm.xxSph[0]),(tmpb,rfm.xxSph[1]),(tmpc,rfm.xxSph[2])])
            Jac_dUrfm_dDCartUD[i][j] = Jac_dUrfm_dDCartUD[i][j].subs([(rfm.xx[0],tmpa),(rfm.xx[1],tmpb),(rfm.xx[2],tmpc)])
            Jac_dUrfm_dDCartUD[i][j] = Jac_dUrfm_dDCartUD[i][j].subs([(tmpa,rfm.xxSph[0]),(tmpb,rfm.xxSph[1]),(tmpc,rfm.xxSph[2])])

    gammaSphDD = ixp.zerorank2()
    for i in range(3):
        for j in range(3):
            gammaSphDD[i][j] += sks.gammaSphDD[i][j].subs(sks.r,rfm.xxSph[0]).subs(sks.th,rfm.xxSph[1])

    betaSphU = ixp.zerorank1()
    for i in range(3):
        betaSphU[i] += sks.betaSphU[i].subs(sks.r,rfm.xxSph[0]).subs(sks.th,rfm.xxSph[1])
    alpha = sks.alphaSph.subs(sks.r,rfm.xxSph[0]).subs(sks.th,rfm.xxSph[1])

    gammaDD = rfm.basis_transform_tensorDD_from_rfmbasis_to_Cartesian(Jac_dUrfm_dDCartUD, gammaSphDD)

    unused_gammaUU,gammaDET = ixp.symm_matrix_inverter3x3(gammaDD)
    sqrtgammaDET = sp.sqrt(gammaDET)

    betaU = rfm.basis_transform_vectorD_from_rfmbasis_to_Cartesian(Jac_dUrfm_dDCartUD, betaSphU)

    # Description and options for this initial data
    desc = "Generate a spinning black hole with Shifted Kerr Schild metric."
    loopopts_id ="AllPoints,Read_xxs"

elif spacetime == "flat":
    gammaDD = ixp.zerorank2(DIM=3)
    for i in range(3):
        for j in range(3):
            if i==j:
                gammaDD[i][j] = sp.sympify(1) # else: leave as zero
    betaU = ixp.zerorank1() # All should be 0
    alpha = sp.sympify(1)

    # Description and options for this initial data
    desc = "Generate a flat spacetime metric."
    loopopts_id ="AllPoints" # we don't need to read coordinates for flat spacetime.

name = "set_initial_spacetime_metric_data"
values_to_print = [\
                   lhrh(lhs=gri.gfaccess("auxevol_gfs","gammaDD00"),rhs=gammaDD[0][0]),\
                   lhrh(lhs=gri.gfaccess("auxevol_gfs","gammaDD01"),rhs=gammaDD[0][1]),\
                   lhrh(lhs=gri.gfaccess("auxevol_gfs","gammaDD02"),rhs=gammaDD[0][2]),\
                   lhrh(lhs=gri.gfaccess("auxevol_gfs","gammaDD11"),rhs=gammaDD[1][1]),\
                   lhrh(lhs=gri.gfaccess("auxevol_gfs","gammaDD12"),rhs=gammaDD[1][2]),\
                   lhrh(lhs=gri.gfaccess("auxevol_gfs","gammaDD22"),rhs=gammaDD[2][2]),\
                   lhrh(lhs=gri.gfaccess("auxevol_gfs","betaU0"),rhs=betaU[0]),\
                   lhrh(lhs=gri.gfaccess("auxevol_gfs","betaU1"),rhs=betaU[1]),\
                   lhrh(lhs=gri.gfaccess("auxevol_gfs","betaU2"),rhs=betaU[2]),\
                   lhrh(lhs=gri.gfaccess("auxevol_gfs","alpha"),rhs=alpha),\
                  ]

outCfunction(
    outfile  = os.path.join(Ccodesdir,name+".h"), desc=desc, name=name,
    params   ="const paramstruct *params,REAL *xx[3],REAL *auxevol_gfs",
    body     = fin.FD_outputC("returnstring",values_to_print,params="outCverbose=False"),
    loopopts = loopopts_id)
Output C function set_initial_spacetime_metric_data() to file GiRaFFE_unstaggered_standalone_Ccodes/set_initial_spacetime_metric_data.h

Now, we will write out the initial data function for the GRFFE variables._____no_output_____

import GiRaFFEfood_NRPy.GiRaFFEfood_NRPy as gid
if initial_data=="ExactWald":
    gid.GiRaFFEfood_NRPy_generate_initial_data(ID_type = initial_data, stagger_enable = False,M=sks.M,KerrSchild_radial_shift=sks.r0,gammaDD=gammaDD,sqrtgammaDET=sqrtgammaDET)
    desc = "Generate exact Wald initial test data for GiRaFFEfood_NRPy."
elif initial_data=="SplitMonopole":
    gid.GiRaFFEfood_NRPy_generate_initial_data(ID_type = initial_data, stagger_enable = False,M=sks.M,a=sks.a,KerrSchild_radial_shift=sks.r0,alpha=alpha,betaU=betaSphU,gammaDD=gammaDD,sqrtgammaDET=sqrtgammaSphDET)
    desc = "Generate Split Monopole initial test data for GiRaFFEfood_NRPy."
elif initial_data=="AlignedRotator":
    gf.GiRaFFEfood_NRPy_generate_initial_data(ID_type = initial_data, stagger_enable = True)
    desc = "Generate aligned rotator initial test data for GiRaFFEfood_NRPy."
else:
    print("Unsupported Initial Data string "+initial_data+"! Supported ID: ExactWald, or SplitMonopole")

name = "initial_data"

values_to_print = [
                   lhrh(lhs=gri.gfaccess("out_gfs","AD0"),rhs=gid.AD[0]),
                   lhrh(lhs=gri.gfaccess("out_gfs","AD1"),rhs=gid.AD[1]),
                   lhrh(lhs=gri.gfaccess("out_gfs","AD2"),rhs=gid.AD[2]),
                   lhrh(lhs=gri.gfaccess("auxevol_gfs","ValenciavU0"),rhs=gid.ValenciavU[0]),
                   lhrh(lhs=gri.gfaccess("auxevol_gfs","ValenciavU1"),rhs=gid.ValenciavU[1]),
                   lhrh(lhs=gri.gfaccess("auxevol_gfs","ValenciavU2"),rhs=gid.ValenciavU[2]),
#                    lhrh(lhs=gri.gfaccess("auxevol_gfs","BU0"),rhs=gid.BU[0]),
#                    lhrh(lhs=gri.gfaccess("auxevol_gfs","BU1"),rhs=gid.BU[1]),
#                    lhrh(lhs=gri.gfaccess("auxevol_gfs","BU2"),rhs=gid.BU[2]),
                   lhrh(lhs=gri.gfaccess("out_gfs","psi6Phi"),rhs=sp.sympify(0))
                  ]

outCfunction(
    outfile  = os.path.join(Ccodesdir,name+".h"), desc=desc, name=name,
    params   ="const paramstruct *params,REAL *xx[3],REAL *auxevol_gfs,REAL *out_gfs",
    body     = fin.FD_outputC("returnstring",values_to_print,params="outCverbose=False"),
    loopopts ="AllPoints,Read_xxs")
Output C function initial_data() to file GiRaFFE_unstaggered_standalone_Ccodes/initial_data.h

# Step 4: Output C codes needed for declaring and setting Cparameters; also set `free_parameters.h` \[Back to [top](#toc)\]
$$\label{cparams}$$

Based on declared NRPy+ Cparameters, first we generate `declare_Cparameters_struct.h`, `set_Cparameters_default.h`, and `set_Cparameters[-SIMD].h`.

Then we output `free_parameters.h`, which sets initial data parameters, as well as grid domain & reference metric parameters, applying `domain_size` and `sinh_width`/`SymTP_bScale` (if applicable) as set above_____no_output_____

# Step 3.e: Output C codes needed for declaring and setting Cparameters; also set free_parameters.h
# Step 3.e.i: Generate declare_Cparameters_struct.h, set_Cparameters_default.h, and set_Cparameters[-SIMD].h
par.generate_Cparameters_Ccodes(os.path.join(Ccodesdir))

# Step 3.e.ii: Set free_parameters.h
with open(os.path.join(Ccodesdir,"free_parameters.h"),"w") as file:
    file.write("""// Override parameter defaults with values based on command line arguments and NGHOSTS.
params.Nxx0 = atoi(argv[1]);
params.Nxx1 = atoi(argv[2]);
params.Nxx2 = atoi(argv[3]);
params.Nxx_plus_2NGHOSTS0 = params.Nxx0 + 2*NGHOSTS;
params.Nxx_plus_2NGHOSTS1 = params.Nxx1 + 2*NGHOSTS;
params.Nxx_plus_2NGHOSTS2 = params.Nxx2 + 2*NGHOSTS;
// Step 0d: Set up space and time coordinates
// Step 0d.i: Declare \Delta x^i=dxx{0,1,2} and invdxx{0,1,2}, as well as xxmin[3] and xxmax[3]:
const REAL xxmin[3] = {-1.5,-1.5,-1.5};
const REAL xxmax[3] = { 1.5, 1.5, 1.5};

params.dxx0 = (xxmax[0] - xxmin[0]) / ((REAL)params.Nxx0+1);
params.dxx1 = (xxmax[1] - xxmin[1]) / ((REAL)params.Nxx1+1);
params.dxx2 = (xxmax[2] - xxmin[2]) / ((REAL)params.Nxx2+1);
printf("dxx0,dxx1,dxx2 = %.5e,%.5e,%.5e\\n",params.dxx0,params.dxx1,params.dxx2);
params.invdx0 = 1.0 / params.dxx0;
params.invdx1 = 1.0 / params.dxx1;
params.invdx2 = 1.0 / params.dxx2;

const int poison_grids = 0;
// Standard GRFFE parameters:
params.GAMMA_SPEED_LIMIT = 2000.0;
params.diss_strength = 0.1;
""")
if initial_data=="ExactWald":
    with open(os.path.join(Ccodesdir,"free_parameters.h"),"a") as file:
        file.write("""params.r0 = 0.4;
params.a = 0.0;
""")
_____no_output_____

# Step 4: Set up boundary condition functions for chosen singular, curvilinear coordinate system \[Back to [top](#toc)\]
$$\label{bc_functs}$$

Next apply singular, curvilinear coordinate boundary conditions [as documented in the corresponding NRPy+ tutorial notebook](Tutorial-Start_to_Finish-Curvilinear_BCs.ipynb)

...But, for the moment, we're actually just using this because it writes the file `gridfunction_defines.h`._____no_output_____

import CurviBoundaryConditions.CurviBoundaryConditions as cbcs
cbcs.Set_up_CurviBoundaryConditions(os.path.join(Ccodesdir,"boundary_conditions/"),Cparamspath=os.path.join("../"),enable_copy_of_static_Ccodes=False)Wrote to file "GiRaFFE_unstaggered_standalone_Ccodes/boundary_conditions/parity_conditions_symbolic_dot_products.h"
Evolved parity: ( AD0:1, AD1:2, AD2:3, StildeD0:1, StildeD1:2, StildeD2:3,
    psi6Phi:0 )

AuxEvol parity: ( AevolParen:0, BU0:1, BU1:2, BU2:3, B_lU0:1, B_lU1:2,
    B_lU2:3, B_rU0:1, B_rU1:2, B_rU2:3, PhievolParenU0:1, PhievolParenU1:2,
    PhievolParenU2:3, Stilde_flux_HLLED0:1, Stilde_flux_HLLED1:2,
    Stilde_flux_HLLED2:3, ValenciavU0:1, ValenciavU1:2, ValenciavU2:3,
    Valenciav_lU0:1, Valenciav_lU1:2, Valenciav_lU2:3, Valenciav_rU0:1,
    Valenciav_rU1:2, Valenciav_rU2:3, alpha:0, alpha_face:0, betaU0:1,
    betaU1:2, betaU2:3, beta_faceU0:1, beta_faceU1:2, beta_faceU2:3,
    gammaDD00:4, gammaDD01:5, gammaDD02:6, gammaDD11:7, gammaDD12:8,
    gammaDD22:9, gamma_faceDD00:4, gamma_faceDD01:5, gamma_faceDD02:6,
    gamma_faceDD11:7, gamma_faceDD12:8, gamma_faceDD22:9 )
Wrote to file "GiRaFFE_unstaggered_standalone_Ccodes/boundary_conditions/EigenCoord_Cart_to_xx.h"

# Step 5: `GiRaFFE_NRPy_standalone.c`: The Main C Code \[Back to [top](#toc)\]
$$\label{mainc}$$_____no_output_____

# Part P0: Define REAL, set the number of ghost cells NGHOSTS (from NRPy+'s FD_CENTDERIVS_ORDER),
#          and set the CFL_FACTOR (which can be overwritten at the command line)

with open(os.path.join(Ccodesdir,"GiRaFFE_NRPy_REAL__NGHOSTS__CFL_FACTOR.h"), "w") as file:
    file.write("""
// Part P0.a: Set the number of ghost cells, from NRPy+'s FD_CENTDERIVS_ORDER
#define NGHOSTS """+str(3)+"""
#define NGHOSTS_A2B """+str(2)+"""
// Part P0.b: Set the numerical precision (REAL) to double, ensuring all floating point
//            numbers are stored to at least ~16 significant digits
#define REAL """+REAL+"""
// Part P0.c: Set the CFL Factor. Can be overwritten at command line.
REAL CFL_FACTOR = """+str(default_CFL_FACTOR)+";")_____no_output_____%%writefile $Ccodesdir/GiRaFFE_NRPy_standalone.c
// Step P0: Define REAL and NGHOSTS; and declare CFL_FACTOR. This header is generated in NRPy+.
#include "GiRaFFE_NRPy_REAL__NGHOSTS__CFL_FACTOR.h"

#include "declare_Cparameters_struct.h"

const int NSKIP_1D_OUTPUT = 1;

// Step P1: Import needed header files
#include "stdio.h"
#include "stdlib.h"
#include "math.h"
#include "time.h"
#include "stdint.h" // Needed for Windows GCC 6.x compatibility
#ifndef M_PI
#define M_PI 3.141592653589793238462643383279502884L
#endif
#ifndef M_SQRT1_2
#define M_SQRT1_2 0.707106781186547524400844362104849039L
#endif

// Step P2: Declare the IDX4S(gf,i,j,k) macro, which enables us to store 4-dimensions of
//           data in a 1D array. In this case, consecutive values of "i"
//           (all other indices held to a fixed value) are consecutive in memory, where
//           consecutive values of "j" (fixing all other indices) are separated by
//           Nxx_plus_2NGHOSTS0 elements in memory. Similarly, consecutive values of
//           "k" are separated by Nxx_plus_2NGHOSTS0*Nxx_plus_2NGHOSTS1 in memory, etc.
#define IDX4S(g,i,j,k) \
( (i) + Nxx_plus_2NGHOSTS0 * ( (j) + Nxx_plus_2NGHOSTS1 * ( (k) + Nxx_plus_2NGHOSTS2 * (g) ) ) )
#define IDX4ptS(g,idx) ( (idx) + (Nxx_plus_2NGHOSTS0*Nxx_plus_2NGHOSTS1*Nxx_plus_2NGHOSTS2) * (g) )
#define IDX3S(i,j,k) ( (i) + Nxx_plus_2NGHOSTS0 * ( (j) + Nxx_plus_2NGHOSTS1 * ( (k) ) ) )
#define LOOP_REGION(i0min,i0max, i1min,i1max, i2min,i2max) \
  for(int i2=i2min;i2<i2max;i2++) for(int i1=i1min;i1<i1max;i1++) for(int i0=i0min;i0<i0max;i0++)
#define LOOP_ALL_GFS_GPS(ii) _Pragma("omp parallel for") \
  for(int (ii)=0;(ii)<Nxx_plus_2NGHOSTS_tot*NUM_EVOL_GFS;(ii)++)

// Step P3: Set gridfunction macros
#include "boundary_conditions/gridfunction_defines.h"

// Step P4: Include the RHS, BC, and primitive recovery functions
#include "GiRaFFE_NRPy_Main_Driver.h"

// Step P5: Include the initial data functions
#include "set_initial_spacetime_metric_data.h"
#include "initial_data.h"
// main() function:
// Step 0: Read command-line input, set up grid structure, allocate memory for gridfunctions, set up coordinates
// Step 1: Set up scalar wave initial data
// Step 2: Evolve scalar wave initial data forward in time using Method of Lines with RK4 algorithm,
//         applying quadratic extrapolation outer boundary conditions.
// Step 3: Output relative error between numerical and exact solution.
// Step 4: Free all allocated memory
int main(int argc, const char *argv[]) {
    paramstruct params;
#include "set_Cparameters_default.h"
    // Step 0a: Read command-line input, error out if nonconformant
    if(argc != 4 || atoi(argv[1]) < NGHOSTS || atoi(argv[2]) < NGHOSTS || atoi(argv[3]) < NGHOSTS) {
      printf("Error: Expected three command-line arguments: ./GiRaFFE_NRPy_standalone [Nx] [Ny] [Nz],\n");
      printf("where Nx is the number of grid points in the x direction, and so forth.\n");
      printf("Nx,Ny,Nz MUST BE larger than NGHOSTS (= %d)\n",NGHOSTS);
      exit(1);
    }
    // Step 0c: Set free parameters, overwriting Cparameters defaults
    //          by hand or with command-line input, as desired.
#include "free_parameters.h"
#include "set_Cparameters-nopointer.h"

    //          ... and then set up the numerical grid structure in time:
    const REAL t_final = 0.5;
    const REAL CFL_FACTOR = 0.5; // Set the CFL Factor

    // Step 0c: Allocate memory for gridfunctions
    const int Nxx_plus_2NGHOSTS_tot = Nxx_plus_2NGHOSTS0*Nxx_plus_2NGHOSTS1*Nxx_plus_2NGHOSTS2;
    // Step 0k: Allocate memory for gridfunctions
#include "MoLtimestepping/RK_Allocate_Memory.h"
    REAL *restrict auxevol_gfs = (REAL *)malloc(sizeof(REAL) * NUM_AUXEVOL_GFS * Nxx_plus_2NGHOSTS_tot);
    REAL *evol_gfs_exact = (REAL *)malloc(sizeof(REAL) * NUM_EVOL_GFS * Nxx_plus_2NGHOSTS_tot);
    REAL *auxevol_gfs_exact  = (REAL *)malloc(sizeof(REAL) * NUM_AUXEVOL_GFS * Nxx_plus_2NGHOSTS_tot);

    // For debugging, it can be useful to set everything to NaN initially.
    if(poison_grids) {
        for(int ii=0;ii<NUM_EVOL_GFS * Nxx_plus_2NGHOSTS_tot;ii++) {
            y_n_gfs[ii] = 1.0/0.0;
            y_nplus1_running_total_gfs[ii] = 1.0/0.0;
            //k_odd_gfs[ii] = 1.0/0.0;
            //k_even_gfs[ii] = 1.0/0.0;
            diagnostic_output_gfs[ii] = 1.0/0.0;
            evol_gfs_exact[ii] = 1.0/0.0;
        }
        for(int ii=0;ii<NUM_AUXEVOL_GFS * Nxx_plus_2NGHOSTS_tot;ii++) {
            auxevol_gfs[ii] = 1.0/0.0;
            auxevol_gfs_exact[ii] = 1.0/0.0;
        }
    }

    // Step 0d: Set up coordinates: Set dx, and then dt based on dx_min and CFL condition
    // This is probably already defined above, but just in case...
    #ifndef MIN
    #define MIN(A, B) ( ((A) < (B)) ? (A) : (B) )
    #endif
    REAL dt = CFL_FACTOR * MIN(dxx0,MIN(dxx1,dxx2)); // CFL condition
    int Nt = (int)(t_final / dt + 0.5); // The number of points in time.
                                        //Add 0.5 to account for C rounding down integers.

    // Step 0e: Set up cell-centered Cartesian coordinate grids
    REAL *xx[3];
    xx[0] = (REAL *)malloc(sizeof(REAL)*Nxx_plus_2NGHOSTS0);
    xx[1] = (REAL *)malloc(sizeof(REAL)*Nxx_plus_2NGHOSTS1);
    xx[2] = (REAL *)malloc(sizeof(REAL)*Nxx_plus_2NGHOSTS2);
    for(int j=0;j<Nxx_plus_2NGHOSTS0;j++) xx[0][j] = xxmin[0] + (j-NGHOSTS+1)*dxx0;
    for(int j=0;j<Nxx_plus_2NGHOSTS1;j++) xx[1][j] = xxmin[1] + (j-NGHOSTS+1)*dxx1;
    for(int j=0;j<Nxx_plus_2NGHOSTS2;j++) xx[2][j] = xxmin[2] + (j-NGHOSTS+1)*dxx2;

    // Step 1: Set up initial data to be exact solution at time=0:
    REAL time = 0.0;
    set_initial_spacetime_metric_data(&params,xx,auxevol_gfs);
    initial_data(&params,xx,auxevol_gfs,y_n_gfs);
    // Fill in  the remaining quantities
    apply_bcs_potential(&params,y_n_gfs);
    driver_A_to_B(&params,y_n_gfs,auxevol_gfs);
    //override_BU_with_old_GiRaFFE(&params,auxevol_gfs,0);
    GiRaFFE_NRPy_prims_to_cons(&params,auxevol_gfs,y_n_gfs);
    apply_bcs_velocity(&params,auxevol_gfs);
    // Extra stack, useful for debugging:
    GiRaFFE_NRPy_cons_to_prims(&params,xx,auxevol_gfs,y_n_gfs);
    //GiRaFFE_NRPy_prims_to_cons(&params,auxevol_gfs,y_n_gfs);
    //GiRaFFE_NRPy_cons_to_prims(&params,xx,auxevol_gfs,y_n_gfs);
    //GiRaFFE_NRPy_prims_to_cons(&params,auxevol_gfs,y_n_gfs);
    //GiRaFFE_NRPy_cons_to_prims(&params,xx,auxevol_gfs,y_n_gfs);

    for(int n=0;n<=Nt;n++) { // Main loop to progress forward in time.
    //for(int n=0;n<=1;n++) { // Main loop to progress forward in time.
        // Step 1a: Set current time to correct value & compute exact solution
        time = ((REAL)n)*dt;

        /* Step 2: Validation: Output relative error between numerical and exact solution, */
        if((n)%NSKIP_1D_OUTPUT ==0) {
            // Step 2c: Output relative error between exact & numerical at center of grid.
            const int i0mid=Nxx_plus_2NGHOSTS0/2;
            const int i1mid=Nxx_plus_2NGHOSTS1/2;
            const int i2mid=Nxx_plus_2NGHOSTS2/2;
            char filename[100];
            sprintf(filename,"out%d-%08d.txt",Nxx0,n);
            FILE *out2D = fopen(filename, "w");
            for(int i0=0;i0<Nxx_plus_2NGHOSTS0;i0++) {
                const int idx = IDX3S(i0,i1mid,i2mid);
                fprintf(out2D,"%.16e %.16e %.16e %.16e %.16e %.16e %.16e %.16e %.16e %.16e %.16e %.16e %.16e %.16e\n",
                        xx[0][i0],
                        auxevol_gfs[IDX4ptS(BU0GF,idx)],auxevol_gfs[IDX4ptS(BU1GF,idx)],auxevol_gfs[IDX4ptS(BU2GF,idx)],
                        y_n_gfs[IDX4ptS(AD0GF,idx)],y_n_gfs[IDX4ptS(AD1GF,idx)],y_n_gfs[IDX4ptS(AD2GF,idx)],
                        y_n_gfs[IDX4ptS(STILDED0GF,idx)],y_n_gfs[IDX4ptS(STILDED1GF,idx)],y_n_gfs[IDX4ptS(STILDED2GF,idx)],
                        auxevol_gfs[IDX4ptS(VALENCIAVU0GF,idx)],auxevol_gfs[IDX4ptS(VALENCIAVU1GF,idx)],auxevol_gfs[IDX4ptS(VALENCIAVU2GF,idx)],
                        y_n_gfs[IDX4ptS(PSI6PHIGF,idx)]);
            }
            fclose(out2D);

            set_initial_spacetime_metric_data(&params,xx,auxevol_gfs_exact);
            initial_data(&params,xx,auxevol_gfs_exact,evol_gfs_exact);
            // Fill in  the remaining quantities
            driver_A_to_B(&params,evol_gfs_exact,auxevol_gfs_exact);
            GiRaFFE_NRPy_prims_to_cons(&params,auxevol_gfs_exact,evol_gfs_exact);

            sprintf(filename,"out%d-%08d_exact.txt",Nxx0,n);
            FILE *out2D_exact = fopen(filename, "w");
            for(int i0=0;i0<Nxx_plus_2NGHOSTS0;i0++) {
                const int idx = IDX3S(i0,i1mid,i2mid);
                fprintf(out2D_exact,"%.16e %.16e %.16e %.16e %.16e %.16e %.16e %.16e %.16e %.16e %.16e %.16e %.16e %.16e\n",
                        xx[0][i0],
                        auxevol_gfs_exact[IDX4ptS(BU0GF,idx)],auxevol_gfs_exact[IDX4ptS(BU1GF,idx)],auxevol_gfs_exact[IDX4ptS(BU2GF,idx)],
                        evol_gfs_exact[IDX4ptS(AD0GF,idx)],evol_gfs_exact[IDX4ptS(AD1GF,idx)],evol_gfs_exact[IDX4ptS(AD2GF,idx)],
                        evol_gfs_exact[IDX4ptS(STILDED0GF,idx)],evol_gfs_exact[IDX4ptS(STILDED1GF,idx)],evol_gfs_exact[IDX4ptS(STILDED2GF,idx)],
                        auxevol_gfs_exact[IDX4ptS(VALENCIAVU0GF,idx)],auxevol_gfs_exact[IDX4ptS(VALENCIAVU1GF,idx)],auxevol_gfs_exact[IDX4ptS(VALENCIAVU2GF,idx)],
                        evol_gfs_exact[IDX4ptS(PSI6PHIGF,idx)]);
            }
            fclose(out2D_exact);

        }

        // Step 3: Evolve scalar wave initial data forward in time using Method of Lines with RK4 algorithm,
        //         applying quadratic extrapolation outer boundary conditions.
        // Step 3.b: Step forward one timestep (t -> t+dt) in time using
        //           chosen RK-like MoL timestepping algorithm
#include "MoLtimestepping/RK_MoL.h"
    } // End main loop to progress forward in time.

    // Step 4: Free all allocated memory
#include "MoLtimestepping/RK_Free_Memory.h"
    free(auxevol_gfs);
    free(auxevol_gfs_exact);
    free(evol_gfs_exact);
    for(int i=0;i<3;i++) free(xx[i]);
    return 0;
}Writing GiRaFFE_unstaggered_standalone_Ccodes//GiRaFFE_NRPy_standalone.c
cmd.C_compile(os.path.join(Ccodesdir,"GiRaFFE_NRPy_standalone.c"),
              os.path.join(Ccodesdir,"output","GiRaFFE_NRPy_standalone"),compile_mode="safe")
# !gcc -g -O2 -fopenmp GiRaFFE_standalone_Ccodes/GiRaFFE_NRPy_standalone.c -o GiRaFFE_NRPy_standalone -lm

# Change to output directory
os.chdir(outdir)
# Clean up existing output files
cmd.delete_existing_files("out*.txt")
cmd.delete_existing_files("out*.png")
# cmd.Execute(os.path.join(Ccodesdir,"output","GiRaFFE_NRPy_standalone"), "640 16 16", os.path.join(outdir,"out640.txt"))
cmd.Execute("GiRaFFE_NRPy_standalone", "64 64 64","out64.txt")
# cmd.Execute("GiRaFFE_NRPy_standalone", "239 15 15","out239.txt")
# !OMP_NUM_THREADS=1 valgrind --track-origins=yes -v ./GiRaFFE_NRPy_standalone 1280 32 32
# Return to root directory
os.chdir(os.path.join("../../"))Compiling executable...
(EXEC): Executing `gcc -std=gnu99 -O2 -g -fopenmp GiRaFFE_unstaggered_standalone_Ccodes/GiRaFFE_NRPy_standalone.c -o GiRaFFE_unstaggered_standalone_Ccodes/output/GiRaFFE_NRPy_standalone -lm`...
(BENCH): Finished executing in 2.4169042110443115 seconds.
Finished compilation.
(EXEC): Executing `taskset -c 0,1,2,3 ./GiRaFFE_NRPy_standalone 64 64 64`...
(BENCH): Finished executing in 13.84980583190918 seconds.

Now, we will load the data generated by the simulation and plot it in order to test for convergence. _____no_output_____

import numpy as np
import matplotlib.pyplot as plt

Data_numer = np.loadtxt(os.path.join(Ccodesdir,"output","out64-00000020.txt"))
# Data_num_2 = np.loadtxt(os.path.join("GiRaFFE_standalone_Ccodes","output","out239-00000080.txt"))
# Data_old = np.loadtxt("/home/penelson/OldCactus/Cactus/exe/ABE-GiRaFFEfood_1D_AlfvenWave/giraffe-grmhd_primitives_bi.x.asc")
# Data_o_2 = np.loadtxt("/home/penelson/OldCactus/Cactus/exe/ABE-GiRaFFEfood_1D_AlfvenWave_2/giraffe-grmhd_primitives_bi.x.asc")
# Data_numer = Data_old[5000:5125,11:15] # The column range is chosen for compatibility with the plotting script.
# Data_num_2 = Data_o_2[19600:19845,11:15] # The column range is chosen for compatibility with the plotting script.
# Data_exact = np.loadtxt(os.path.join(Ccodesdir,"output","out64-00000020_exact.txt"))
Data_exact = np.loadtxt(os.path.join(Ccodesdir,"output","out64-00000020.txt"))
# Data_exa_2 = np.loadtxt(os.path.join("GiRaFFE_standalone_Ccodes","output","out239-00000080_exact.txt"))

predicted_order = 2.0
column = 3
plt.figure()
# # plt.plot(Data_exact[2:-2,0],np.log2(np.absolute((Data_numer[2:-2,column]-Data_exact[2:-2,column])/\
# #                                     (Data_num_2[2:-2:2,column]-Data_exa_2[2:-2:2,column]))),'.')
plt.plot(Data_exact[:,0],Data_exact[:,column])
plt.plot(Data_numer[:,0],Data_numer[:,column],'.')
# plt.xlim(-0.0,1.0)
# # plt.ylim(-1.0,5.0)
# # plt.ylim(-0.0005,0.0005)
# plt.xlabel("x")
# plt.ylabel("BU2")
plt.show()

# #          0   1     2     3     4     5     6     7          8          9          10            11            12             13
# labels = ["x","BU0","BU1","BU2","AD0","AD1","AD2","StildeD0","StildeD1","StildeD2","ValenciavU0","ValenciavU1","ValenciavU2", "psi6Phi"]
# old_files = ["",
#              "giraffe-grmhd_primitives_bi.x.asc","giraffe-grmhd_primitives_bi.x.asc","giraffe-grmhd_primitives_bi.x.asc",
# #              "giraffe-em_ax.x.asc","giraffe-em_ay.x.asc","giraffe-em_az.x.asc",
#              "cell_centered_Ai.txt","cell_centered_Ai.txt","cell_centered_Ai.txt",
#              "giraffe-grmhd_conservatives.x.asc","giraffe-grmhd_conservatives.x.asc","giraffe-grmhd_conservatives.x.asc",
#              "giraffe-grmhd_primitives_allbutbi.x.asc","giraffe-grmhd_primitives_allbutbi.x.asc","giraffe-grmhd_primitives_allbutbi.x.asc",
#              "giraffe-em_psi6phi.x.asc"]
# column = 5
# column_old = [0,12,13,14,0,1,2,12,13,14,12,13,14,12]
# old_path = "/home/penelson/OldCactus/Cactus/exe/ABE-GiRaFFEfood_1D_AlfvenWave"
# new_path = os.path.join("GiRaFFE_standalone_Ccodes","output")
# data_old = np.loadtxt(os.path.join(old_path,old_files[column]))
# # data_old = data_old[250:375,:]# Select only the second timestep
# # data_old = data_old[125:250,:]# Select only the first timestep
# # data_old = data_old[0:125,:]# Select only the zeroth timestep
# data_new = np.loadtxt(os.path.join(new_path,"out119-00000001.txt"))

# deltaA_old = data_old[125:250,:] - data_old[0:125,:]
# data_new_t0 = np.loadtxt(os.path.join(new_path,"out119-00000000.txt"))
# deltaA_new = data_new[:,:] - data_new_t0[:,:]

# plt.figure()
# # plt.plot(data_new[3:-3,0],data_new[3:-3,column]-data_old[3:-3,column_old[column]])
# # plt.plot(data_new[:,0],data_new[:,column]-((3*np.sin(5*np.pi*data_new[:,0]/np.sqrt(1 - (-0.5)**2))/20 + 23/20)*(data_new[:,0]/2 + np.sqrt(1 - (-0.5)**2)/20 + np.absolute(data_new[:,0] + np.sqrt(1 - (-0.5)**2)/10)/2)*(-1e-100/2 + data_new[:,0]/2 - np.sqrt(1 - (-0.5)**2)/20 - np.absolute(-1e-100 + data_new[:,0] - np.sqrt(1 - (-0.5)**2)/10)/2)/((-1e-100 + data_new[:,0] - np.sqrt(1 - (-0.5)**2)/10)*(1e-100 + data_new[:,0] + np.sqrt(1 - (-0.5)**2)/10)) + 13*(data_new[:,0]/2 - np.sqrt(1 - (-0.5)**2)/20 + np.absolute(data_new[:,0] - np.sqrt(1 - (-0.5)**2)/10)/2)/(10*(1e-100 + data_new[:,0] - np.sqrt(1 - (-0.5)**2)/10)) + (-1e-100/2 + data_new[:,0]/2 + np.sqrt(1 - (-0.5)**2)/20 - np.absolute(-1e-100 + data_new[:,0] + np.sqrt(1 - (-0.5)**2)/10)/2)/(-1e-100 + data_new[:,0] + np.sqrt(1 - (-0.5)**2)/10))/np.sqrt(1 - (-0.5)**2))
# # plt.plot(data_new[1:,0]-(data_new[0,0]-data_new[1,0])/2.0,(data_new[0:-1,column]+data_new[1:,column])/2,'.',label="GiRaFFE_NRPy+injected BU")
# # plt.plot(data_new[1:,0]-(data_new[0,0]-data_new[1,0])/2.0,data_old[1:,column_old[column]],label="old GiRaFFE")
# # -(data_old[0,9]-data_old[1,9])/2.0
# # plt.plot(data_new[3:-3,0],deltaA_new[3:-3,column],'.')
# plt.plot(data_new[3:-3,0],deltaA_old[3:-3,column_old[column]]-deltaA_new[3:-3,column])
# # plt.xlim(-0.1,0.1)
# # plt.ylim(-0.2,0.2)
# plt.legend()
# plt.xlabel(labels[0])
# plt.ylabel(labels[column])
# plt.show()
# # print(np.argmin(deltaA_old[3:-3,column_old[column]]-deltaA_new[3:-3,column]))_____no_output_____

This code will create an animation of the wave over time._____no_output_____

# import matplotlib.pyplot as plt
from matplotlib.pyplot import savefig
from IPython.display import HTML
import matplotlib.image as mgimg

import glob
import sys
from matplotlib import animation

cmd.delete_existing_files("out119-00*.png")
globby = glob.glob(os.path.join('GiRaFFE_standalone_Ccodes','output','out119-00*.txt'))
file_list = []
for x in sorted(globby):
    file_list.append(x)

number_of_files = int(len(file_list)/2)

for timestep in range(number_of_files):
    fig = plt.figure()
    numer_filename = file_list[2*timestep]
    exact_filename = file_list[2*timestep+1]
    Numer = np.loadtxt(numer_filename)
    Exact = np.loadtxt(exact_filename)

    plt.title("Alfven Wave")
    plt.xlabel("x")
    plt.ylabel("BU2")
    plt.xlim(-0.5,0.5)
    plt.ylim(1.0,1.7)

    plt.plot(Numer[3:-3,0],Numer[3:-3,3],'.',label="Numerical")
    plt.plot(Exact[3:-3,0],Exact[3:-3,3],label="Exact")
    plt.legend()
    savefig(numer_filename+".png",dpi=150)
    plt.close(fig)
    sys.stdout.write("%c[2K" % 27)
    sys.stdout.write("Processing file "+numer_filename+"\r")
    sys.stdout.flush()_____no_output_____## VISUALIZATION ANIMATION, PART 2: Combine PNGs to generate movie ##
# https://stackoverflow.com/questions/14908576/how-to-remove-frame-from-matplotlib-pyplot-figure-vs-matplotlib-figure-frame
# https://stackoverflow.com/questions/23176161/animating-pngs-in-matplotlib-using-artistanimation
# !rm -f GiRaFFE_NRPy-1D_tests.mp4
cmd.delete_existing_files("GiRaFFE_NRPy-1D_tests.mp4")

fig = plt.figure(frameon=False)
ax = fig.add_axes([0, 0, 1, 1])
ax.axis('off')

myimages = []

for i in range(number_of_files):
    img = mgimg.imread(file_list[2*i]+".png")
    imgplot = plt.imshow(img)
    myimages.append([imgplot])

ani = animation.ArtistAnimation(fig, myimages, interval=100,  repeat_delay=1000)
plt.close()
ani.save('GiRaFFE_NRPy-1D_tests.mp4', fps=5,dpi=150)_____no_output_____%%HTML
<video width="480" height="360" controls>
  <source src="GiRaFFE_NRPy-1D_tests.mp4" type="video/mp4">
</video>_____no_output_____import cmdline_helper as cmd    # NRPy+: Multi-platform Python command-line interface
cmd.output_Jupyter_notebook_to_LaTeXed_PDF("Tutorial-GiRaFFE_NRPy_Main_Driver",location_of_template_file=os.path.join(".."))Created Tutorial-GiRaFFE_NRPy_Main_Driver.tex, and compiled LaTeX file to
    PDF file Tutorial-GiRaFFE_NRPy_Main_Driver.pdf


# Notebook from litosly/thesis
Path: data exploration/.ipynb_checkpoints/Yelp Explanation-checkpoint.ipynb

from ast import literal_eval
from os import listdir
from os.path import isfile, join
from scipy.sparse import csr_matrix, load_npz, save_npz
from tqdm import tqdm
from sklearn.preprocessing import normalize

import seaborn as sns
import datetime
import json
import numpy as np
import pandas as pd
import time
import yaml
import scipy.sparse as sparse
from ast import literal_eval

# For Python2 this have to be done
from __future__ import division_____no_output_____

# Load Dataframe_____no_output_____

df = pd.read_csv('../../data/yelp/Data.csv')_____no_output_____

# Split Data_____no_output_____

def to_sparse_matrix(df, num_user, num_item, user_col, item_col, rating_col):

    dok = df[[user_col, item_col, rating_col]].copy()
    dok = dok.values
    dok = dok[dok[:, 2] > 0]
    shape = [num_user, num_item]

    return sparse.csr_matrix((dok[:, 2].astype(np.float32), (dok[:, 0], dok[:, 1])), shape=shape)

def leave_one_out_split(df, user_col, ratio, random_state=None):
    grouped = df.groupby(user_col, as_index=False)
    valid = grouped.apply(lambda x: x.sample(frac=ratio, random_state=random_state))
    train = df.loc[~df.index.isin([x[1] for x in valid.index])]
    return train, valid

def time_ordered_split(df, ratio, user_col = None, random_state=None):
    # Sort data based on timestamp
    argsort = np.argsort(df['timestamp'])
    df_ordered = df.reindex(argsort)
    train_offset = int((1-ratio)*len(df_ordered))

    train = df_ordered[:train_offset]
    valid = df_ordered[train_offset:]
    return train, valid

def main(enable_validation = False, time_ordered_split_en = True, implicit_en = False):
    df = pd.read_csv('../../data/yelp/' + 'Data.csv')

    num_users = df['UserIndex'].nunique()
    num_items = df['ItemIndex'].nunique()

    # Get timestamp
    date_time_df = df[['Day','Month','Year']]
    date_time_df.rename(columns={'Year': 'year', 'Month': 'day', 'Day':'month'}, inplace=True)
    date_time = pd.to_datetime(date_time_df)
    df['timestamp'] = date_time

    rating_col = 'rating'
    if implicit_en == True:
        rating_col = 'Binary'

    if time_ordered_split_en:
        df_train, df_test = time_ordered_split(df, 0.2)
    else:
        df_train, df_test = leave_one_out_split(df, 'UserIndex', 0.2, random_state=8292)

    if enable_validation:
        if time_ordered_split_en:
            df_train, df_valid = time_ordered_split(df_train, 0.2)
        else:
            df_train, df_valid = leave_one_out_split(df_train, 'UserIndex', 0.2, random_state=8292)

        # Clean empty rows
        df_valid = df_valid.dropna().reset_index(drop = True)

        # Save
        df_valid.to_csv('../../data/yelp/' + 'Valid.csv')
        R_valid = to_sparse_matrix(df_valid, num_users, num_items, 'UserIndex','ItemIndex', rating_col)
        sparse.save_npz('../../data/yelp/' + 'Rvalid.npz', R_valid)

    # Clean empty rows
    df_train = df_train.dropna().reset_index(drop = True)
    df_test = df_test.dropna().reset_index(drop = True)

    # Save
    df_train.to_csv('../../data/yelp/'  + 'Train.csv')
    R_train = to_sparse_matrix(df_train, num_users, num_items, 'UserIndex', 'ItemIndex', rating_col)
    sparse.save_npz('../../data/yelp/' + 'Rtrain.npz', R_train)

    df_test.to_csv('../../data/yelp/' + 'Test.csv')
    R_test = to_sparse_matrix(df_test, num_users, num_items, 'UserIndex', 'ItemIndex', rating_col)
    sparse.save_npz('../../data/yelp/' + 'Rtest.npz', R_test)

def date_to_timestamp(date, **not_used):
    dt = datetime.datetime.strptime(date, '%Y-%m-%d')
    return time.mktime(dt.timetuple())
_____no_output_____df.columns_____no_output_____main(enable_validation = True, time_ordered_split_en = False, implicit_en = True)_____no_output_____

# Load Data_____no_output_____

# Load Original Data
df_train = pd.read_csv('../../data/yelp/Train.csv')
df_valid = pd.read_csv('../../data/yelp/Valid.csv')
df_test = pd.read_csv('../../data/yelp/Test.csv')
keyphrases = pd.read_csv('../../data/yelp/KeyPhrases.csv')['Phrases'].tolist()_____no_output_____ItemIndex = pd.read_csv('../../data/yelp/ItemIndex.csv')
ItemIndex = ItemIndex.sort_values('ItemIndex').drop_duplicates(subset=['ItemIndex', 'business_id'])_____no_output_____# Load U-I Data
rtrain = load_npz("../../data/yelp/Rtrain.npz")
rvalid = load_npz("../../data/yelp/Rvalid.npz")
rtest = load_npz("../../data/yelp/Rtest.npz")_____no_output_____rtrain_____no_output_____len(keyphrases)_____no_output_____# Generate U_K and I_K
# For validation set
U_K = get_I_K(df_train, row_name = 'UserIndex', shape = (2473, 235))
I_K = get_I_K(df_train, row_name = 'ItemIndex', shape = (10282, 235))
# For test set
U_K_test = get_I_K(df_test, row_name = 'UserIndex', shape = (2473, 235))
I_K_test = get_I_K(df_test, row_name = 'ItemIndex', shape = (10282, 235))100%|███████████████████████████████████████████████████████████████████████| 102741/102741 [00:05<00:00, 18012.10it/s]
100%|███████████████████████████████████████████████████████████████████████| 102741/102741 [00:05<00:00, 18245.60it/s]
100%|█████████████████████████████████████████████████████████████████████████| 32133/32133 [00:01<00:00, 18143.99it/s]
100%|█████████████████████████████████████████████████████████████████████████| 32133/32133 [00:01<00:00, 18215.99it/s]
# Save
save_npz('../../data/yelp/U_K.npz',U_K)
save_npz('../../data/yelp/I_K.npz',I_K)
save_npz( '../../data/yelp/U_K_test.npz',U_K_test)
save_npz('../../data/yelp/I_K_test.npz',I_K_test)_____no_output_____# Load
U_K = load_npz('../../data/yelp/U_K.npz')
I_K = load_npz('../../data/yelp/I_K.npz')_____no_output_____U_K_____no_output_____# Models
from sklearn.metrics.pairwise import cosine_similarity
def train(matrix_train):
    similarity = cosine_similarity(X=matrix_train, Y=None, dense_output=True)
    return similarity

def get_I_K(df, row_name = 'ItemIndex', shape = (3668,75)):
    rows = []
    cols = []
    vals = []
    for i in tqdm(range(df.shape[0])):
        key_vector = literal_eval(df['keyVector'][i])
        rows.extend([df[row_name][i]]*len(key_vector)) ## Item index
        cols.extend(key_vector) ## Keyword Index
#         if binary:
        vals.extend(np.array([1]*len(key_vector)))
#         else:
#             vals.extend(arr[arr.nonzero()])
    return csr_matrix((vals, (rows, cols)), shape=shape)

def predict(matrix_train, k, similarity, item_similarity_en = False):
    """
    res = similarity * matrix_train    if item_similarity_en = False
    res = similarity * matrix_train.T  if item_similarity_en = True
    """
    prediction_scores = []

    if item_similarity_en:
        matrix_train = matrix_train.transpose()

    for user_index in tqdm(range(matrix_train.shape[0])):
        # Get user u's prediction scores to all users
        vector_u = similarity[user_index]

        # Get closest K neighbors excluding user u self
        similar_users = vector_u.argsort()[::-1][1:k+1]
        # Get neighbors similarity weights and ratings
        similar_users_weights = similarity[user_index][similar_users]
        similar_users_ratings = matrix_train[similar_users].toarray()

        prediction_scores_u = similar_users_ratings * similar_users_weights[:, np.newaxis]

        prediction_scores.append(np.sum(prediction_scores_u, axis=0))
    res = np.array(prediction_scores)

    if item_similarity_en:
        res = res.transpose()

    return res

def prediction(prediction_score, topK, matrix_Train):

    prediction = []

    for user_index in tqdm(range(matrix_Train.shape[0])):
        vector_u = prediction_score[user_index]
        vector_train = matrix_Train[user_index]
        if len(vector_train.nonzero()[0]) > 0:
            vector_predict = sub_routine(vector_u, vector_train, topK=topK)
        else:
            vector_predict = np.zeros(topK, dtype=np.float32)

        prediction.append(vector_predict)

    return np.vstack(prediction)

def sub_routine(vector_u, vector_train, topK=500):

    train_index = vector_train.nonzero()[1]

    vector_u = vector_u

    candidate_index = np.argpartition(-vector_u, topK+len(train_index))[:topK+len(train_index)]
    vector_u = candidate_index[vector_u[candidate_index].argsort()[::-1]]
    vector_u = np.delete(vector_u, np.isin(vector_u, train_index).nonzero()[0])

    return vector_u[:topK]
_____no_output_____# Evluation
def recallk(vector_true_dense, hits, **unused):
    hits = len(hits.nonzero()[0])
    return float(hits)/len(vector_true_dense)

def precisionk(vector_predict, hits, **unused):
    hits = len(hits.nonzero()[0])
    return float(hits)/len(vector_predict)

def average_precisionk(vector_predict, hits, **unused):
    precisions = np.cumsum(hits, dtype=np.float32)/range(1, len(vector_predict)+1)
    return np.mean(precisions)

def r_precision(vector_true_dense, vector_predict, **unused):
    vector_predict_short = vector_predict[:len(vector_true_dense)]
    hits = len(np.isin(vector_predict_short, vector_true_dense).nonzero()[0])
    return float(hits)/len(vector_true_dense)

def _dcg_support(size):
    arr = np.arange(1, size+1)+1
    return 1./np.log2(arr)

def ndcg(vector_true_dense, vector_predict, hits):
    idcg = np.sum(_dcg_support(len(vector_true_dense)))
    dcg_base = _dcg_support(len(vector_predict))
    dcg_base[np.logical_not(hits)] = 0
    dcg = np.sum(dcg_base)
    return dcg/idcg

def click(hits, **unused):
    first_hit = next((i for i, x in enumerate(hits) if x), None)
    if first_hit is None:
        return 5
    else:
        return first_hit/10

def evaluate(matrix_Predict, matrix_Test, metric_names =['R-Precision', 'NDCG', 'Precision', 'Recall', 'MAP'], atK = [5, 10, 15, 20, 50], analytical=False):
    """
    :param matrix_U: Latent representations of users, for LRecs it is RQ, for ALSs it is U
    :param matrix_V: Latent representations of items, for LRecs it is Q, for ALSs it is V
    :param matrix_Train: Rating matrix for training, features.
    :param matrix_Test: Rating matrix for evaluation, true labels.
    :param k: Top K retrieval
    :param metric_names: Evaluation metrics
    :return:
    """
    global_metrics = {
        "R-Precision": r_precision,
        "NDCG": ndcg,
        "Clicks": click
    }

    local_metrics = {
        "Precision": precisionk,
        "Recall": recallk,
        "MAP": average_precisionk
    }

    output = dict()

    num_users = matrix_Predict.shape[0]

    for k in atK:

        local_metric_names = list(set(metric_names).intersection(local_metrics.keys()))
        results = {name: [] for name in local_metric_names}
        topK_Predict = matrix_Predict[:, :k]

        for user_index in tqdm(range(topK_Predict.shape[0])):
            vector_predict = topK_Predict[user_index]
            if len(vector_predict.nonzero()[0]) > 0:
                vector_true = matrix_Test[user_index]
                vector_true_dense = vector_true.nonzero()[1]
                hits = np.isin(vector_predict, vector_true_dense)

                if vector_true_dense.size > 0:
                    for name in local_metric_names:
                        results[name].append(local_metrics[name](vector_true_dense=vector_true_dense,
                                                                 vector_predict=vector_predict,
                                                                 hits=hits))

        results_summary = dict()
        if analytical:
            for name in local_metric_names:
                results_summary['{0}@{1}'.format(name, k)] = results[name]
        else:
            for name in local_metric_names:
                results_summary['{0}@{1}'.format(name, k)] = (np.average(results[name]),
                                                              1.96*np.std(results[name])/np.sqrt(num_users))
        output.update(results_summary)

    global_metric_names = list(set(metric_names).intersection(global_metrics.keys()))
    results = {name: [] for name in global_metric_names}

    topK_Predict = matrix_Predict[:]

    for user_index in tqdm(range(topK_Predict.shape[0])):
        vector_predict = topK_Predict[user_index]

        if len(vector_predict.nonzero()[0]) > 0:
            vector_true = matrix_Test[user_index]
            vector_true_dense = vector_true.nonzero()[1]
            hits = np.isin(vector_predict, vector_true_dense)

            # if user_index == 1:
            #     import ipdb;
            #     ipdb.set_trace()

            if vector_true_dense.size > 0:
                for name in global_metric_names:
                    results[name].append(global_metrics[name](vector_true_dense=vector_true_dense,
                                                              vector_predict=vector_predict,
                                                              hits=hits))

    results_summary = dict()
    if analytical:
        for name in global_metric_names:
            results_summary[name] = results[name]
    else:
        for name in global_metric_names:
            results_summary[name] = (np.average(results[name]), 1.96*np.std(results[name])/np.sqrt(num_users))
    output.update(results_summary)

    return output

def explain_evaluate(matrix_Predict, matrix_Test, metric_names =['R-Precision', 'NDCG', 'Precision', 'Recall', 'MAP'], atK = [5, 10, 15, 20, 50], analytical=False):
    """
    :param matrix_U: Latent representations of users, for LRecs it is RQ, for ALSs it is U
    :param matrix_V: Latent representations of items, for LRecs it is Q, for ALSs it is V
    :param matrix_Train: Rating matrix for training, features.
    :param matrix_Test: Rating matrix for evaluation, true labels.
    :param k: Top K retrieval
    :param metric_names: Evaluation metrics
    :return:
    """
    global_metrics = {
        "R-Precision": r_precision,
        "NDCG": ndcg,
        "Clicks": click
    }

    local_metrics = {
        "Precision": precisionk,
        "Recall": recallk,
        "MAP": average_precisionk
    }

    output = dict()

    num_users = matrix_Predict.shape[0]

    for k in atK:

        local_metric_names = list(set(metric_names).intersection(local_metrics.keys()))
        results = {name: [] for name in local_metric_names}
        topK_Predict = matrix_Predict[:, :k]

        for user_index in tqdm(range(topK_Predict.shape[0])):
            vector_predict = topK_Predict[user_index]
            if len(vector_predict.nonzero()[0]) > 0:
#                 vector_true = matrix_Test[user_index]
                vector_true = np.ravel(matrix_Test.todense()[0])
                vector_true_dense = np.argsort(vector_true)[::-1][:k]
#                 vector_true_dense = vector_true.nonzero()[1]
                hits = np.isin(vector_predict, vector_true_dense)

                if vector_true_dense.size > 0:
                    for name in local_metric_names:
                        results[name].append(local_metrics[name](vector_true_dense=vector_true_dense,
                                                                 vector_predict=vector_predict,
                                                                 hits=hits))

        results_summary = dict()
        if analytical:
            for name in local_metric_names:
                results_summary['{0}@{1}'.format(name, k)] = results[name]
        else:
            for name in local_metric_names:
                results_summary['{0}@{1}'.format(name, k)] = (np.average(results[name]),
                                                              1.96*np.std(results[name])/np.sqrt(num_users))
        output.update(results_summary)

    global_metric_names = list(set(metric_names).intersection(global_metrics.keys()))
    results = {name: [] for name in global_metric_names}

    topK_Predict = matrix_Predict[:]

    for user_index in tqdm(range(topK_Predict.shape[0])):
        vector_predict = topK_Predict[user_index]

        if len(vector_predict.nonzero()[0]) > 0:
#             vector_true = matrix_Test[user_index]
#             vector_true_dense = vector_true.nonzero()[1]
            vector_true = np.ravel(matrix_Test.todense()[0])
            vector_true_dense = np.argsort(vector_true)[::-1][:k]
            hits = np.isin(vector_predict, vector_true_dense)

            # if user_index == 1:
            #     import ipdb;
            #     ipdb.set_trace()

            if vector_true_dense.size > 0:
                for name in global_metric_names:
                    results[name].append(global_metrics[name](vector_true_dense=vector_true_dense,
                                                              vector_predict=vector_predict,
                                                              hits=hits))

    results_summary = dict()
    if analytical:
        for name in global_metric_names:
            results_summary[name] = results[name]
    else:
        for name in global_metric_names:
            results_summary[name] = (np.average(results[name]), 1.96*np.std(results[name])/np.sqrt(num_users))
    output.update(results_summary)

    return output_____no_output_____

# Explanation Model_____no_output_____

def explain(R,W2,k, model = "Cosine_similarity", item_similarity_en = True):
    """
    k: knn's hyperparameter k
    R: Rating Matrix with size U*I
    r_ij: observed rating with user i and item j
    s_ij: explanation vector with user i and item j
    Z: Joint Embedding/Latent Space with size U*U, generate r_ij and s_ij
    W2: Reconstruction matrix with size U*K
    S: Output explanation prediction matrix with size U*K (dense numpy ndarray)
    """
    Z = train(R) # Cosine similarity as default
    S = predict(W2, k, Z, item_similarity_en=item_similarity_en)
    if normalize_en == True:
        return normalize(S) # prediction score
    return S

def predict(matrix_train, k, similarity, item_similarity_en = False, normalize_en = False):
    """
    matrix_train: Rating Matrix with size U*I
    k: knn's hyperparameter k
    similarity: Joint Embedding/Latent Space with size U*U or I*I

    res = similarity * matrix_train    if item_similarity_en = False
    res = similarity * matrix_train.T  if item_similarity_en = True

    r_ij: observed rating with user i and item j
    s_ij: explanation vector with user i and item j
    """
    prediction_scores = []

    if item_similarity_en:
        matrix_train = matrix_train.transpose()

    for user_index in tqdm(range(matrix_train.shape[0])):
        # Get user u's prediction scores to all users
        vector_u = similarity[user_index]

        # Get closest K neighbors excluding user u self
        similar_users = vector_u.argsort()[::-1][1:k+1]
        # Get neighbors similarity weights and ratings
        similar_users_weights = similarity[user_index][similar_users]
        similar_users_ratings = matrix_train[similar_users].toarray()

        prediction_scores_u = similar_users_ratings * similar_users_weights[:, np.newaxis]

        prediction_scores.append(np.sum(prediction_scores_u, axis=0))
    res = np.array(prediction_scores)

    if item_similarity_en:
        res = res.transpose()
    if normalize_en:
        res = normalize(res)
    return res

def explain_prediction(prediction_score, topK, matrix_Train):
    """
    output prediction res of the  top K items/keyphrase/whatever
    """
    prediction = []

    for user_index in tqdm(range(matrix_Train.shape[0])):
        vector_u = prediction_score[user_index]
        vector_train = matrix_Train[user_index]
        if len(vector_train.nonzero()[0]) > 0:
            vector_predict = sub_routine_explain(vector_u, vector_train, topK=topK)
        else:
            vector_predict = np.zeros(topK, dtype=np.float32)

        prediction.append(vector_predict)
    return np.vstack(prediction)
#     return prediction

def sub_routine_explain(vector_u, vector_train, topK=30):
    """
    vector_u: predicted user vector
    vector_train: true user vector
    topK: top k items in vector
    vector_u: top k items predicted
    """
#     train_index = vector_train.nonzero()[1]
#     candidate_index = np.argpartition(-vector_u, topK+75)[:topK+75] #  10 here to make res consistent
#     candidate_index = np.argpartition(-vector_u, 74)[:topK]
#     vector_u = candidate_index[vector_u[candidate_index].argsort()[::-1]]
#     vector_u = np.delete(vector_u, np.isin(vector_u, train_index).nonzero()[0])

    candidate_index = np.argsort(vector_u)[::-1][:topK]
    return candidate_index

def predict_pilot_explanation(explanation_scores, top_keyphrase = 10):
    """
    Used for retrieve the 1st row of prediction scores, used for pilot test
    """
    explanation = []
    for explanation_score in tqdm(explanation_scores):
        explanation.append(np.argsort(explanation_score)[::-1][:top_keyphrase])
    return np.array(explanation)_____no_output_____def explain(R,W2,k):
    """
    R: Rating Matrix with size U*I
    r_ij: observed rating with user i and item j
    s_ij: explanation vector with user i and item j
    Z: Joint Embedding/Latent Space with size U*U, generate r_ij and s_ij
    W2: Reconstruction matrix with size U*K
    S: Output explanation prediction matrix with size U*K
    """
    Z = train(R)
    S = predict(W2, k, Z)
    return normalize(S)

# Evaluation Model
def recall(vector_true_dense, vector_true_predict):
    """
    The fraction of relevant instances that have been retrieved over the total amount of relevant instances
    The length of vector_true_dense and vector_true_predict has to be the same
    Out put recall
    """
    hits = len(np.isin(vector_true_predict, vector_true_dense).nonzero()[0])
    return float(hits)/len(vector_true_dense)

# Total Recall
def recall_all(true_matrix, predict_matrix, topK = 20):
    res = []
    for i in tqdm(range(len(Explanation_res1))):
        true_vector = np.argsort(np.ravel(normalize(true_matrix).todense()[i]))[-topK:]
        predict_vector = np.argsort(predict_matrix[i])[-topK:]
        res.append(recall(true_vector,predict_vector))
    return sum(res)/len(res)_____no_output_____

# Evaluation_____no_output_____### explicit with time-ordered split_____no_output_____

# ( U_I * I_U ) * U_K
similarity = train(rtrain)
explanation_scores = predict(U_K, 100, similarity)
# explanation =  predict_explanation(explanation_scores)
explanation = explain_prediction(explanation_scores, 100, U_K)100%|████████████████████████████████████████████████████████████████████████████| 2473/2473 [00:01<00:00, 1852.43it/s]
100%|████████████████████████████████████████████████████████████████████████████| 2473/2473 [00:00<00:00, 7316.57it/s]
explain_evaluate(explanation, U_K, atK=[5,10,20,50])100%|█████████████████████████████████████████████████████████████████████████████| 2473/2473 [00:02<00:00, 907.52it/s]
100%|█████████████████████████████████████████████████████████████████████████████| 2473/2473 [00:02<00:00, 912.55it/s]
100%|█████████████████████████████████████████████████████████████████████████████| 2473/2473 [00:02<00:00, 896.01it/s]
100%|█████████████████████████████████████████████████████████████████████████████| 2473/2473 [00:02<00:00, 876.64it/s]
100%|█████████████████████████████████████████████████████████████████████████████| 2473/2473 [00:03<00:00, 814.29it/s]

### implicit with time-ordered split_____no_output_____

# ( U_I * I_U ) * U_K
similarity = train(rtrain)
explanation_scores = predict(U_K, 100, similarity)
# explanation =  predict_explanation(explanation_scores)
explanation = explain_prediction(explanation_scores, 100, U_K)100%|████████████████████████████████████████████████████████████████████████████| 2473/2473 [00:01<00:00, 1967.38it/s]
100%|████████████████████████████████████████████████████████████████████████████| 2473/2473 [00:00<00:00, 7585.88it/s]
explain_evaluate(explanation, U_K, atK=[5,10,20,50])100%|█████████████████████████████████████████████████████████████████████████████| 2473/2473 [00:02<00:00, 905.53it/s]
100%|█████████████████████████████████████████████████████████████████████████████| 2473/2473 [00:02<00:00, 882.27it/s]
100%|█████████████████████████████████████████████████████████████████████████████| 2473/2473 [00:02<00:00, 844.89it/s]
100%|█████████████████████████████████████████████████████████████████████████████| 2473/2473 [00:02<00:00, 890.21it/s]
100%|█████████████████████████████████████████████████████████████████████████████| 2473/2473 [00:03<00:00, 813.49it/s]

### Implicit without time-ordered split_____no_output_____

# ( U_I * I_U ) * U_K
similarity = train(rtrain)
explanation_scores = predict(U_K, 100, similarity)
# explanation =  predict_explanation(explanation_scores)
explanation = explain_prediction(explanation_scores, 100, U_K)100%|████████████████████████████████████████████████████████████████████████████| 2343/2343 [00:01<00:00, 2128.07it/s]
100%|████████████████████████████████████████████████████████████████████████████| 2343/2343 [00:00<00:00, 7607.15it/s]
explain_evaluate(explanation, U_K_test, atK=[5,20,40]) 100%|█████████████████████████████████████████████████████████████████████████████| 2343/2343 [00:02<00:00, 939.08it/s]
100%|█████████████████████████████████████████████████████████████████████████████| 2343/2343 [00:02<00:00, 889.18it/s]
100%|█████████████████████████████████████████████████████████████████████████████| 2343/2343 [00:02<00:00, 913.09it/s]
100%|█████████████████████████████████████████████████████████████████████████████| 2343/2343 [00:02<00:00, 882.49it/s]

# Tuning _____no_output_____TODO_____no_output_____# Item-based Explanation_____no_output_____### explicit with time-oredered split _____no_output_____

similarity = train(np.transpose(rtrain))
explanation_scores = predict(I_K, 100, similarity)
# explanation =  predict_explanation(explanation_scores)
explanation = explain_prediction(explanation_scores, 70, I_K)100%|██████████████████████████████████████████████████████████████████████████| 10282/10282 [00:06<00:00, 1505.06it/s]
100%|██████████████████████████████████████████████████████████████████████████| 10282/10282 [00:01<00:00, 7333.81it/s]
explain_evaluate(explanation, I_K, atK=[5,10,30,50])100%|███████████████████████████████████████████████████████████████████████████| 10282/10282 [00:44<00:00, 232.73it/s]
100%|███████████████████████████████████████████████████████████████████████████| 10282/10282 [00:43<00:00, 236.53it/s]
100%|███████████████████████████████████████████████████████████████████████████| 10282/10282 [00:42<00:00, 244.69it/s]
100%|███████████████████████████████████████████████████████████████████████████| 10282/10282 [00:42<00:00, 243.45it/s]
100%|███████████████████████████████████████████████████████████████████████████| 10282/10282 [00:44<00:00, 233.05it/s]
explain_evaluate(explanation, I_K, atK=[5,10,30,50])100%|███████████████████████████████████████████████████████████████████████████| 10282/10282 [00:40<00:00, 252.01it/s]
100%|███████████████████████████████████████████████████████████████████████████| 10282/10282 [00:42<00:00, 242.54it/s]
100%|███████████████████████████████████████████████████████████████████████████| 10282/10282 [00:43<00:00, 235.80it/s]
100%|███████████████████████████████████████████████████████████████████████████| 10282/10282 [00:44<00:00, 233.16it/s]
100%|███████████████████████████████████████████████████████████████████████████| 10282/10282 [00:45<00:00, 224.58it/s]

### implicit without time-oredered split_____no_output_____

similarity = train(np.transpose(rtrain))
explanation_scores = predict(I_K, 100, similarity)
# explanation =  predict_explanation(explanation_scores)
explanation = explain_prediction(explanation_scores, 70, I_K)100%|██████████████████████████████████████████████████████████████████████████| 10282/10282 [00:05<00:00, 1837.71it/s]
100%|██████████████████████████████████████████████████████████████████████████| 10282/10282 [00:01<00:00, 7765.86it/s]
explain_evaluate(explanation, I_K_test, atK=[5,10,30,50]) 100%|███████████████████████████████████████████████████████████████████████████| 10282/10282 [00:40<00:00, 251.24it/s]
100%|███████████████████████████████████████████████████████████████████████████| 10282/10282 [00:40<00:00, 256.84it/s]
100%|███████████████████████████████████████████████████████████████████████████| 10282/10282 [00:40<00:00, 255.53it/s]
100%|███████████████████████████████████████████████████████████████████████████| 10282/10282 [00:39<00:00, 261.03it/s]
100%|███████████████████████████████████████████████████████████████████████████| 10282/10282 [00:40<00:00, 251.50it/s]

# Understand item-based Explanation_____no_output_____## Get restaurant names correspond to ItemIndex_____no_output_____

def get_business_df(path = "../../data/yelp/business.json" ):
    with open(path) as json_file:
        data = json_file.readlines()
        data = list(map(json.loads, data))
    df = pd.DataFrame(data)

    return df

def get_restaurant_info(business_df, business_id, name = True, review_count = True, stars = True ):
    output_list = {}
    row_idx = int(business_df.index[business_df['business_id'] == business_id].tolist()[0])
    if name == True:
        output_list['name'] = business_df['name'][row_idx].encode('utf-8').strip()
    if review_count == True:
        output_list['review_count'] = business_df['review_count'][row_idx]
    if stars == True:
        output_list['stars'] = business_df['stars'][row_idx]
    return output_list

def get_businessid_from_Itemindex(ItemIndex_list, itemindex):
    return ItemIndex_list['business_id'].tolist()[itemindex]
_____no_output_____business_df = get_business_df()_____no_output_____get_businessid_from_Itemindex(ItemIndex, 8010)_____no_output_____get_restaurant_info(business_df, 'l_uAw0K2lkOsyVJATcnwsA')_____no_output_____rtrain_____no_output_____

## Find specific Restaurant_____no_output_____

x = ['sushi on bloor' in business_df['name'][i].lower() for i in range(len(business_df))]
for i in np.array(x).nonzero():
    print business_df['name'][i]35611     Sushi On Bloor
114653    Sushi On Bloor
Name: name, dtype: object
x = ['mafia' in business_df['name'][i].lower() for i in range(len(business_df))]
for i in np.array(x).nonzero():
    print business_df['name'][i]7031                       Spicy Mafia
28409                 Caricature Mafia
51959               Mafia Mike's Pizza
113184    Cakefacemafia Brows & Beauty
119974                     Spicy Mafia
140192                    Mafia Mike's
156211                  La'Bella MAFIA
180250              Mafia Mike's PIzza
Name: name, dtype: object
x = ['crown prince' in business_df['name'][i].lower() for i in range(len(business_df))]
for i in np.array(x).nonzero():
    print business_df['name'][i]74627     Crown Prince Fine Dining & Banquet
106186            Crown Princess Fine Dining
Name: name, dtype: object
x = ['red lobster' in business_df['name'][i].lower() for i in range(len(business_df))]
for i in np.array(x).nonzero():
    print business_df['name'][i]4501                  Red Lobster
9494                  Red Lobster
9599                  Red Lobster
11793                 Red Lobster
15146                 Red Lobster
21612                 Red Lobster
25103                 Red Lobster
37750                 Red Lobster
39311                 Red Lobster
42883                 Red Lobster
46058                 Red Lobster
46742                 Red Lobster
51418                 Red Lobster
51780                 Red Lobster
52374                 Red Lobster
56591                 Red Lobster
66201                 Red Lobster
68260                 Red Lobster
70318                 Red Lobster
79983                 Red Lobster
105382                Red Lobster
108167                Red Lobster
112307                Red Lobster
115861                Red Lobster
119864                Red Lobster
126004                Red Lobster
134059                Red Lobster
140933                Red Lobster
145037                Red Lobster
154559                Red Lobster
159619                Red Lobster
161507                Red Lobster
162753                Red Lobster
163962                Red Lobster
167619    Red Lobster Restaurants
176109                Red Lobster
179596                Red Lobster
179732                Red Lobster
184775                Red Lobster
185524                Red Lobster
185736                Red Lobster
188347                Red Lobster
Name: name, dtype: object
x = ['queen and beaver' in business_df['name'][i].lower() for i in range(len(business_df))]
for i in np.array(x).nonzero():
    print business_df['name'][i]73128    The Queen And Beaver Public House
Name: name, dtype: object
x = ['miku' in business_df['name'][i].lower() for i in range(len(business_df))]
for i in np.array(x).nonzero():
    print business_df['name'][i]107476                            Miku
116702    Mikush Home Appliance Center
Name: name, dtype: object

## If the restaurant exists in df_____no_output_____

business_df['business_id'][73128]_____no_output_____get_itemindex_from_business_id(ItemIndex, 'qaNt4vtVdge_S68DVjw5Jg')_____no_output_____len(np.where(df.ItemIndex == 273)[0])_____no_output_____

# Compare Prediction with Ground Truth_____no_output_____

def get_itemindex_from_business_id(ItemIndex_list, business_id):
    business_id_list = ItemIndex_list['business_id'].tolist()
    return business_id_list.index(business_id)_____no_output_____

## Spicy Mafia (fewer reviews Chinese food)_____no_output_____

print get_restaurant_info(business_df, 'l_uAw0K2lkOsyVJATcnwsA')
print get_itemindex_from_business_id(ItemIndex, 'l_uAw0K2lkOsyVJATcnwsA'){'review_count': 9, 'name': 'Spicy Mafia', 'stars': 3.0}
8010
### Predicted
test_list = list(map(int, explanation[8010]))
np.array(keyphrases)[test_list]_____no_output_____print '[%s]' % ', '.join(map(str, np.array(keyphrases)[test_list][:20]))[noodle, wait, thai, beef, tea, pork, mall, chicken, soup, spicy, quick, fresh, curry, pot, meat, lunch, busy, friendly, egg, dinner]
### Ground Truth
np.array(keyphrases)[np.argsort(np.ravel(I_K[8010].todense()))[::-1][:70]]_____no_output_____print '[%s]' % ', '.join(map(str, np.array(keyphrases)[np.argsort(np.ravel(I_K[8010].todense()))[::-1][:20]]))[tomato, soup, meat, noodle, pork, fish, egg, busy, pot, tart, lunch, tofu, juice, avocado, apple, strawberry, pop, lettuce, miso, skewer]
# number of Hit
len(np.where([i in np.array(keyphrases)[np.argsort(np.ravel(I_K[8010].todense()))[::-1][:70]] for i in np.array(keyphrases)[test_list]])[0])_____no_output_____

## Crown Princess Fine Dining (many reviews Chinese food)_____no_output_____

print get_restaurant_info(business_df, 'ovlWOSKVjGecnaPuZLv_OQ')
print get_itemindex_from_business_id(ItemIndex, 'ovlWOSKVjGecnaPuZLv_OQ'){'review_count': 235, 'name': 'Crown Princess Fine Dining', 'stars': 3.5}
8517
### Predicted
test_list = list(map(int, explanation[8517]))
np.array(keyphrases)[test_list]_____no_output_____print '[%s]' % ', '.join(map(str, np.array(keyphrases)[test_list][:20]))[wait, thai, tea, mall, fresh, soup, pot, noodle, bar, chicken, beef, friendly, dumpling, dinner, curry, spicy, egg, pork, busy, tart]
### Ground Truth
np.array(keyphrases)[np.argsort(np.ravel(I_K[8517].todense()))[::-1][:70]]_____no_output_____print '[%s]' % ', '.join(map(str, np.array(keyphrases)[np.argsort(np.ravel(I_K[8517].todense()))[::-1][:20]]))[dim sum, tea, chinese, dumpling, shrimp, pork, downtown, wait, bun, pot, fried, dinner, chicken, congee, cake, egg, dessert, tart, bbq, mall]
# number of Hit
len(np.where([i in np.array(keyphrases)[np.argsort(np.ravel(I_K[8517].todense()))[::-1][:70]] for i in np.array(keyphrases)[test_list]])[0])_____no_output_____

## The Queen And Beaver Public House (many reviews dessert place)_____no_output_____

print get_restaurant_info(business_df, 'qaNt4vtVdge_S68DVjw5Jg')
print get_itemindex_from_business_id(ItemIndex, 'qaNt4vtVdge_S68DVjw5Jg'){'review_count': 355, 'name': 'The Queen And Beaver Public House', 'stars': 4.0}
8764
test_list = list(map(int, explanation[8764]))
np.array(keyphrases)[test_list]_____no_output_____print '[%s]' % ', '.join(map(str, np.array(keyphrases)[test_list][:20]))[mall, pot, wait, bar, friendly, beer, tea, fresh, chicken, pizza, beef, salad, sandwich, busy, quick, lunch, tart, meat, egg, store]
np.array(keyphrases)[np.argsort(np.ravel(I_K[8764].todense()))[::-1][:70]]_____no_output_____print '[%s]' % ', '.join(map(str, np.array(keyphrases)[np.argsort(np.ravel(I_K[8764].todense()))[::-1][:20]]))[pub, beer, dinner, pot, bar, chip, burger, mall, fish, egg, cozy, tea, wait, fry, fresh, beef, brunch, cocktail, salt, cheese]
# number of Hit
len(np.where([i in np.array(keyphrases)[np.argsort(np.ravel(I_K[8764].todense()))[::-1][:70]] for i in np.array(keyphrases)[test_list]])[0])_____no_output_____

## Red Lobster_____no_output_____

print get_restaurant_info(business_df, 'hTdJAjSZtHWwqqh5cCeAfA')
print get_itemindex_from_business_id(ItemIndex, 'hTdJAjSZtHWwqqh5cCeAfA'){'review_count': 93, 'name': 'Red Lobster', 'stars': 3.0}
7312
# Predicted
test_list = list(map(int, explanation[7312]))
np.array(keyphrases)[test_list]_____no_output_____print '[%s]' % ', '.join(map(str, np.array(keyphrases)[test_list][:20]))[tea, chicken, mall, wait, friendly, fresh, fast, pot, quick, milk, clean, bubble, bubble tea, busy, rib, dessert, fry, pop, cake, lunch]
# Ground Truth
np.array(keyphrases)[np.argsort(np.ravel(I_K[7312].todense()))[::-1][:70]]_____no_output_____print '[%s]' % ', '.join(map(str, np.array(keyphrases)[np.argsort(np.ravel(I_K[7312].todense()))[::-1][:20]]))[lobster, wait, seafood, shrimp, friendly, fresh, mall, fish, dinner, pot, potato, tea, salad, dessert, fried, lunch, crispy, refill, dip, bar]
# number of Hits
len(np.where([i in np.array(keyphrases)[np.argsort(np.ravel(I_K[7312].todense()))[::-1][:70]] for i in np.array(keyphrases)[test_list]])[0])_____no_output_____

## Miku_____no_output_____

print get_restaurant_info(business_df, '0a2O150ytxrDjDzXNfRWkA')
print get_itemindex_from_business_id(ItemIndex, '0a2O150ytxrDjDzXNfRWkA'){'review_count': 604, 'name': 'Miku', 'stars': 4.0}
273
# Predicted
test_list = list(map(int, explanation[273]))
np.array(keyphrases)[test_list]_____no_output_____print '[%s]' % ', '.join(map(str, np.array(keyphrases)[test_list][:20]))[wait, mall, tea, fresh, cake, pot, egg, meat, pork, dessert, chicken, friendly, tart, dinner, cheese, soup, busy, ice cream, bar, fish]
# Ground Truth
np.array(keyphrases)[np.argsort(np.ravel(I_K[273].todense()))[::-1][:70]]_____no_output_____print '[%s]' % ', '.join(map(str, np.array(keyphrases)[np.argsort(np.ravel(I_K[273].todense()))[::-1][:20]]))[sushi, dessert, salmon, fish, tea, fresh, sashimi, green tea, cake, dinner, japanese, matcha, tart, lunch, ice cream, tuna, chocolate, wait, scallop, mall]
# number of Hits
len(np.where([i in np.array(keyphrases)[np.argsort(np.ravel(I_K[273].todense()))[::-1][:70]] for i in np.array(keyphrases)[test_list]])[0])_____no_output_____

# TF-IDF_____no_output_____## User-based TF-IDF_____no_output_____

def df(U_K):
    """
    Get the df for keyphrase matrix
    """
    DF = {}
    for i in range(U_K.shape[0]):
        vector_u = np.ravel(U_K[i].todense())
        for keyword_idx in range(U_K.shape[1]):
            keyword_frequency = vector_u[keyword_idx]
            if keyword_idx in DF:
                DF[keyword_idx] += keyword_frequency
            else:
                DF[keyword_idx] = keyword_frequency
    return DF

def keyword_popularity_matrix(U_K,df_uk):
    """
    return the U_K normalized by keyphrase frequency
    """
    df = normalize(np.array(df_uk.values()).reshape(1,-1))[0]
    for i in tqdm(range(U_K.shape[0])):
        U_K[i] = U_K[i]/df
#         vector_u = np.ravel(U_K[i].todense())
#         for entry in range(U_K.shape[1]):
#             U_K[i,entry] = U_K[i,entry]/df[entry]
    return normalize(U_K)

def tf_idf(U_K,df_uk):
    """
    Change the U_K/I_K keyphrase matrix to TF-IDF.
    where:
    IDF = log(N/(DF+1))
    TF_IDF = (1+log(TF))* IDF
    """
    for i in tqdm(range(U_K.shape[0])):
        vector_u = np.ravel(U_K[i].todense())
        words_count = np.sum(vector_u)

        for entry in range(U_K.shape[1]):
            if U_K[i,entry] == 0:
                continue
            else:
                tf = U_K[i,entry]/words_count
                df = df_uk[entry]
                idf = np.log(U_K.shape[0]/(df+1))
                tf_idf = (1+ np.log(tf))*idf
                U_K[i,entry] = tf_idf
    return normalize(U_K)_____no_output_____# Load
U_K = load_npz('../../data/beer/U_K.npz')
I_K = load_npz('../../data/beer/I_K.npz')
U_K_test = load_npz('../../data/beer/U_K_test.npz')
I_K_test = load_npz('../../data/beer/I_K_test.npz')_____no_output_____df_uk = df(U_K)_____no_output_____U_K_tfidf = tf_idf(U_K, df_uk)_____no_output_____similarity = train(rtrain)
explanation_scores = predict(U_K_tfidf, 100, similarity)
# explanation =  predict_explanation(explanation_scores)
explanation = explain_prediction(explanation_scores, 70, U_K_tfidf)_____no_output_____explain_evaluate(explanation, U_K_test_tfidf, atK=[5,10,20])_____no_output_____

## Item-based TF-IDF_____no_output_____# Create Synthetic User_____no_output_____

# Get top_items items associated with keyphrase_ids
def item_associated_with_keyphrase(I_K, keyphrase_ids, top_items = 100):
    """
    I_K: Item Keyphrase Matrix
    Keyphrase_ids: top items described by keyphrase_ids
    top_n: how many top items described by the keyphrases will be output
    output item list (unique)
    """
    res = []
    for keyphrase_id in keyphrase_ids:
        res.append(np.argsort(np.ravel(I_K.todense()[:,keyphrase_id]))[::-1][:top_items])
    return np.unique(res)

# Modify U_U latent Space from U_I
def modify_user_preference(U_I, items, user_id = 0):
    """
    TODO: Fix the function s.t. it will not modify the initial matrix
    """
    U_I[user_id,:] = 0
    for i in items:
        U_I[0,i] = 1
    return U_I

def clear_user_keyphrase(U_K, user_id = 0):
    U_K[user_id,:] = 0

def explain_synthetic_user(rtrain, U_K, I_K, keyphrases, top_keyphrase = 20, user_id = 0, k = 100, top_items = 100, **Not_used):
    """

    """
    items = item_associated_with_keyphrase(I_K, keyphrases, top_items = top_items) # 8 is coffee
    U_I = modify_user_preference(rtrain, items, user_id=user_id)
    modified_U_U = train(U_I)
    U_K[user_id, :] = 0
    synthetic_user_keyphrase = normalize(predict(U_K, k, modified_U_U))[user_id]
    return np.argsort(synthetic_user_keyphrase)[::-1][:top_keyphrase]

def modify_user_keyphrase(U_K, keyphrase_ids, normalization = True, keyval = 1, user_id = 0, **Not_Used):
    """
    Change all keyphrase_ids to some fixed number, all others to 0
    Return the U_K matrix with user_id row the synthetic user1
    """
    U_K[user_id,:] = 0
    for keyphrase_id in keyphrase_ids:
        U_K[user_id,keyphrase_id] = keyval
    if normalization == True:
        return normalize(U_K)
    return U_K_____no_output_____# Modify user preference matrix
items = item_associated_with_keyphrase(I_K, [0], top_items = 200) # 'chinese'
U_I = modify_user_preference(rtrain, items, user_id = 0)D:\Anaconda\envs\py27\lib\site-packages\ipykernel_launcher.py:19: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
# make synthetic user1's keyphrase preference all 0
clear_user_keyphrase(U_K, user_id = 0)D:\Anaconda\envs\py27\lib\site-packages\ipykernel_launcher.py:25: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
U_K_____no_output_____I_K_____no_output_____# Get latent user similarity embedding
modified_U_U = train(U_I)
# predict
explanation_scores1 = predict(U_K, 100, modified_U_U)
explanation1 =  predict_pilot_explanation(explanation_scores1, top_keyphrase = 10)100%|████████████████████████████████████████████████████████████████████████████| 2343/2343 [00:01<00:00, 2151.52it/s]
100%|███████████████████████████████████████████████████████████████████████████| 2343/2343 [00:00<00:00, 97624.32it/s]
explanation1[0]_____no_output_____def rank_in_prediction(rtrain, U_K, I_K, top_items = 200, keyphrase = 0,user_i = 0):
    """
    Get the rank for user_i with keyphrase
    TODO: modify so that no need to reload U_K,I_K
    """
    U_K = load_npz('../../data/yelp/U_K.npz')
    U_K = normalize(U_K)
    rtrain = load_npz("../../data/yelp/Rtrain.npz")

    # Modify user preference matrix
    items = item_associated_with_keyphrase(I_K, keyphrase, top_items = top_items) # 'raspberry'
    U_I = modify_user_preference(rtrain, items, user_id = 0)

    # make synthetic user1's keyphrase preference all 0
    clear_user_keyphrase(U_K, user_id = 0)

    # Get latent user similarity embedding
    modified_U_U = train(U_I)
    # predict
    explanation_scores1 = predict(U_K, 100, modified_U_U)
    explanation1 =  predict_pilot_explanation(explanation_scores1, top_keyphrase = 230)
    return list(explanation1[user_i]).index(keyphrase[0])

def evaluate_pilot_test(rtrain,U_K, I_K,keyphrase_list, top_items = 200, user_i = 0):
    # Get the average rank for user_i with keyphrase
    res1 = 0
    for i in range(75):
        a = rank_in_prediction(rtrain, U_K, I_K, top_items = top_items, keyphrase = [i],user_i = user_i)
        print "keyphrase", keyphrase_list[i], "'s rank is ", a
        res1+= a
    return res1/75_____no_output_____evaluate_pilot_test(rtrain,U_K,I_K,keyphrases)_____no_output_____

# Create Synthetic item_____no_output_____

# Get top_users items associated with keyphrase_ids
def users_with_keyphrase_preference(U_K, keyphrase_ids, top_users = 100, norm = True):
    """
    U_K: User Keyphrase Matrix
    Keyphrase_ids: top_users who like keyphrase_ids
    output item list (unique)
    """
    res = []
    if norm:
        U_K = normalize(U_K)
    try:
        for keyphrase_id in keyphrase_ids:
            res.append(np.argsort(np.ravel(U_K.todense()[:,keyphrase_id]))[::-1][:top_users])
    except:
        return np.argsort(np.ravel(U_K.todense()[:,keyphrase_ids]))[::-1][:top_users]
    return np.unique(res)

# Modify I_I latent Space from U_I
def modify_item_history(U_I, users, item_id = 0):
    """
    TODO: Fix the function s.t. it will not modify the initial matrix
    """
    U_I[:,item_id] = 0
    for i in users:
        U_I[i, item_id] = 1
    return normalize(U_I)

def clear_item_keyphrase(I_K, item_id = 0):
    I_K[item_id, :] = 0

def explain_synthetic_item(rtrain, U_K, I_K, keyphrases, top_keyphrase = 20, user_id = 0, k = 100, top_users = 100, **Not_used):
    """

    """
    items = item_associated_with_keyphrase(I_K, keyphrases, top_items = top_items) # 8 is coffee
    U_I = modify_user_preference(rtrain, items, user_id=user_id)
    modified_U_U = train(U_I)
    U_K[user_id, :] = 0
    synthetic_user_keyphrase = normalize(predict(U_K, k, modified_U_U))[user_id]
    return np.argsort(synthetic_user_keyphrase)[::-1][:top_keyphrase]

def modify_user_keyphrase(U_K, keyphrase_ids, normalization = True, keyval = 1, user_id = 0, **Not_Used):
    """
    Change all keyphrase_ids to some fixed number, all others to 0
    Return the U_K matrix with user_id row the synthetic user1
    """
    U_K[user_id,:] = 0
    for keyphrase_id in keyphrase_ids:
        U_K[user_id,keyphrase_id] = keyval
    if normalization == True:
        return normalize(U_K)
    return U_K_____no_output_____

## Prediction Pipeline_____no_output_____

U_K = load_npz('../../data/yelp/U_K.npz')
I_K = load_npz('../../data/yelp/I_K.npz')
U_K = normalize(U_K)
I_K = normalize(I_K)
rtrain = load_npz("../../data/yelp/Rtrain.npz")_____no_output_____# Modify user preference matrix
users = users_with_keyphrase_preference(U_K, 70, top_users = 100, norm = True) # 'raspberry'
U_I = modify_item_history(rtrain, users, item_id = 0)D:\Anaconda\envs\py27\lib\site-packages\ipykernel_launcher.py:23: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
# make synthetic item1's keyphrase to all 0
clear_item_keyphrase(I_K, item_id = 0)_____no_output_____# Get latent item similarity embedding
modified_I_I = train(np.transpose(U_I))
# predict
explanation_scores1 = predict(I_K, 100, modified_I_I)
explanation1 =  predict_pilot_explanation(explanation_scores1, top_keyphrase = 70)100%|████████████████████████████████████████████████████████████████████████████| 7456/7456 [00:03<00:00, 2184.59it/s]
100%|██████████████████████████████████████████████████████████████████████████| 7456/7456 [00:00<00:00, 112969.72it/s]
def rank_in_prediction_item(rtrain, U_K, I_K, top_users = 200, keyphrase = 70, item_i = 0):
    """
    Get the rank for user_i with keyphrase
    TODO: modify so that no need to reload U_K,I_K
    """
    U_K = load_npz('../../data/yelp/U_K.npz')
    I_K = load_npz('../../data/yelp/I_K.npz')
    U_K = normalize(U_K)
    I_K = normalize(I_K)
    rtrain = load_npz("../../data/yelp/Rtrain.npz")

    # Modify user preference matrix
    users = users_with_keyphrase_preference(U_K, keyphrase, top_users = top_users, norm = True) # 'raspberry'
    U_I = modify_item_history(rtrain, users, item_id = item_i)

    # make synthetic item1's keyphrase to all 0
    clear_item_keyphrase(I_K, item_id = item_i)

    # Get latent item similarity embedding
    modified_I_I = train(np.transpose(U_I))
    # predict
    explanation_scores1 = predict(I_K, 100, modified_I_I)
    explanation1 =  predict_pilot_explanation(explanation_scores1, top_keyphrase = 230)
    return list(explanation1[item_i]).index(keyphrase)

def evaluate_pilot_test_item(rtrain,U_K, I_K,keyphrase_list, top_users = 200, item_i = 0):
    # Get the average rank for item_i with keyphrase
    res1 = 0
    for i in range(75):
        a = rank_in_prediction_item(rtrain, U_K, I_K, top_users = top_users, keyphrase = i, item_i = item_i)
        print "keyphrase", keyphrase_list[i], "'s rank is ", a
        res1+= a
    return res1/75_____no_output_____rank_in_prediction_item(rtrain, U_K, I_K, top_users = 200, keyphrase = 10, item_i = 0)D:\Anaconda\envs\py27\lib\site-packages\ipykernel_launcher.py:23: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
D:\Anaconda\envs\py27\lib\site-packages\ipykernel_launcher.py:29: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
100%|████████████████████████████████████████████████████████████████████████████| 7456/7456 [00:03<00:00, 2163.04it/s]
100%|██████████████████████████████████████████████████████████████████████████| 7456/7456 [00:00<00:00, 112969.72it/s]
evaluate_pilot_test_item(rtrain,U_K, I_K,keyphrases, top_users = 200, item_i = 0)_____no_output_____


# Notebook from theislab/cellrank_reproducibility
Path: notebooks/suppl_fig_time_benchmark/analysis_notebooks/MK_2020-16-10_dynamo_UNUSED.ipynb

<code>
import utils.utilities as utilities
import cellrank as cr
import scanpy as sc
import numpy as np

from pathlib import Path

cr.logging.print_versions()Dynamo custom version (not broken fates): 0.95.1.dev4+bb86e69.dirty
cellrank==1.0.0-rc.12 scanpy==1.5.1 anndata==0.7.4 numpy==1.19.2 numba==0.51.2 scipy==1.5.2 pandas==1.1.2 scikit-learn==0.23.1 statsmodels==0.11.1 python-igraph==0.8.2 scvelo==0.1.26.dev302+g8f6764f.d20201002 pygam==0.8.0 matplotlib==3.3.0 seaborn==0.10.1
adata = sc.read(Path.home() / 'server_home/crb_bak/data/merged.h5ad')
del adata.layers['matrix']
del adata.layers['ambiguous']
adata_____no_output_____dfs = utilities.get_split(Path.home() / 'server_home/crb_bak/splits')
dfs.keys()_____no_output_____dst_dir = Path.home() / "server_home/analysis/dynamo.pickle"
dst_dir.parent.mkdir(exist_ok=True)
dst_dir_____no_output_____# this also needs the roots (not used below)
# we
utilities.benchmark_dynamo(adata, dfs, path=dst_dir,
                           dynamo_roots=Path.home() / "server_home/analysis/gpcca_bwd.pickle")Subsetting data to `10000`, split `0`.
Preprocessing and clustering
Calculating moments
Calculating dynamics
Dimnesionality reduction
Calculating velocities

</code>