# Notebook from tguillemLSST/eochar
Path: notebooks/CTE/CTE_diagnostic.ipynb

# Notebook : CTE_diagnostic

Goal : This Notebook will generate plots allowing to perform  a diagnostic for channels with a bad CTE .

Author : P.Antilogus

Version : 4th July 2019 13:30

    The data used are the PTC flats ( large data set at many fluxes), the study of the evolution of the first overscan pixels with the flux give an unique indication on :
    - is it a trap ? :  number of defered charges ~ a few e- ( 1 to 12) ~ constant for flux between 100 e- 10^4
    - is there issue in the overscan shape ( various plot related to the overscan shape including :
        - evolution of the overscan in function of the flux
        - comparison of the noise estimated with the overscan associated to the image area or to the non-image corner
        - direct plot of the overscan for all the fluxes

    The data for different runs can be overploted to see the evolution / differences
    The raft , dataset ( BNL , SLAC ..) has to be provided , and a subselection can be done
    on the run , devices and amplifiers .

    You can descide to only do the summary plots (for all amplifiers ) , or display the diagnostic plots
    for all or a fraction of the amplifiers . See box 3 for configuration parameters ( serial_polt ...)

Remark  :

     To provide a flux scale in e- a gain is needed , the one computed from the Fe55 data from the same run is used
     by doing a query of the eotest DB . If you haven't installed the needed software the gain will be set to 1
     for all channels ... but you can change this default value if you want  (0.7 is better for e2v and 0.9 for itl )
_____no_output_____

<code>
%run -i ../../python/lsst/eochar/frame_study.py
get_ipython().magic('matplotlib inline')_____no_output_____# CONFIGURATION FOR THE CURRENT EXECUTION  ========================================
# ---- raft and associated run ============ To be updated if needed
# 'RTM' :  full RTM name
# 'Data_location'  :  0 = BNL_RAFT_ROOT ,  1 = SLAC_RAFT_ROOT
# 'run' :  list of run , ex : ['9876','9874']  (remark : run is a string )   for all run of this raft : '*'
# 'sensor' :  list of sensor , ex ['S00','S01','S02']  ,  for all sensor ['S*']
# amplifier : list of amplifier , ex [1,2]  , for 1 to 16 you can use [-1] instead
#data_to_select={'RTM':'RTM-004','Data_location':1,'run':['*'],'sensor':['S*'],'amplifier':[-1]}
#data_to_select={'RTM':'RTM-004','Data_location':1,'run':['9012'],'sensor':['S01'],'amplifier':[4]}
#data_to_select={'RTM':'RTM-004','Data_location':1,'run':['9012'],'sensor':['S*'],'amplifier':[-1]}
data_to_select={'RTM':'RTM-017','Data_location':1,'run':['11166'],'sensor':['S*'],'amplifier':[-1]}
# Do we look for serial CTE ?
serial=True
# do we plot per amplifiers serial CTE diagnostic plots
serial_plot=True
# Do we look for // CTE ?
parallel=True
# do we plot per amplifiers // CTE diagnostic plots
parallel_plot=True
# default gain if eotest data not available ( 0.7 is better of e2v , 0.9 for itl )
default_gain=1._____no_output_____import numpy as np
import glob
try :
    from get_EO_analysis_results import get_EO_analysis_results
    from get_EO_analysis_files import get_EO_analysis_files
    from exploreFocalPlane import exploreFocalPlane
    from exploreRaft import exploreRaft
    from eTraveler.clientAPI.connection import Connection
    g = get_EO_analysis_results(db='Prod')
    eotest_db=True
except:
    print('No access to eotest DB , so the gain of all devices will be set to 1 ')
    eotest_db=False_____no_output_____# the list of super flat for a raft - run : BNL_RAFT_ROOT+raft+'/'+run+'/cte_raft/*/*/'+'Sxx'+'/*_superflat_high.fits'
##
count =!set | grep  .ncsa.illinois.edu | wc
if int(count[0].split()[0])> 1 :
    NCSA=True
else :
    NCSA=False
if NCSA :
    SLACmirror='/project/rgruendl/SLACmirror'
    # the list of super flat for a raft - run : BNL_RAFT_ROOT+raft+'/'+run+'/cte_raft/*/*/'+'Sxx'+'/*_superflat_high.fits'
    BNL_RAFT_ROOT=SLACmirror+'/BNLmirror/mirror/BNL-prod/prod/LCA-11021_RTM/LCA-11021_'
    #
    SLAC_RAFT_ROOT=SLACmirror+'/fs3/jh_archive/LCA-11021_RTM/LCA-11021_'
else :
    # the list of super flat for a raft - run : BNL_RAFT_ROOT+raft+'/'+run+'/cte_raft/*/*/'+'Sxx'+'/*_superflat_high.fits'
    BNL_RAFT_ROOT='/nfs/farm/g/lsst/u1/mirror/BNL-prod/prod/LCA-11021_RTM/LCA-11021_'
    #
    SLAC_RAFT_ROOT='/gpfs/slac/lsst/fs1/g/data/jobHarness/jh_archive/LCA-11021_RTM/LCA-11021_'
    #
    #
# order in which they are added to ROOT should follow what will be given to 'data_location'
ROOT=[BNL_RAFT_ROOT,SLAC_RAFT_ROOT]
_____no_output_____# how many run & sensor
sensor_dir=[]
run_loc=[]
# get all directory  for all sensor and all run
for run_cur in data_to_select['run'] :
    for sensor_cur  in data_to_select['sensor'] :
        root_dir=ROOT[data_to_select['Data_location']]+data_to_select['RTM']+'/'+run_cur+'/flat_pair_raft_acq/v0/*/'+sensor_cur
        run_loc+=glob.glob(root_dir)
# order them all
run_loc.sort()
# filter : keep only 1 proceesing for a given run and sensor (the first seen )
run_name_list=[]
sensor_list=[]
sensor_all=[]
sensor_dir=[]
for run_cur in run_loc :
    name_list=run_cur.split('/')
    run_number=name_list[-5]
    if not(run_number in run_name_list ) :
        # new run , let see which sensor are in it
        run_name_list.append(run_number)
        if len(sensor_all) > len(sensor_list) :
            sensor_list=sensor_all
        sensor_all=[]
    sensor_name=name_list[-1]
    if not(sensor_name in sensor_all) :
        sensor_all.append(sensor_name)
        # we keep the first processed data for this run and sensor
        sensor_dir.append({'dir':run_cur,'run':run_number,'sensor':sensor_name})
if len(sensor_all) > len(sensor_list) :
    sensor_list=sensor_all
sensor_list.sort()
#
number_of_run=len(run_name_list)
number_of_sensor=len(sensor_list)
#
if number_of_run < 1 :
    print('No data found for ',data_to_select)
    raise
#
txt='We will analyse the '
if serial :
    txt+='serial '
    if parallel :
        txt+='and parallel '
elif parallel :
    txt+='parallel '
else :
    print('None of the possible option (serial and/or parallel) have been selected ....')
    raise
print(txt,'CTE for ',number_of_run,' run(s) (',run_name_list,') from the raft ',data_to_select['RTM'])
#how many amplifiers
if -1 in data_to_select['amplifier'] :
    amplifier_list=np.array(range(1,17))
else :
    amplifier_list=np.array(data_to_select['amplifier'])
amplifier_list.sort()
number_of_amplifier=len(amplifier_list)
if number_of_amplifier ==16 :
    print('For the 16 amplifiers of ',number_of_sensor,' sensors (',sensor_list,')')
else :
    print('including ',number_of_sensor,' sensor(s) (',sensor_list,') and ',number_of_amplifier,'amplifier(s) (',amplifier_list,')')
# create the data table with the right dimension
if parallel :
    cte_p_data=np.zeros((number_of_run,number_of_sensor),dtype=np.object_)
if serial :
    cte_s_data=np.zeros((number_of_run,number_of_sensor),dtype=np.object_)
_____no_output_____# fill the cte data  and do the key diagnotic plots for each selected channels
for dir_cur in sensor_dir :
    irun=run_name_list.index(dir_cur['run'])
    iccd=sensor_list.index(dir_cur['sensor'])
    #
    all_file=glob.glob(dir_cur['dir']+'/*flat*flat*.fits')
    if len(all_file)<1 :
        continue
    print('Processing ',len(all_file),' flats for run',dir_cur['run'],' and sensor ',dir_cur['sensor'])
    #
    if eotest_db :
        raft_list, data = g.get_tests(site_type="I&T-Raft", test_type="gain", run =run_name_list[irun])  # get the data for I&T-Raft
        res = g.get_results(test_type='gain', data=data, device=raft_list[0])  # get the data for a raft
     # get the gain for this sensor :
    if eotest_db :
        sensor_id=[]
        for d in data['steps']['fe55_raft_analysis']['fe55_raft_analysis']:
            if 'slot' in d.keys():
                if d['slot']==sensor_list[iccd]:
                    sensor_id=d['sensor_id']
        if len(sensor_id)==0 :
                    all_gain=np.ones((16))*default_gain
        else :
            all_gain=np.array(res[sensor_id])
            # protection against badly measured gain : we accept only a gain within 15% of the median
            medgain=np.median(all_gain)
            if medgain < 0.5 :
                # sensor not operational or what ?
                print('Gain found in eotest for this sensor looks bad (median gain=',medgain,'), sensor dead  ? We set all gain to ',default_gain)
                all_gain=np.ones((16))*default_gain
            else :
                for igain in range(len(all_gain)):
                    if np.abs(all_gain[igain]-medgain)>.15 :
                        print('Gain channel ',igain+1,' found out of range ',all_gain[igain],' Channel could be dead or its a bad measurement . We set it to the median gain =',medgain)
                        all_gain[igain]=medgain
            print('sensor =',sensor_id)
    else :
        # if no DB info set the gain to 1
        sensor_id=''
        all_gain=np.ones((16))*default_gain
    #
    all_file.sort()
    file=Ifile(dirall=all_file,fkey={},Slow=False)
    if serial :
        cte_s_data[irun,iccd]=cte(all_file=file.all_file,gain=all_gain,serie=True)
    if parallel:
        cte_p_data[irun,iccd]=cte(all_file=file.all_file,gain=all_gain,serie=False)
    nf=0
    ccd=data_to_select['RTM']+','+run_name_list[irun]+':'+sensor_list[iccd]
    for ichan in range(number_of_amplifier)  :
        ch=amplifier_list[ichan]-1
        if serial and  serial_plot :
            cte_s_data[irun,iccd].plot_cte(ch=ch,ccd_name=ccd,nf=nf,on_screen=True)
        #
        if parallel and parallel_plot :
            cte_p_data[irun,iccd].plot_cte(ch=ch,ccd_name=ccd,nf=nf,on_screen=True)

_____no_output_____# Summary plots (cte vs flux ) for the # selected sensors , overlaping the # selected run
#
sel_color=['r','g','b','c','m','y','k']
nb_color=len(sel_color)
sel_mark=['<','>','*','^','v','+']
nb_mark=len(sel_mark)
#
yv=5.0e-6
yy=[yv,yv]
xx=[10.,300000.]
#
if serial :
    for iccd in range(number_of_sensor) :
        fig=plt.figure(figsize=[20,20])
        fig.suptitle('Serial CTE :'+data_to_select['RTM']+':'+sensor_list[iccd],fontsize=14)
        for ch in range(16):
            ax=fig.add_subplot(4,4,ch+1)
            for irun in range(number_of_run) :
                  plt.errorbar(cte_s_data[irun,iccd].cte_flux_s[ch,0:cte_s_data[irun,iccd].lmax[ch]],cte_s_data[irun,iccd].ylev[ch,0:cte_s_data[irun,iccd].lmax[ch]], yerr=cte_s_data[irun,iccd].ylev_std[ch,0:cte_s_data[irun,iccd].lmax[ch]],fmt=sel_mark[irun%nb_mark],color=sel_color[irun%nb_color],label=run_name_list[irun])
            #
            plt.plot(xx,yy,'g')
            label='channel %d  : <flux> in e-' % (ch+1)
            plt.xlabel(label)
            if ch%4 == 0 :
                plt.ylabel('1-CTE (serial CTE)')
            plt.xscale('log')
            plt.yscale('log')
            if (ch+1)%4==0 :
                plt.legend(bbox_to_anchor=(1.05, 1),loc=2, borderaxespad=0.)
            plt.xlim(xx[0],xx[1])
            plt.ylim(1e-8,5e-5)
        plt.show()
        plotfile=data_to_select['RTM']+':'+sensor_list[iccd]+'_serialCTE.png'
        fig.savefig(plotfile,bbox_inches='tight')
if parallel :
    for iccd in range(number_of_sensor) :
        fig=plt.figure(figsize=[20,20])
        fig.suptitle('// CTE :'+data_to_select['RTM']+':'+sensor_list[iccd],fontsize=14)
        for ch in range(16):
            ax=fig.add_subplot(4,4,ch+1)
            for irun in range(number_of_run) :
                  plt.errorbar(cte_p_data[irun,iccd].cte_flux_s[ch,0:cte_p_data[irun,iccd].lmax[ch]],cte_p_data[irun,iccd].ylev[ch,0:cte_p_data[irun,iccd].lmax[ch]], yerr=cte_p_data[irun,iccd].ylev_std[ch,0:cte_p_data[irun,iccd].lmax[ch]],fmt=sel_mark[irun%nb_mark],color=sel_color[irun%nb_color],label=run_name_list[irun])
            #
            plt.plot(xx,yy,'g')
            label='channel %d  : <flux> in e-' % (ch+1)
            plt.xlabel(label)
            if ch%4 == 0 :
                plt.ylabel('1-CTE (// CTE)')
            plt.xscale('log')
            plt.yscale('log')
            if (ch+1)%4==0 :
                plt.legend(bbox_to_anchor=(1.05, 1),loc=2, borderaxespad=0.)
            plt.xlim(xx[0],xx[1])
            plt.ylim(1e-8,5e-5)
        plt.show()
        plotfile=data_to_select['RTM']+':'+sensor_list[iccd]+'_parallelCTE.png'
        fig.savefig(plotfile,bbox_inches='tight')
_____no_output_____# Summary plots (cte vs flux ) for the # selected sensors , overlaping the # selected run
#
# colors for each channel setup
cmap=plt.get_cmap('gist_ncar')
colors=[cmap(j)[:3] for j in np.linspace(0,1,17)]
#
sel_color=['r','g','b','c','m','y','k']
nb_color=len(sel_color)
sel_mark=['<-','>-','*-','^-','v-','+-']
nb_mark=len(sel_mark)
#
yv=5.0e-6
yy=[yv,yv]
xx=[10.,300000.]
#
if serial :
    fig=plt.figure(figsize=[20,20])
    fig.suptitle('Serial CTE :'+data_to_select['RTM'],fontsize=14)
    for iccd in range(number_of_sensor) :
        ax=fig.add_subplot(3,3,iccd+1)
        for ch in range(16):
            for irun in range(number_of_run) :
                if ch==0 or irun==0 :
                    labelx='%d (%s)' % (ch+1,run_name_list[irun])
                    plt.errorbar(cte_s_data[irun,iccd].cte_flux_s[ch,0:cte_s_data[irun,iccd].lmax[ch]],cte_s_data[irun,iccd].ylev[ch,0:cte_s_data[irun,iccd].lmax[ch]], yerr=cte_s_data[irun,iccd].ylev_std[ch,0:cte_s_data[irun,iccd].lmax[ch]],fmt=sel_mark[irun%nb_mark],color=colors[ch],label=labelx)
                else :
                    plt.errorbar(cte_s_data[irun,iccd].cte_flux_s[ch,0:cte_s_data[irun,iccd].lmax[ch]],cte_s_data[irun,iccd].ylev[ch,0:cte_s_data[irun,iccd].lmax[ch]], yerr=cte_s_data[irun,iccd].ylev_std[ch,0:cte_s_data[irun,iccd].lmax[ch]],fmt=sel_mark[irun%nb_mark],color=colors[ch])
            #
        plt.plot(xx,yy,'g')
        label='CCD %s' % sensor_list[iccd]
        plt.xlabel(label)
        if iccd%3 == 0 :
            plt.ylabel('1-CTE (serial CTE)')
        plt.xscale('log')
        plt.yscale('log')
        if (iccd+1)%3==0 :
                plt.legend(bbox_to_anchor=(1.05, 1),loc=2, borderaxespad=0.)
        plt.xlim(xx[0],xx[1])
        plt.ylim(1e-8,5e-5)
    plt.show()
    plotfile=data_to_select['RTM']+':all_CCD_serialCTE.png'
    fig.savefig(plotfile,bbox_inches='tight')
if parallel :
    fig=plt.figure(figsize=[20,20])
    fig.suptitle('// CTE :'+data_to_select['RTM'],fontsize=14)
    for iccd in range(number_of_sensor) :
        ax=fig.add_subplot(3,3,iccd+1)
        for ch in range(16):
            for irun in range(number_of_run) :
                if ch==0 or irun==0 :
                    labelx='%d (%s)' % (ch+1,run_name_list[irun])
                    plt.errorbar(cte_p_data[irun,iccd].cte_flux_s[ch,0:cte_p_data[irun,iccd].lmax[ch]],cte_p_data[irun,iccd].ylev[ch,0:cte_p_data[irun,iccd].lmax[ch]], yerr=cte_p_data[irun,iccd].ylev_std[ch,0:cte_p_data[irun,iccd].lmax[ch]],fmt=sel_mark[irun%nb_mark],color=colors[ch],label=labelx)
                else :
                    plt.errorbar(cte_p_data[irun,iccd].cte_flux_s[ch,0:cte_p_data[irun,iccd].lmax[ch]],cte_p_data[irun,iccd].ylev[ch,0:cte_p_data[irun,iccd].lmax[ch]], yerr=cte_p_data[irun,iccd].ylev_std[ch,0:cte_p_data[irun,iccd].lmax[ch]],fmt=sel_mark[irun%nb_mark],color=colors[ch])
        #
        plt.plot(xx,yy,'g')
        label='CCD %s' % sensor_list[iccd]
        plt.xlabel(label)
        if iccd%3 == 0 :
            plt.ylabel('1-CTE (// CTE)')
        plt.xscale('log')
        plt.yscale('log')
        if (iccd+1)%4==0 :
            plt.legend(bbox_to_anchor=(1.05, 1),loc=2, borderaxespad=0.)
        plt.xlim(xx[0],xx[1])
        plt.ylim(1e-8,5e-5)
    plt.show()
    plotfile=data_to_select['RTM']+':all_CCD_parallelCTE.png'
    fig.savefig(plotfile,bbox_inches='tight')
_____no_output_____#
yv=5.0e-6
yy=[yv,yv]
xx=[10.,300000.]
#sel_color=['r','g','b','c','m','y','k']
nb_color=len(sel_color)
sel_mark=['<','>','*','^','v','+']
nb_mark=len(sel_mark)
flux_sel=[1000,50000]
if serial :
    x=range(cte_s_data[0,0].first,cte_s_data[0,0].first+28)
    for flux in flux_sel:
        for iccd in range(number_of_sensor) :
            fig=plt.figure(figsize=[20,20])
            title='Serial overscan at %d e- %s:%s' % (flux,data_to_select['RTM'],sensor_list[iccd])
            fig.suptitle(title,fontsize=14)
            for ch in range(16):
                ax=fig.add_subplot(4,4,ch+1)
                for irun in range(number_of_run) :
                    ii=np.argmin(np.abs(cte_s_data[irun,iccd].cte_flux_s[ch,:cte_s_data[irun,iccd].lmax[ch]]-flux))
                    yplt=cte_s_data[irun,iccd].cte_y_s[ch,ii,:]
                    #y_min=min(max(min(np.min(yplt)*1.2,0.),-10.),y_min)
                    #y_max=max(min(np.max(yplt)*1.2,100.),y_max)
                    plt.plot(x,yplt,color=sel_color[irun%nb_color],label=run_name_list[irun])
                label='channel %d  : serial overscan ' % (ch+1)
                plt.xlabel(label)
                if ch%4 == 0 :
                    plt.ylabel('flux in e-')
                #plt.xscale('log')
                plt.yscale('symlog')
                if (ch+1)%4==0 :
                    plt.legend(bbox_to_anchor=(1.05, 1),loc=2, borderaxespad=0.)
                #plt.xlim(xx[0],xx[1])
                #plt.ylim(,5e-5)
            plt.show()
            plotfile='%s:%sat%d_serialoverscan.png' % (data_to_select['RTM'],sensor_list[iccd],flux)
            fig.savefig(plotfile,bbox_inches='tight')
if parallel :
    x=range(cte_p_data[0,0].first,cte_p_data[0,0].first+28)
    for flux in flux_sel:
        for iccd in range(number_of_sensor) :
            fig=plt.figure(figsize=[20,20])
            title='// overscan at %d e- %s:%s' % (flux,data_to_select['RTM'],sensor_list[iccd])
            fig.suptitle(title,fontsize=14)
            for ch in range(16):
                ax=fig.add_subplot(4,4,ch+1)
                for irun in range(number_of_run) :
                    ii=np.argmin(np.abs(cte_p_data[irun,iccd].cte_flux_s[ch,:cte_s_data[irun,iccd].lmax[ch]]-flux))
                    yplt=cte_p_data[irun,iccd].cte_y_s[ch,ii,:]
                    #y_min=min(max(min(np.min(yplt)*1.2,0.),-10.),y_min)
                    #y_max=max(min(np.max(yplt)*1.2,100.),y_max)
                    plt.plot(x,yplt,color=sel_color[irun%nb_color],label=run_name_list[irun])
                label='channel %d  : // overscan ' % (ch+1)
                plt.xlabel(label)
                if ch%4 == 0 :
                    plt.ylabel('flux in e-')
                #plt.xscale('log')
                plt.yscale('symlog')
                if (ch+1)%4==0 :
                    plt.legend(bbox_to_anchor=(1.05, 1),loc=2, borderaxespad=0.)
                #plt.xlim(xx[0],xx[1])
                #plt.ylim(,5e-5)
            plt.show()
            plotfile='%s:%sat%d_paralleloverscan.png' % (data_to_select['RTM'],sensor_list[iccd],flux)
            fig.savefig(plotfile,bbox_inches='tight')

_____no_output_____# summary for the full raft at low and high flux overlaing the # selected run
# this box is executed only if all channels of the raft are selected
#
if number_of_sensor < 9 :
    print('You should not run this box with less than 9 sensors selected for a given raft')
    raise
# create the associated table with the right dimension
if parallel :
    cte_p_1000=np.zeros((number_of_run,number_of_sensor,16))
    cte_p_1000_std=np.zeros((number_of_run,number_of_sensor,16))
    cte_p_50000=np.zeros((number_of_run,number_of_sensor,16))
    cte_p_50000_std=np.zeros((number_of_run,number_of_sensor,16))
if serial :
    cte_s_1000=np.zeros((number_of_run,number_of_sensor,16))
    cte_s_1000_std=np.zeros((number_of_run,number_of_sensor,16))
    cte_s_50000=np.zeros((number_of_run,number_of_sensor,16))
    cte_s_50000_std=np.zeros((number_of_run,number_of_sensor,16))
#
# fill the cte data
for irun in range(number_of_run) :
    for iccd in range(number_of_sensor) :
         for ch in range(16) :
            # this parameter is to avoid to look for low flux in saturated image
            alf_icur=int(cte_s_data[irun,iccd].lmax[ch]/2)
            #
            if serial :
                ii=np.argmin(np.abs(np.array(cte_s_data[irun,iccd].cte_flux_s[ch,:alf_icur])-1000))
                 # as error are large at 1000 I extrapolate from points above 1000 ???? hum
                cte_s_1000[irun,iccd,ch]=cte_s_data[irun,iccd].ylev[ch,ii]
                cte_s_1000_std[irun,iccd,ch]=cte_s_data[irun,iccd].ylev_std[ch,ii]
                ii=np.argmin(np.abs(np.array(cte_s_data[irun,iccd].cte_flux_s[ch,:])-50000))
                cte_s_50000[irun,iccd,ch]=cte_s_data[irun,iccd].ylev[ch,ii]
                cte_s_50000_std[irun,iccd,ch]=cte_s_data[irun,iccd].ylev_std[ch,ii]
            #
            if parallel :
                ii=np.argmin(np.abs(np.array(cte_p_data[irun,iccd].cte_flux_s[ch,:alf_icur])-1000))
                # as error are large at 1000 I extrapolate from points above 1000 ???? hum
                cte_p_1000[irun,iccd,ch]=cte_p_data[irun,iccd].ylev[ch,ii]
                cte_p_1000_std[irun,iccd,ch]=cte_p_data[irun,iccd].ylev_std[ch,ii]
                ii=np.argmin(np.abs(np.array(cte_p_data[irun,iccd].cte_flux_s[ch,:])-50000))
                cte_p_50000[irun,iccd,ch]=cte_p_data[irun,iccd].ylev[ch,ii]
                cte_p_50000_std[irun,iccd,ch]=cte_p_data[irun,iccd].ylev_std[ch,ii]

# labels for the plots
label_txt=np.zeros((9),dtype=np.object_)
label_pos=np.zeros((9))
label_chan=np.zeros((18),dtype=np.object_)
label_chan_pos=np.zeros((18))
for jcd in range(3) :
    for icd in range(3) :
        s='S%d%d' % (jcd,icd)
        iccd=icd+3*jcd
        label_txt[iccd]=s
        label_pos[iccd]=iccd*16+7.5
        label_chan[iccd*2]='%d' % (1)
        label_chan[iccd*2+1]='%d' % (9)
        label_chan_pos[iccd*2]=iccd*16
        label_chan_pos[iccd*2+1]=iccd*16+8
xx1=[0.,144.]
yv1=5.0e-6
yy1=[yv1,yv1]
sel_mark=['<','^','>','v','*','+']
nb_mark=len(sel_mark)
sel_color=['r','g','b','c','m','y','k']
nb_color=len(sel_color)
#
if serial :
    fig=plt.figure(figsize=[15,10])
    fig.suptitle(data_to_select['RTM']+' serial CTE ',fontsize=14)
    ax1=fig.add_subplot(1,1,1)
    ax2=ax1.twiny()
    #
    for irun in range(number_of_run) :
        ax1.plot(np.ravel(cte_s_1000[irun,:,:]),sel_mark[(2*irun)%nb_mark],color=sel_color[(2*irun)%nb_mark],label='low flux'+':'+run_name_list[irun])
        ax1.plot(np.ravel(cte_s_50000[irun,:,:]),sel_mark[(2*irun+1)%nb_mark],color=sel_color[(2*irun+1)%nb_mark],label='high flux'+':'+run_name_list[irun])
    ax1.plot(xx1,yy1,'g')
    ax1.legend(bbox_to_anchor=(1.05, 1),loc=2, borderaxespad=0.)
    ax1.set_yscale('log')
    ax1.set_ylabel('1-CTE')
    ax1.set_xlabel('Sensor')
    ax1.set_xlim(0,144)
    ax1.set_xticks(label_pos)
    ax1.set_xticklabels(label_txt)
    ax2.set_xlim(ax1.get_xlim())
    ax2.tick_params(labeltop='on')
    ax2.set_xticks(label_chan_pos)
    ax2.set_xticklabels(label_chan)
    ax2.set_xlabel(data_to_select['RTM']+' Channels')
    plt.show()
    plotfile=data_to_select['RTM']+'_serialCTE.png'
    fig.savefig(plotfile,bbox_inches='tight')
if parallel :
    fig=plt.figure(figsize=[15,10])
    fig.suptitle(data_to_select['RTM']+' // CTE ',fontsize=14)
    ax1=fig.add_subplot(1,1,1)
    ax2=ax1.twiny()
    for irun in range(number_of_run) :
        ax1.plot(np.ravel(cte_p_1000[irun,:,:]),sel_mark[(irun*2)%nb_mark],color=sel_color[(2*irun)%nb_mark],label='low flux'+':'+run_name_list[irun])
        ax1.plot(np.ravel(cte_p_50000[irun,:,:]),sel_mark[(irun*2+1)%nb_mark],color=sel_color[(2*irun+1)%nb_mark],label='high flux'+':'+run_name_list[irun])
    ax1.plot(xx1,yy1,'g')
    ax1.set_xlim(0,144)
    ax1.legend(bbox_to_anchor=(1.05, 1),loc=2, borderaxespad=0.)
    ax1.set_yscale('log')
    ax1.set_ylabel('1-CTE')
    ax1.set_xlabel('Sensor')
    ax1.set_xticks(label_pos)
    ax1.set_xticklabels(label_txt)
    ax2.set_xlim(ax1.get_xlim())
    ax2.tick_params(labeltop='on')
    ax2.set_xticks(label_chan_pos)
    ax2.set_xticklabels(label_chan)
    ax2.set_xlabel(data_to_select['RTM']+' Channels')
    plt.show()
    plotfile=data_to_select['RTM']+'_paralCTE.png'
    fig.savefig(plotfile,bbox_inches='tight')
_____no_output_____# serial trap overview (manely usefull for e2v device)
flux_cte_s=[100.,1000.,5000.]
#flux_cte_p=[1000.,50000.]
flux_flag=['<','>','*','v']
sel_color=['r','g','b','c','m','y','k']
nb_color=len(sel_color)
flux_label=['100e-','1ke-','5ke-','1k-100e-']
trap_s_flux=np.zeros((number_of_run,len(flux_cte_s)+1,9,16))
#
xx1=[0.,17.]
yv1=15
yy1=[yv1,yv1]
yv2=1
yy2=[yv2,yv2]

#
if serial :
    for iccd in range(number_of_sensor) :
        fig=plt.figure(figsize=[15,10])
        fig.suptitle('Serial trap identification :'+data_to_select['RTM']+':'+sensor_list[iccd],fontsize=14)
        ax1=fig.add_subplot(1,1,1)
        for irun in range(number_of_run) :
            for i_flux in range(len(flux_cte_s)) :
                for ch in range(16) :
                    ii=np.argmin(np.abs(cte_s_data[irun,iccd].cte_flux_s[ch,:cte_s_data[irun,iccd].lmax[ch]]-flux_cte_s[i_flux]))
                    trap_s_flux[irun,i_flux,iccd,ch]=cte_s_data[irun,iccd].cte_y_s[ch,ii,0]+cte_s_data[irun,iccd].cte_y_s[ch,ii,1]
                    if i_flux==1 :
                        trap_s_flux[irun,-1,iccd,ch]= trap_s_flux[irun,i_flux,iccd,ch]-trap_s_flux[irun,i_flux-1,iccd,ch]
                ax1.plot(range(1,17),np.ravel(trap_s_flux[irun,i_flux,iccd,:]),flux_flag[i_flux],color=sel_color[irun%nb_color],label=flux_label[i_flux]+':'+run_name_list[irun])
                if i_flux==1 :
                    ax1.plot(range(1,17),np.ravel(trap_s_flux[irun,-1,iccd,:]),flux_flag[-1],color=sel_color[irun%nb_color],label=flux_label[-1]+':'+run_name_list[irun])

        ax1.plot(xx1,yy1,'g')
        ax1.plot(xx1,yy2,'g--')
        ax1.legend(bbox_to_anchor=(1.05, 1),loc=2, borderaxespad=0.)
        #ax1.set_yscale('log')
        ax1.set_ylabel('Overscan pix1 + pix2 content in e-',fontsize=14)
        ax1.set_xlabel('Channel')
        ax1.set_xlim(0,16)
        plt.show()
        plotfile=data_to_select['RTM']+':'+sensor_list[iccd]+'_trapserial.png'
        fig.savefig(plotfile,bbox_inches='tight')
_____no_output_____
</code>


# Notebook from ilam56/wg-evolution
Path: implementations/notebooks_df/reviews.ipynb

## Reviews
This is the reference implementation for [Reviews](https://github.com/chaoss/wg-evolution/blob/master/metrics/Reviews.md),
a metric specified by the
[Evolution Working Group](https://github.com/chaoss/wg-evolution) of the
[CHAOSS project](https://chaoss.community).
This implementation is specific to Git repositories.

Have a look at [README.md](../README.md) to find out how to run this notebook (and others in this directory) as well as to get a better understanding of the purpose of the implementations.

The implementation is described in two parts (see below):

* Class for computing Reviews
* An explanatory analysis of the class' functionality

Some more auxiliary information in this notebook:

* Examples of the use of the implementation_____no_output_____As discussed in the [README](../README.md) file, the scripts required to analyze the data fetched by Perceval are located in the `code_df` package. Due to python's import system, to import modules from a package which is not in the current directory, we have to either add the package to `PYTHONPATH` or simply append a `..` to `sys.path`, so that `code_df` can be successfully imported. _____no_output_____

from datetime import datetime
import matplotlib.pyplot as plt
import sys
sys.path.append('..')
from code_df import utils
from code_df import conditions
from code_df.pullrequest import PullRequest
%matplotlib inline_____no_output_____class Reviews(PullRequest):
    """
    Class for Reviews
    """

    def compute(self):
        """
        Compute the total number of reviews created, from the Perceval data.

        :returns count: The total number of reviews created
        """

        count = len(self.df['hash'].unique())
        return count

    def _agg(self, df, period):
        """
        Perform an aggregation operation on a DataFrame to find
        the total number of reviews created in every
        interval of the period specified in the time_series method,
        like 'M', 'W',etc.

        It computes the count of the "category" column of the
        DataFrame.

        :param df: a pandas DataFrame on which the aggregation will be
            applied.

        :param period: A string which can be any one of the pandas time
            series rules:
            'W': week
            'M': month
            'D': day

        :returns df: The aggregated dataframe, where aggregations have
            been performed on the "category" column
        """

        df = df.resample(period)['category'].agg(['count'])

        return df_____no_output_____

## Performing the Analysis
Using the above class, we can perform several kinds of analysis on the JSON data file, fetched by Perceval.

For starters, we can perform a simple count of all reviews in the data. For this analysis, we can vary the value passed to the `date_range` parameter._____no_output_____### Counting the total number of commits
We first read the JSON file containing Perceval data using the `read_json_file` utility function. _____no_output_____

items = utils.read_json_file('../pull_requests.json')_____no_output_____

Let's use the `compute` method to count the total number of reviews created. First, we will do it without passing any since and until dates.
Next, we can pass in the start and end dates as a tuple. The format would be `%Y-%m-%d`.

A third kind of analysis we can perform is passing only one of the dates to `date_range` --- either `since` or `until`._____no_output_____

reviews = Reviews(items)
print("The total number of reviews "
      "in the file is {}.".format(reviews.compute()))

date_since = datetime.strptime("2018-01-01", "%Y-%m-%d")
date_until = datetime.strptime("2018-07-01", "%Y-%m-%d")

reviews_dated = Reviews(items,
                        date_range=(date_since, date_until))

print("The total number of reviews created between "
      "2018-01-01 and 2018-07-01 is {}.".format(reviews_dated.compute()))

reviews_after = Reviews(items,
                        date_range=(date_since, None))

print("The total number of reviews created after "
      "2018-01-01 is {}.".format(reviews_after.compute()))The total number of reviews in the file is 229.
The total number of reviews created between 2018-01-01 and 2018-07-01 is 43.
The total number of reviews created after 2018-01-01 is 75.

## Counting reviews over regular time intervals
Using the `time_series` method, it is possible to compute the number of reviews created every month, or every week, or infact any valid pandas timeperiod. This kind of analysis is useful in finding trends over time, as we will see in the cell below.

Let's perform a basic analysis: lets see the change in the number of reviews created between the same dates we used above on a weekly basis: 2018-01-01 and 2018-07-01.
The Reviews object, `reviews_dated`, will be the same as used above._____no_output_____

weekly_df = reviews_dated.time_series(period='W')_____no_output_____

Lets see what the dataframe returned by `time_series` looks like. As you will notice, the dataframe has rows corresponding to each and every week between the start and end dates. To do this, we simply set the `created_date` column of the DataFrame `changes_dated.df`, as its index and then `resample` it to whatever time period we need. In this case, we have used `W`. _____no_output_____

weekly_df_____no_output_____

Lets plot the dataframe `weekly_df` using matplotlib.pyplot. We use the `seaborn` theme and plot a simple line plot --- commit count vs time interval. Using the `plt.fill_between` method allows us to "fill up" the area between the line plot and the x axis._____no_output_____

plt.style.use('seaborn')
weekly_df.plot(y='count', use_index=True)
plt.fill_between(y1=weekly_df['count'], y2=0, x=weekly_df.index)
plt.title("Review Count");_____no_output_____

The same thing can be tried for months, instead of weeks. By passing `month` in place of week, we get a similar dataframe but with only a few rows, due to the larger timescale. _____no_output_____

monthly_df = reviews_dated.time_series('M')_____no_output_____monthly_df_____no_output_____

Lets plot it just like we did for `weekly_df`._____no_output_____

plt.style.use('seaborn')
monthly_df.plot(y='count', use_index=True)
plt.fill_between(y1=monthly_df['count'], y2=0, x=monthly_df.index)
plt.title("Review Count");_____no_output_____


# Notebook from SevdanurGENC/Quantum-Programming-With-Qiskit
Path: Bronze/classical-systems/CS20_Probabilistic_Operators.ipynb

$ \newcommand{\bra}[1]{\langle #1|} $
$ \newcommand{\ket}[1]{|#1\rangle} $
$ \newcommand{\braket}[2]{\langle #1|#2\rangle} $
$ \newcommand{\dot}[2]{ #1 \cdot #2} $
$ \newcommand{\biginner}[2]{\left\langle #1,#2\right\rangle} $
$ \newcommand{\mymatrix}[2]{\left( \begin{array}{#1} #2\end{array} \right)} $
$ \newcommand{\myvector}[1]{\mymatrix{c}{#1}} $
$ \newcommand{\myrvector}[1]{\mymatrix{r}{#1}} $
$ \newcommand{\mypar}[1]{\left( #1 \right)} $
$ \newcommand{\mybigpar}[1]{ \Big( #1 \Big)} $
$ \newcommand{\sqrttwo}{\frac{1}{\sqrt{2}}} $
$ \newcommand{\dsqrttwo}{\dfrac{1}{\sqrt{2}}} $
$ \newcommand{\onehalf}{\frac{1}{2}} $
$ \newcommand{\donehalf}{\dfrac{1}{2}} $
$ \newcommand{\hadamard}{ \mymatrix{rr}{ \sqrttwo & \sqrttwo \\ \sqrttwo & -\sqrttwo }} $
$ \newcommand{\vzero}{\myvector{1\\0}} $
$ \newcommand{\vone}{\myvector{0\\1}} $
$ \newcommand{\stateplus}{\myvector{ \sqrttwo \\  \sqrttwo } } $
$ \newcommand{\stateminus}{ \myrvector{ \sqrttwo \\ -\sqrttwo } } $
$ \newcommand{\myarray}[2]{ \begin{array}{#1}#2\end{array}} $
$ \newcommand{\X}{ \mymatrix{cc}{0 & 1 \\ 1 & 0}  } $
$ \newcommand{\I}{ \mymatrix{rr}{1 & 0 \\ 0 & 1}  } $
$ \newcommand{\Z}{ \mymatrix{rr}{1 & 0 \\ 0 & -1}  } $
$ \newcommand{\Htwo}{ \mymatrix{rrrr}{ \frac{1}{2} & \frac{1}{2} & \frac{1}{2} & \frac{1}{2} \\ \frac{1}{2} & -\frac{1}{2} & \frac{1}{2} & -\frac{1}{2} \\ \frac{1}{2} & \frac{1}{2} & -\frac{1}{2} & -\frac{1}{2} \\ \frac{1}{2} & -\frac{1}{2} & -\frac{1}{2} & \frac{1}{2} } } $
$ \newcommand{\CNOT}{ \mymatrix{cccc}{1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 1 \\ 0 & 0 & 1 & 0} } $
$ \newcommand{\norm}[1]{ \left\lVert #1 \right\rVert } $
$ \newcommand{\pstate}[1]{ \lceil \mspace{-1mu} #1 \mspace{-1.5mu} \rfloor } $
$ \newcommand{\greenbit}[1] {\mathbf{{\color{green}#1}}} $
$ \newcommand{\bluebit}[1] {\mathbf{{\color{blue}#1}}} $
$ \newcommand{\redbit}[1] {\mathbf{{\color{red}#1}}} $
$ \newcommand{\brownbit}[1] {\mathbf{{\color{brown}#1}}} $
$ \newcommand{\blackbit}[1] {\mathbf{{\color{black}#1}}} $_____no_output_____ Probabilistic Operators

_prepared by Abuzer Yakaryilmaz_

[](https://youtu.be/zYgym4NVq6o)

_____no_output_____Remember Asja's biased coins, and her coin-flipping protocol.

$
GameCoins = \begin{array}{c|cc} \hookleftarrow & \mathbf{Head} & \mathbf{Tail} \\ \hline \mathbf{Head} & 0.6 & 0.3\\  \mathbf{Tail} & 0.4 & 0.7  \end{array} \equiv \begin{array}{c|cc} \hookleftarrow & \mathbf{0} & \mathbf{1} \\ \hline \mathbf{0} & 0.6 & 0.3 \\  \mathbf{1} & 0.4 & 0.7  \end{array}
$_____no_output_____We trace Asja's outcomes after two coin flips.

 At the beginning:

 Remember the protocol:

 whenever she gets a head, she flips one euro;

 whenever she gets a tail, she flips one cent; and

 she starts with flipping one euro by assuming that she got head from nowhere

She starts in state 0: $ v_0 = \myvector{1 \\ 0} $.

State 0 represents Head and state 1 represents Tail._____no_output_____After first coin flip:

$  \mypar{
        GameCoins = \begin{array}{c|cc} \hookleftarrow & \mathbf{Head} & \mathbf{Tail} \\ \hline \mathbf{Head} & 0.6 & 0.3\\  \mathbf{Tail} & 0.4 & 0.7  \end{array}
        }
    \mypar{
        CurrentState=\myvector{1 \\ 0}
    } $

Each entry of the new vector is calculated by a row and current state:

$
\myvector{  0.6 \cdot 1 \\ + \\ 0.3 \cdot 0 \\ \hline 0.4 \cdot 1 \\ + \\ 0.7 \cdot 0  }
=
\myvector{ 0.6 + 0 \\ 0.4 + 0 }
=
\myvector{0.6 \\ 0.4}.
$_____no_output_____After second coin flip:

$  \mypar{
        GameCoins = \begin{array}{c|cc} \hookleftarrow & \mathbf{Head} & \mathbf{Tail} \\ \hline \mathbf{Head} & 0.6 & 0.3\\  \mathbf{Tail} & 0.4 & 0.7  \end{array}
        }
    \mypar{
        CurrentState=\myvector{0.6 \\ 0.4}
    } $

Each entry of the new vector is calculated by a row and current state:

$
\myvector{  0.6 \cdot 0.6 \\ + \\ 0.3 \cdot 0.4 \\ \hline 0.4 \cdot 0.6 \\ + \\ 0.7 \cdot 0.4  }
=
\myvector{ 0.36 + 0.12 \\ 0.24 + 0.28 }
=
\myvector{0.48 \\ 0.52}.
$_____no_output_____Coin-flipping protocol of Asja is a probabilistic operator.

Similarly to any operator, depending on the current state, Asja's coin-flipping protocol determines the next state.

$$
    \myvector{1 \\ 0} \xrightarrow{\mbox{Asja's coin-flipping protocol}} \myvector{0.6 \\ 0.4}
    \xrightarrow{\mbox{Asja's coin-flipping protocol}}  \myvector{0.48 \\ 0.52}.
$$_____no_output_____A probabilistic operator evolves the system from a probabilistic state to a probabilistic state.

Asja's coin-flipping protocol transforms $ \myvector{ 0.8 \\ 0.2 } $ to $ \myvector{ 0.54 \\ 0.46 } $.

When calculating the new state, we use the table $  GameCoins = \begin{array}{c|cc} \hookleftarrow & \mathbf{Head} & \mathbf{Tail} \\ \hline \mathbf{Head} & 0.6 & 0.3\\  \mathbf{Tail} & 0.4 & 0.7  \end{array} $ and the current state $ \myvector{ 0.8 \\ 0.2 } $:

$$
    \myvector{ \myarray{c}{0.6 \cdot 0.8 \\ + \\ 0.3 \cdot 0.2} \\ \hline \myarray{c}{0.4 \cdot 0.8 \\ + \\ 0.7 \cdot 0.2} } = \myvector{ \myarray{c}{0.48 + 0.06} \\ \myarray{c}{0.32 + 0.14}  } = \myvector{ 0.54 \\ 0.46 }.
$$_____no_output_____
 Probabilistic operator

A probabilistic operator can be represented as a square table or matrix.

The entries of a probabilistic operator represent the transition probabilities between the states.

Therefore, each entry is nonnegative.

Each column represents the transition probabilities from a state to all states. Therefore, the summation of all entries in each column is 1, i.e., probability 1 is distributed over all states.

Any matrix satisfying these two properties is called a  stochastic matrix.

A probabilistic operator is a stochastic matrix, and vice versa.

 Remark that the operator of any linear system is represented as a matrix. _____no_output_____
 Probabilistic evolution

A probabilistic state is a stochastic vector,  say $ v $.

A probabilistic operator is a stochastic matrix, say $ A $,

If probabilistic operator $ A $ is applied to probabilistic state $ v $, the new state, say $v'$, is calculated as

$  v' = A \cdot v. $

 Remark that the evolution of linear system is represented by matrix-vector multiplication. _____no_output_____If we represent $ GameCoins $ as a matrix: $ \mymatrix{cc}{ 0.6 & 0.3 \\ 0.4 & 0.7 } $, then the new probabilistic state is calculated as

$
    \myvector{ 0.54 \\ 0.46 } = \mymatrix{cc}{ 0.6 & 0.3 \\ 0.4 & 0.7 } \myvector{ 0.8 \\ 0.2 }.
$_____no_output_____
 Task 1

The operator $ GameCoins = \mymatrix{cc}{ 0.6 & 0.3 \\ 0.4 & 0.7 } $ is applied to the probabilistic state $ \myvector{ 0.1 \\ 0.9 } $.

Then, the new probabilistic state is

$ \mymatrix{cc}{ 0.6 & 0.3 \\ 0.4 & 0.7 } \myvector{ 0.1 \\ 0.9 } = \myvector{ 0.33 \\ 0.67 }. $

Please verify the correctness of matrix-vector multiplication._____no_output_____
 Task 2

We are given the following probabilistic operator: $ B =  \mymatrix{ccc}{ 0.4 & 0.6 & 0 \\ 0.2 & 0.1 & 0.7 \\ 0.4 & 0.3 & 0.3 } $.

What is the transition probability from the second state to the third state?

What is the transition probability from the third state to the first state?

What is the transition probability from the first state to the second state?_____no_output_____click for our solution_____no_output_____
Task 3

Randomly construct a $ (3 \times 3 ) $-dimensional probabilistic operator.

That is, randomly determine the entries of the matrix that represents a probabilistic operator._____no_output_____

#
# your solution is here
#
_____no_output_____

click for our solution_____no_output_____
 Task 4

What is the new probabilistic state if the operator $ B =  \mymatrix{ccc}{ 0.4 & 0.6 & 0 \\ 0.2 & 0.1 & 0.7 \\ 0.4 & 0.3 & 0.3 } $ is applied to the state $ \myvector{ 0.1 \\ 0.3 \\ 0.6 } $.

Please find the result by using matrix-vector multiplication.

Please do not use any python library for matrix-vector multiplication.

 The new probabilistic state should be $ \myvector{0.22 \\ 0.47 \\ 0.31} $. _____no_output_____

# operator B
B = [
    [0.4,0.6,0],
    [0.2,0.1,0.7],
    [0.4,0.3,0.3]
]

# the current state
v = [0.1,0.3,0.6]

#
# your solution is here
#_____no_output_____

click for our solution_____no_output_____
 Task 5

Write a function named linear_evolve that takes a probabilistic operator and a probabilistic state, and then returns the new probabilistic state.

Please do not use any python library for matrix-vector multiplication.

Your function should work for any dimension.

Save your function so that you can use it later.

Write the content of a code cell into an external file:

    %%writefile linear_evolve.py
This command should be placed in the first line of cell, and then the cell should be executed.

Test your function on $ B = \mymatrix{ccc}{ 0.4 & 0.6 & 0 \\ 0.2 & 0.1 & 0.7 \\ 0.4 & 0.3 & 0.3 } $ and $ \myvector{0.1 \\ 0.3 \\ 0.6} $.

The new probabilistic state should be $ \myvector{0.22 \\ 0.47 \\ 0.31} $.

Then, evolve your system for 5, 10, 20, and 40 steps.

This system should evolve to a fixed probabilistic state.

Change your initial state to  $ \myvector{1 \\ 0 \\ 0} $, and see whether the converged state is the same or not._____no_output_____

# %%writefile linear_evolve.py_____no_output_____# run your script
# %run linear_evolve.py

# operator for the test
B = [
    [0.4,0.6,0],
    [0.2,0.1,0.7],
    [0.4,0.3,0.3]
]

# state for test
v = [0.1,0.3,0.6]

#
# your solution is here
#
_____no_output_____

click for our solution_____no_output_____---

 Extra: Task 6

The operator $ \widetilde{I} = \mymatrix{cc}{0.999 & 0.001\\ 0.001 & 0.999 } $ is very similar to Identity operator. However, it converges to a certain matrix.

Find $  \underbrace{\widetilde{I} \cdot \widetilde{I} \cdots \widetilde{I}}_{k\mbox{ times}} $ for $ k = 10, 100, 1000, 10000, 100000 $ and then guess the converging matrix._____no_output_____
 Extra: Task 7

Repeat Task 6 for the operator $ \widetilde{NOT} = \mymatrix{cc}{0.001 & 0.999\\ 0.999 & 0.001 } $._____no_output_____


# Notebook from Deena-B/ILI
Path: Figures/Fig1_CDC-ILI/Fig1_CDC-ILI_20180726_1258_summarizeMunging.ipynb

# 20180726 Thursday

Actions
- import programs
- open & read in csv file, assign header row
- drop 1st row
- add columns with calcs_____no_output_____

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline_____no_output_____# read my ILINet.csv file into a dataframe using pandas
# Explicitly pass header=<row index number> to be able to replace existing names
ILIcsvDF = pd.read_csv('../../Data/CDCvirusILI/FluViewPhase2Data/ILINet.csv', header=1)
# show first few entries
ILIcsvDF.head()_____no_output_____# List column headers
list(ILIcsvDF)_____no_output_____# delete the line at the top of the file that says
# "PERCENTAGE OF VISITS FOR INFLUENZA-LIKE-ILLNESS REPORTED BY SENTINEL PROVIDERS"
ILIcsvDF.drop(index=0)
# without the cmd below
# the 1st and last 30 rows print below
ILIcsvDF.head()_____no_output_____

What are the numbers in the left column: keys, IDs, types, names, axes? They are not indecies!
- They are probably index locations .iloc! _____no_output_____

# List the data types for each column
ILIcsvDF.dtypes_____no_output_____

Note that YEAR & WEEK are integers :)_____no_output_____

# Divide week by 53 to determine PortionOfYear
ILIcsvDF["PortionOfYear"] = ILIcsvDF["WEEK"] / 53
ILIcsvDF.head()_____no_output_____# create a new column "YearWPortion"
# add "YEAR" + "YearWPortion"
ILIcsvDF["YearWPortion"] = ILIcsvDF["YEAR"] + ILIcsvDF["PortionOfYear"]
ILIcsvDF.head()_____no_output_____

Next Action
- export new file_____no_output_____

# export elements from dataframe into a new csv
# use the instance method to_csv
# filepath is the only required argument
# for more arguments
# see section 24.1.26.1 of pandas doc, pg 1206
ILIcsvDF.to_csv('../../Data/CDCvirusILI/FluViewPhase2Data/ILINet20180726.csv')_____no_output_____

This ^ worked :)

Next Action
- graph x-axis = "YearWPortion", y-axis = "%UNWEIGHTED ILI"
- do I need to take the space out of the column header? _____no_output_____## How to make a graph

Types of Plots in matplotlib
- matplotlib Gallery
    - https://matplotlib.org/gallery.html
        - a simple line graph (using sin) https://matplotlib.org/examples/lines_bars_and_markers/line_demo_dash_control.html
        - example code for lines: solid, dashed, dotted, etc
            - https://matplotlib.org/examples/lines_bars_and_markers/linestyles.html
            - https://matplotlib.org/examples/lines_bars_and_markers/line_styles_reference.html

- matplotlib.pyplot module summary
    - https://matplotlib.org/api/pyplot_summary.html
    - contains functions that allow you to generate many kinds of plots quickly
    - While it is easy to quickly generate plots with the matplotlib.pyplot module, we recommend using the object-oriented approach for more control and customization of your plots.
    - includes section on colors
- matplotlib.pyplot Functions
    - https://matplotlib.org/api/_as_gen/matplotlib.pyplot.html#module-matplotlib.pyplot
    - matplotlib.pyplot provides a MATLAB-like way of plotting
- matplotlib.pyplot gallery
    - https://matplotlib.org/gallery/index.html#pyplots-examples
- matplotlib.pyplot tutorial
    - https://matplotlib.org/tutorials/introductory/pyplot.html#sphx-glr-tutorials-introductory-pyplot-py

_____no_output_____

# assign "YearWPortion" to x-values
x = ILIcsvDF['YearWPortion']
# assign "%UNWEIGHTED ILI" to y-values
y = ILIcsvDF['%UNWEIGHTED ILI']

# label the axes
plt.xlabel('Time (years)')
plt.ylabel('% ILI')

# sets the min and max of the axes [xmin, xmax, ymin, ymax]
# plt.axis([1997, 2019, 0, 100])

# plots x versus y
plt.plot(x,y)

# do I need to show - no!
# plt.show()_____no_output_____

Next Action
- explicitly define the axes limits_____no_output_____

# assign x and y values
x = ILIcsvDF['YearWPortion']
y = ILIcsvDF['%UNWEIGHTED ILI']

# label axes
plt.xlabel('Time (weeks)')
plt.ylabel('% ILI')

# set the min and max of the axes [xmin, xmax, ymin, ymax]
plt.axis([1997, 2019, -.1, 8])

# plots x versus y
plt.plot(x,y)_____no_output_____

Next Action
- Add more years to the x-axis

Questions:
what is a subplot()_____no_output_____Python’s range() Function Explained
- https://www.pythoncentral.io/pythons-range-function-explained/ _____no_output_____

# generate a list of years from 1997 to 2019
for xAxisTicks in range(1997, 2020, 1):
    print(xAxisTicks)1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019

Python 3 turn range to a list
- https://stackoverflow.com/questions/11480042/python-3-turn-range-to-a-list_____no_output_____

# generate a list of years from 1997 to 2019
xAxisTicks = list(range(1997, 2020, 1))
print(xAxisTicks)[1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019]
# assign x and y values
x = ILIcsvDF['YearWPortion']
y = ILIcsvDF['%UNWEIGHTED ILI']

# label axes
plt.xlabel('Time (weeks)')
plt.ylabel('% ILI')

# set the min and max of the axes
# [xmin, xmax, ymin, ymax]
plt.axis([1997, 2019, -.1, 8])

# generate a list of years from 1997 to 2019
xAxisTicks = list(range(1997, 2020, 1))

# Set tick locations and labels on axes
plt.xticks(xAxisTicks, xAxisTicks, rotation=90)

# plots x versus y
plt.plot(x,y)_____no_output_____

Questions
- xticks(locs, [labels], \**kwargs)
    - do I need [] around xAxisTicks?
        - no
    - or is it indicating that a list should go there?
        - yes
- do I need to make sure locs are integers?
- do I need to make sure [labels] are strings?_____no_output_____Next Action
- what is a subplot?
    - subplot is a function that allows you to graph 2 or more plots in one figure
    - https://matplotlib.org/api/_as_gen/matplotlib.pyplot.subplot.html
- plot 2017 - 2019
    - https://matplotlib.org/api/_as_gen/matplotlib.axes.Axes.set_xlim.html
- _____no_output_____

# assign x and y values
x = ILIcsvDF['YearWPortion']
y = ILIcsvDF['%UNWEIGHTED ILI']

# label axes
plt.xlabel('Time (weeks)')
plt.ylabel('% ILI')

# set the min and max of the axes
# [xmin, xmax, ymin, ymax]
plt.axis([1997, 2019, -.1, 8])

# generate a list of years from 1997 to 2019
xAxisTicks = list(range(2017, 2020, 1))

# Set tick locations and labels on axes
plt.xticks(xAxisTicks, xAxisTicks, rotation=90)

# Plot only 2017 - 2019
plt.xlim(2017, 2019)

# plots x versus y
plt.plot(x,y)_____no_output_____

Next Action
- show more axes ticks_____no_output_____

# assign x and y values
x = ILIcsvDF['YearWPortion']
y = ILIcsvDF['%UNWEIGHTED ILI']

# label axes
plt.xlabel('Time (weeks)')
plt.ylabel('% ILI')

# set the min and max of the axes
# [xmin, xmax, ymin, ymax]
plt.axis([1997, 2019, -.1, 8])

# generate a list of years from 1997 to 2019
xAxisTicks = list(range(2017, 2020, 0.15))

# Set tick locations and labels on axes
plt.xticks(xAxisTicks, xAxisTicks, rotation=90)

# Plot only 2017 - 2019
plt.xlim(2017, 2019)

# plots x versus y
plt.plot(x,y)_____no_output_____

Next Actions
- Find examples of blogs with links to Jupyter Notebooks
    - choose a blog host
- Figure out how to embed Jupyter Notebooks in my blog
- Do I need to put my jupyter notebooks on github so I can link to them from my blog?_____no_output_____Write in my blog
I have decided to switch career paths from molecular virology to become a bioinformatician (wish me luck :) At first I thought learning R would be the best way to start, this is because I was at Johns Hopkins and mostly asking biostatisticians. When I moved to SF, the people I asked said I should learn Python. These pepople were mostly computer engineer and data scientists. I have chosen to start with Python since there are a lot of good support groups in the SF bay area, including PyLadies and SF Python (check out their MeetUps, one which I now host!).
I am motivated by end goals, not processes. Naturally, I am a perfectionist, but I am fighting against this tendency. My first goal is to be able to add something useful to nextstrain.org.  _____no_output_____

# assign x and y values
x = ILIcsvDF['YearWPortion']
y = ILIcsvDF['%UNWEIGHTED ILI']

# label axes
plt.xlabel('Time (weeks)')
plt.ylabel('% ILI')

# set the min and max of the axes
# [xmin, xmax, ymin, ymax]
plt.axis([1997, 2019, -.1, 8])

# generate a list of years from 1997 to 2019
xAxisTicks = list(range(2017, 2020, 0.15))

# Set tick locations and labels on axes
plt.xticks(xAxisTicks, float(xAxisTicks), rotation=90)

# Plot only 2017 - 2019
plt.xlim(2017, 2019)

# plots x versus y
plt.plot(x,y)_____no_output_____

github landing page that converts repository to a landing page, blog: "github pages" or wordpress_____no_output_____

# assign x and y values
x = ILIcsvDF['YearWPortion']
y = ILIcsvDF['%UNWEIGHTED ILI']

# label axes
plt.xlabel('Time (weeks)')
plt.ylabel('% ILI')

# set the min and max of the axes
# [xmin, xmax, ymin, ymax]
plt.axis([1997, 2019, -.1, 8])

# generate a list of years from 1997 to 2019
xAxisTicks = list(np.arange(2017, 2018.6, 0.05))

# Set tick locations and labels on axes
plt.xticks(xAxisTicks, xAxisTicks, rotation=90)

# Plot only 2017 - 2019
plt.xlim(2017.5, 2018.5)

# plots x versus y
plt.plot(x,y)_____no_output_____# assign x and y values
x = ILIcsvDF['YearWPortion']
y = ILIcsvDF['%UNWEIGHTED ILI']

# label axes
plt.xlabel('Time (weeks)')
plt.ylabel('% ILI')

# set the min and max of the axes
# [xmin, xmax, ymin, ymax]
plt.axis([1997, 2019, -.1, 8])

# generate a list of years from 1997 to 2019
xAxisTicks = list(np.arange(2017, 2018.6, 0.03))

# Set tick locations and labels on axes
plt.xticks(xAxisTicks, xAxisTicks, rotation=90)

# Plot only 2017 - 2019
plt.xlim(2017.7, 2018.3)

# plots x versus y
plt.plot(x,y)_____no_output_____

Next Actions
- Make graph with year-week x-axis lables
    - create a column in the dataframe that has "YEAR"-"WEEK" (Concat with "-")
    - sort by the values in the YEAR-WEEK column
    - make a bar/column graph, rather than an x,y graph, but still draw a line (if possible)
    - graph all years (notice that there is no longer a gap for week 53 in the earlier years
    - zoom into x-axis coordinates 2017-37 to 2017-10
        - I worked these week numbers out by divding the decimal by (1/53 = 0.01887)
        - The peak was during week 5-6, the first week of February
- make a dataframe that allows conversion from NIH weeks to calendar days (use Sunday-Saturday as 1 week. When does week 1 start? - email the NIH to ask)
    - ask NIH "How do I convert from week to the first day of the week?"
    - list all the days in every year
    - use a looping function to label 7 consecutive days with a week number
        - week += 1
    - somehow output the 1st (and separately the last) day within a week
    - the output should be a dictionary with week number as the key and date as the value
        - or should it be date as the key and week number as the value
    - in the end I want to be able to write "Isolates from the first day of week X through the last day of week Y were downloaded."_____no_output_____

# I need to create a column in my df with the x header ...
# assign x and y values
x = ILIcsvDF['YEAR-WEEK']
y = ILIcsvDF['%UNWEIGHTED ILI']

# label axes
plt.xlabel('Time (weeks)')
plt.ylabel('% ILI')

# set the min and max of the axes
# [xmin, xmax, ymin, ymax]
plt.axis([1997, 2019, -.1, 8])

# generate a list of years from 1997 to 2019
xAxisTicks = list(np.arange(2017, 2018.6, 0.05))

# Set tick locations and labels on axes
plt.xticks(xAxisTicks, xAxisTicks, rotation=90)

# Plot only 2017 - 2019
plt.xlim(2017.7, 2018.2)

# plots x versus y
plt.plot(x,y)_____no_output_____ILIcsvDF.dtypes_____no_output_____

Next Actions
- concatenate year-week
- Make a bar graph_____no_output_____

# Many ways to concatenate
'{0}{1}'.format('a', 'b')
''.join([a, b])
''.append(a)
# f-string
f'<text> {<expression> <optional !s, !r, or !a> <optional : format specifier>} <text>
+
+= _____no_output_____ILIcsvDF["YearWeek3"] = str(ILIcsvDF["YEAR"])+'-'+ILIcsvDF["WEEK"].style.format('{:02d}')
# d means _____no_output_____ILIcsvDF["YearWeek2"] = '{0}{1}'.format(ILIcsvDF["YEAR"], '\-', ILIcsvDF["WEEK"])
ILIcsvDF.head()_____no_output_____ILIcsvDF["YearWeek1"] = '{0}{1}'.format(ILIcsvDF["YEAR"], '\-', ILIcsvDF["WEEK"])
ILIcsvDF.head()_____no_output_____# create a new column "YearWeek1"
# concatenate "YEAR" "-" "WEEK"
ILIcsvDF["YearWeek1"] = ILIcsvDF["YEAR"].join(ILIcsvDF["WEEK"])
_____no_output_____

Zero prefix so that I don't have 9
- building strings _____no_output_____

pd.ILIcsvDF['YEAR'].astype('str')_____no_output_____df['A'].apply(lambda x: '{:02d}'.format(x))_____no_output_____pd.concat(str(df['B']))_____no_output_____df['B'].astype('str') + '-' + df['A'].apply(lambda x: '{:02d}'.format(x))_____no_output_____df['C'] = df['B'].astype('str') + '-' + df['A'].apply(lambda x: '{:02d}'.format(x))_____no_output_____ILIcsvDF['YearWeek4'] = ILIcsvDF['YEAR'].astype('str') + '-' + ILIcsvDF['WEEK'].apply(lambda x: '{:02d}'.format(x))
ILIcsvDF.head()_____no_output_____# assign "YearWPortion" to x-values
x = ILIcsvDF['YearWeek4']
# assign "%UNWEIGHTED ILI" to y-values
y = ILIcsvDF['%UNWEIGHTED ILI']

# label the axes
plt.xlabel('Time')
plt.ylabel('% ILI')

# sets the min and max of the axes [xmin, xmax, ymin, ymax]
plt.axis([1997, 2002, -.1, 9])

# plots x versus y
plt.bar(x, y)_____no_output_____ILIcsvDF.info()_____no_output_____ILIcsvDF1997 = ILIcsvDF.loc[ILIcsvDF['YEAR']==1997]
ILIcsvDF1997_____no_output_____# create a dataframe
df = pd.dataframe[{0, 1, 2, 3, 4, 5}]_____no_output_____# assign "YearWPortion" to x-values
x = ILIcsvDF1997['YearWeek4']
# assign "%UNWEIGHTED ILI" to y-values
y = ILIcsvDF1997['%UNWEIGHTED ILI']

# label the axes
plt.xlabel('Time')
plt.ylabel('% ILI')

# sets the min and max of the axes [xmin, xmax, ymin, ymax]
# plt.axis([1997, 2002, -.1, 9])

# plots
plt.bar(x, y)

_____no_output_____# assign "YearWPortion" to x-values
x = ILIcsvDF1997['YearWeek4']
# assign "%UNWEIGHTED ILI" to y-values
y = ILIcsvDF1997['%UNWEIGHTED ILI']

# label the axes
plt.xlabel('Time')
plt.ylabel('% ILI')

# sets the min and max of the axes [xmin, xmax, ymin, ymax]
# plt.axis([1997, 2002, -.1, 9])

plt.xticks(rotation='vertical')

# plots
plt.plot(x, y)

_____no_output_____

#### Google
- "pd add a zero prefix to a series of integers"
- "plt format xticks float"
- Andrea's suggestion to skip the question, don't read it because it is bad code

#### Found
- https://stackoverflow.com/questions/29188757/matplotlib-specify-format-of-floats-for-tick-lables
    - from matplotlib.ticker import FormatStrFormatter
    - fig, ax = plt.subplots()
    - ax.yaxis.set_major_formatter(FormatStrFormatter('%.2f'))
- https://stackoverflow.com/questions/42929550/ticks-format-of-an-axis-in-matplotlib/42929894
    - import matplotlib.ticker as mtick
    - plt.gca().yaxis.set_major_formatter(mtick.FormatStrFormatter('%.0e'))
#### Google
- "create a list of floats python 3"
#### Found
- https://stackoverflow.com/questions/41120646/python-3-how-to-create-list-out-of-float-numbers
    - result=[]
    - for value in response_i['objcontent'][0]['rowvalues']:
        - lat = value[0]
        - print(lat)
        - result.append(float(lat))

    - print (result)
#### Google
- "numpy range to make a list of floats"
#### Found
- https://stackoverflow.com/questions/7267226/range-for-floats
- lots of answers
    - import decimal
    - list(frange(0, 100, decimal.Decimal('0.1')))[-1]
    - output: Decimal('99.9')
    - or
    - import decimal

    - def drange(x, y, jump):
        - while x < y:
        - yield float(x)
        - x += decimal.Decimal(jump)
- and then
    - list(drange(0, 100, '0.1'))[-1]
    - output: 99.9
- or
    - [x / 10.0 for x in range(5, 50, 15)]
- or
    - map(lambda x: x/10.0, range(5, 50, 15))
_____no_output_____Other things that I googled
- "plt. bar plot"
- "python 3 concatenate strings"
Found
- https://www.pythonforbeginners.com/concatenation/string-concatenation-and-formatting-in-python
    - String Concatenation and Formatting
        - Concatenation, Formatting in Python, Formatting with the % Operator, Formatting with the { } Operators, Using the Join Method
- https://softwareengineering.stackexchange.com/questions/304445/why-is-s-better-than-for-concatenation
    - Why is %s better than + for concatenation?
        - readability & performance
- https://www.python.org/dev/peps/pep-0498/
    - PEP 498 -- Literal String Interpolation
- http://blog.lerner.co.il/speedy-string-concatenation-python/
def concat1():
    z = x + y
    return z

 def concat2():
    z = "%s%s" % (x, y)
    return z

def concat3():
    z = "{}{}".format(x, y)
    return z

def concat4():
    z = "{0}{1}".format(x, y)
    return z

concat1: 153ns/loop
concat2: 275ns/loop
concat3: 398ns/loop
concat4: 393ns/loop

_____no_output_____#### Yaqi Shi, the Clover host recommended

- read the summary of SciKit Learn

- http://scikit-learn.org/stable/
- http://scikit-learn.org/stable/documentation.html
- http://scikit-learn.org/stable/tutorial/basic/tutorial.html
- http://scikit-learn.org/stable/tutorial/index.html
- http://scikit-learn.org/stable/user_guide.html

_____no_output_____## Blogging

The question is Medium, Wordpress or Git Pages?

- https://firstsiteguide.com/tools/blogging/
- http://www.aroyaldaughter.com/2013/01/04/blogger-vs-wordpress-content-ownership/
- https://webhostinggeeks.com/blog/best-10-blogging-platforms/
- http://www.dearblogger.org/blogger-or-wordpress-better
    - Discover The Best Place to Start a Blog (and Host a Blog) – 2018_____no_output_____


# Notebook from edjuaro/cuscatlan
Path: tests/OC_advanced_hierarchical_clustering.ipynb

Note: if this notebook is taking too long to run, consider using the module HierarchicalClustering v7.3.4 available on gp-beta-ami.genepattern.org_____no_output_____

A note on some of the parameters we are using:

- **gene expression**: We use a compilation of 19 RNASeq samples taken from TCGA, 9 of those are Breast Cancer primary tumors (BRCA) and 10 are matched normal tissue. Here is the GCT file which contains those HTSeq counts:
https://datasets.genepattern.org/data/test_data/BRCA_minimal_60x19.gct_____no_output_____

import genepattern
# import cuzcatlan as cusca
# import pandas as pd
from cuzcatlan import HierarchicalClustering

genepattern.GPUIBuilder(HierarchicalClustering, name="Advanded Hierarchical Clustering.",
                        description="This function performs hierarchical clustering to group samples (columns) and/or genes (rows) with similar expression profiles.",
                        parameters={
                                "pwd": { "hide": True, "default": '.' },
                                "show": { "hide": True, "default": True },
                                "col_distance_metric":{
                                                      "default": "pearson",
                                                      "choices":{'Information Coefficient':"information_coefficient",
                                                                 'City Block (Manhattan or L1-norm)':'manhattan',
                                                                 'Euclidean (L2-norm)':"euclidean",
                                                                 'Pearson Correlation':"pearson",
                                                                 'Uncentered Pearson Correlation':'uncentered_pearson',
                                                                 'Uncentered Pearson Correlation, absolute value':'absolute_uncentered_pearson',
                                                                 'Spearman Correlation':'spearman',
                                                                 "Kendall's Tau": 'kendall',
                                                                 'Cosine distance':'cosine',
                                                                }
                                                     },
                            "row_distance_metric":{
                                                      "default": "pearson",
                                                      "choices":{'Information Coefficient':"information_coefficient",
                                                                 'City Block (Manhattan or L1-norm)':'manhattan',
                                                                 'Euclidean (L2-norm)':"euclidean",
                                                                 'Pearson Correlation':"pearson",
                                                                 'Uncentered Pearson Correlation':'uncentered_pearson',
                                                                 'Uncentered Pearson Correlation, absolute value':'absolute_uncentered_pearson',
                                                                 'Spearman Correlation':'spearman',
                                                                 "Kendall's Tau": 'kendall',
                                                                 'Cosine distance':'cosine',
                                                                }
                                                     },
                            "clustering_method":{
                                                  "default": "average",
                                                  "choices":{'Pairwise average-linkage':"average",
                                                             'Pairwise complete-linkage':'complete',
                                                             'Pairwise ward-linkage':'ward',
                                                            }
                                                     },
                            "row_centering":{
                                              "default": "Mean",
                                              "choices":{'Subtract the mean from each row':"Mean",
                                                         'Subtract the median from each row':'Median',
                                                         'No':None,
                                                        }
                                                 },
                            "col_centering":{
                                              "default": "Mean",
                                              "choices":{'Subtract the mean from each column':"Mean",
                                                         'Subtract the median from each column':'Median',
                                                         'No':None,
                                                        }
                                                 },
                            "custom_plot":{
                                              "default": "Both",
                                              "choices":{'Plot dendrogram only for samples':"Samples",
                                                         'Plot dendrogram only for genes':'Genes',
                                                         'Plot dendrogram for both samples and genes':'Both',
                                                        }
                                                 },

                        })_____no_output_____


# Notebook from vishalbelsare/teaching
Path: sarscov2/protein_annotation.ipynb

# Protein annotation SARS-CoV-2

This notebook uses Python tools to find all amino acid mutations in SARS-CoV-2 protein sequences.

## Dependencies:

You need a package I made called pathogenie to run this code.

Install with `pip install -e git+https://github.com/dmnfarrell/pathogenie.git#egg=pathogenie`

## Links:

* https://github.com/dmnfarrell/pathogenie
* https://www.gisaid.org/
* http://virological.org/t/selection-analysis-of-gisaid-sars-cov-2-data/448/2

This notebook is available at https://github.com/dmnfarrell/teaching/blob/master/sarscov2/protein_annotation.ipynb_____no_output_____

import os, math, time, pickle, subprocess
from importlib import reload
from collections import OrderedDict, defaultdict
import numpy as np
import pandas as pd
pd.set_option('display.width', 150)
from Bio import SeqIO,AlignIO
from Bio.Seq import Seq
from Bio.SeqRecord import SeqRecord
import pylab as plt
import pathogenie_____no_output_____sc2ref = pathogenie.genbank_to_dataframe('NC_045512.2.gb',cds=True).drop_duplicates('locus_tag')
sc2ref[['gene','locus_tag','length','start']]_____no_output_____

## annotate reference to check match with genbank one_____no_output_____

sc2,sc2recs = pathogenie.run_annotation('NC_045512.fa', kingdom='viruses')
sc2[['gene','product','length','start']]/home/damien/.config/pathogenie/prokka/IS.fa
blasting 10 ORFs to IS
blastp -out tempseq_blast.txt -outfmt "6 qseqid sseqid qseq sseq pident qcovs length mismatch gapopen qstart qend sstart send evalue bitscore stitle" -query tempseq.fa -db /home/damien/.config/pathogenie/prokka/IS.fa -evalue 1e-10 -max_target_seqs 1 -num_threads 4
/home/damien/.config/pathogenie/prokka/amr.fa
blasting 10 ORFs to amr
blastp -out tempseq_blast.txt -outfmt "6 qseqid sseqid qseq sseq pident qcovs length mismatch gapopen qstart qend sstart send evalue bitscore stitle" -query tempseq.fa -db /home/damien/.config/pathogenie/prokka/amr.fa -evalue 1e-100 -max_target_seqs 1 -num_threads 4
/home/damien/.config/pathogenie/prokka/sprot_viruses.fa
blasting 10 ORFs to sprot_viruses
blastp -out tempseq_blast.txt -outfmt "6 qseqid sseqid qseq sseq pident qcovs length mismatch gapopen qstart qend sstart send evalue bitscore stitle" -query tempseq.fa -db /home/damien/.config/pathogenie/prokka/sprot_viruses.fa -evalue 0.0001 -max_target_seqs 1 -num_threads 4
3 sequences unassigned
running aragorn
done
spikeref = sc2[sc2.gene=='S'].iloc[0]
spikeref.translation
refrec = SeqRecord(Seq(spikeref.translation),id='ref')_____no_output_____

## Load unique GIS sequences _____no_output_____

#gistable = pd.read_excel('gisaid_cov2020_acknowledgement_table.xls')

def load_deduplicated_sequences(filename):
    """Load a fasta file of sequences and ignore duplicates"""

    newrecs = {}
    unique_seqs=defaultdict(list)
    with open(filename, 'r') as in_handle:
        for rec in SeqIO.parse(in_handle, "fasta"):
            if rec.seq in unique_seqs:
                continue
            if not rec.id in newrecs:
                try:
                    id = rec.id.split('|')[1]
                    newrecs[id] = rec
                    unique_seqs[str(rec.seq)] = id
                except:
                    pass
    return newrecs

gisrecs = load_deduplicated_sequences('gisaid_cov2020_sequences.fasta')
print (len(gisrecs))/usr/local/lib/python3.8/dist-packages/Bio/Seq.py:175: BiopythonWarning: Biopython Seq objects now use string comparison. Older versions of Biopython used object comparison. During this transition, please use hash(id(my_seq)) or my_dict[id(my_seq)] if you want the old behaviour, or use hash(str(my_seq)) or my_dict[str(my_seq)] for the new string hashing behaviour.
  warnings.warn(
k = list(gisrecs.keys())[:2000]
annot = pathogenie.annotate_files(gisrecs, keys=k, outdir='gisaid_annot')_____no_output_____

## Extract protein of interest across all genomes_____no_output_____

spike_seqs = pathogenie.get_similar_sequences('Spike glycoprotein', annot)
print (len(spike_seqs))1998

## Collapse to unique protein sequences_____no_output_____

unique_seqs, counts = pathogenie.collapse_sequences(spike_seqs, refrec)
print (len(unique_seqs))117

## Find mutations_____no_output_____

def find_mutations(recs, ref):
    """Find the mutations in a set of protein records relative to a
    reference protein sequence"""

    mutations = {}
    positions = []
    for rec in recs:
        aln = pathogenie.clustal_alignment(seqs=[ref, rec])
        #print (aln)
        x = []
        for pos in range(len(aln[0])):
            refaa = aln[0,pos]
            aa = aln[1,pos]
            if aa != refaa and aa!='-':
                #print (refaa, aln[:,pos], aa)
                mut = refaa+str(pos+1)+aa
                x.append(mut)
        if len(x)>0:
            mutations[rec.seq] = x
    return mutations

mutations = find_mutations(unique_seqs, refrec)
mutations.values()_____no_output_____muts = {seq:'+'.join(mutations[seq]) for seq in mutations if counts[seq]>1}
freqs = [(muts[seq], counts[seq]) for seq in muts]
freqs
fdf = pd.DataFrame(freqs,columns=['mutation','count']).sort_values('count',ascending=False)_____no_output_____

## make a phylogenetic tree from protein sequences_____no_output_____

def make_tree(infile, kind='nucl'):

    #aln = tools.clustal_alignment(seqs=seqs)
    cmd = 'mafft --retree 1 %s > aligned.fasta' %infile
    print (cmd)
    subprocess.check_output(cmd, shell=True)
    #AlignIO.write(aln, 'aligned.fasta', 'fasta')
    AlignIO.convert("aligned.fasta", "fasta", "aligned.phy", "phylip-relaxed")
    if kind=='nucl':
        cmd = 'mpirun -np 4 phyml-mpi -i aligned.phy -b 12 -m HKY85 -a e'
    else:
        cmd = 'mpirun -np 4 phyml-mpi -i aligned.phy -d aa -b 12 -m WAG -a e'
    tmp = subprocess.check_output(cmd, shell=True)
    return _____no_output_____#write dict with mutations to a fasta file
temp={}
for s in muts:
    rec=SeqRecord(s,id=muts[s])
    temp[muts[s]] = rec
SeqIO.write(temp.values(),'temp_mutants.faa','fasta')
make_tree('temp_mutants.faa',kind='prot')_____no_output_____from Bio import Phylo
tree = Phylo.read("aligned.phy_phyml_tree.txt", "newick")
Phylo.draw_ascii(tree)_____no_output_____import nglview
w = nglview.show_file("model_spike.pdb")
w.add_representation('licorice', selection=positions, color='green')
w_____no_output_____

## Run all proteins using the above methods_____no_output_____

print (annot.groupby(['product','gene']).agg({'sequence':np.size}).reset_index())                           product gene  sequence
0  Envelope small membrane protein    E        41
1                 Membrane protein    M      1999
2                    Nucleoprotein    N      1997
3                       Protein 3a   3a      1991
4                       Protein 7a   7a      1996
5                       Protein 9b   9b        42
6         Replicase polyprotein 1a   1a      2003
7        Replicase polyprotein 1ab  rep      2000
8               Spike glycoprotein    S      1999
#k = list(gisrecs.keys())[:500]
#annot = app.annotate_files(gisrecs, outdir='gisaid_annot', kingdom='viruses')

names = ['Protein 7a', 'Protein 3a', 'Spike glycoprotein','Membrane protein',
         'Nucleoprotein','Replicase polyprotein 1a','Replicase polyprotein 1ab']
res=[]
mutant_seqs = {}
for protname in names:
    refprot = sc2[sc2['product']==protname].iloc[0]
    refrec = SeqRecord(Seq(refprot.translation),id='ref')
    print (protname)
    annot_seqs = pathogenie.get_similar_sequences(protname, annot)
    unique_seqs, counts = pathogenie.collapse_sequences(annot_seqs, refrec)
    print ('%s unique sequences' %len(unique_seqs))
    mutations = get_mutations(unique_seqs, refrec)
    #convert mutations to string and count the frequency
    muts = {seq:'+'.join(mutations[seq]) for seq in mutations}
    #save the sequences for later use
    mutant_seqs[protname] = muts
    freqs = [(muts[seq], counts[seq]) for seq in muts]
    #convert freq table to dataframe
    fdf = pd.DataFrame(freqs,columns=['mutation','count']).sort_values('count',ascending=False)
    fdf['protein'] = protname
    print (fdf[:10])
    res.append(fdf)

res = pd.concat(res).sort_values('count',ascending=False)
res.to_csv('sarscov2_mutations.csv',index=False)_____no_output_____#fdf = pd.DataFrame(freqs,columns=['mutation','count']).sort_values('count',ascending=False)
res[:50].plot(x='mutation',kind='bar',width=.9,figsize=(10,4))
plt.tight_layout()
plt.savefig('sarscov2_mut_freq.png',dpi=100)
#res[:20]_____no_output_____


# Notebook from foxan/dataquest
Path: Data Analysis with Pandas - Intermediate/Challenge - Summarizing Data.ipynb

https://www.dataquest.io/mission/113/challenge-summarizing-data/_____no_output_____## 2: College Majors And Employment

The American Community Survey is a survey run by the US Census Bureau that collects data on everything from the affordability of housing to employment rates for different industries. For this challenge, you'll be using the data derived from the American Community Survey for years 2010-2012. The team at FiveThirtyEight has cleaned the dataset and made it available on their [Github repo](https://github.com/fivethirtyeight/data/tree/master/college-majors).

Here's a quick overview of the files we'll be working with:

[all-ages.csv](https://github.com/fivethirtyeight/data/blob/master/college-majors/all-ages.csv) - employment data by major for all ages

[recent-grads.csv](https://github.com/fivethirtyeight/data/blob/master/college-majors/recent-grads.csv) - employment data by major for just  recent college graduates
_____no_output_____

# %sh
# # download source file
# wget https://raw.githubusercontent.com/fivethirtyeight/data/master/college-majors/all-ages.csv
# wget https://raw.githubusercontent.com/fivethirtyeight/data/master/college-majors/recent-grads.csv
# ls -l_____no_output_____import pandas as pd

all_ages = pd.read_csv("all-ages.csv")
print all_ages.columns
print all_ages.head(3)

recent_grads = pd.read_csv("recent-grads.csv")
print recent_grads.columns
print recent_grads.head(3)Index([u'Major_code', u'Major', u'Major_category', u'Total', u'Employed',
       u'Employed_full_time_year_round', u'Unemployed', u'Unemployment_rate',
       u'Median', u'P25th', u'P75th'],
      dtype='object')
   Major_code                                  Major  \
0        1100                    GENERAL AGRICULTURE
1        1101  AGRICULTURE PRODUCTION AND MANAGEMENT
2        1102                 AGRICULTURAL ECONOMICS

                    Major_category   Total  Employed  \
0  Agriculture & Natural Resources  128148     90245
1  Agriculture & Natural Resources   95326     76865
2  Agriculture & Natural Resources   33955     26321

   Employed_full_time_year_round  Unemployed  Unemployment_rate  Median  \
0                          74078        2423           0.026147   50000
1                          64240        2266           0.028636   54000
2                          22810         821           0.030248   63000

   P25th    P75th
0  34000  80000.0
1  36000  80000.0
2  40000  98000.0
Index([u'Rank', u'Major_code', u'Major', u'Major_category', u'Total',
       u'Sample_size', u'Men', u'Women', u'ShareWomen', u'Employed',
       u'Full_time', u'Part_time', u'Full_time_year_round', u'Unemployed',
       u'Unemployment_rate', u'Median', u'P25th', u'P75th', u'College_jobs',
       u'Non_college_jobs', u'Low_wage_jobs'],
      dtype='object')
   Rank  Major_code                           Major Major_category  Total  \
0     1        2419           PETROLEUM ENGINEERING    Engineering   2339
1     2        2416  MINING AND MINERAL ENGINEERING    Engineering    756
2     3        2415       METALLURGICAL ENGINEERING    Engineering    856

   Sample_size   Men  Women  ShareWomen  Employed      ...        Part_time  \
0           36  2057    282    0.120564      1976      ...              270
1            7   679     77    0.101852       640      ...              170
2            3   725    131    0.153037       648      ...              133

   Full_time_year_round  Unemployed  Unemployment_rate  Median  P25th   P75th  \
0                  1207          37           0.018381  110000  95000  125000
1                   388          85           0.117241   75000  55000   90000
2                   340          16           0.024096   73000  50000  105000

   College_jobs  Non_college_jobs  Low_wage_jobs
0          1534               364            193
1           350               257             50
2           456               176              0

[3 rows x 21 columns]

## 3: Summarizing Major Categories

In both of these datasets, majors are grouped into categories. There are multiple rows with a common value for `Major_category` but different values for `Major`. We would like to know the total number of people in each `Major_category` for both datasets._____no_output_____

all_ages_major_categories = {}
recent_grads_major_categories = {}

def calculate_major_cat_totals(df):
  counts_dictionary = {}
  for cat in df["Major_category"].value_counts().index:
    counts_dictionary[cat] = df["Total"][df["Major_category"] == cat].sum()
  return counts_dictionary

all_ages_major_categories = calculate_major_cat_totals(all_ages)
recent_grads_major_categories = calculate_major_cat_totals(recent_grads)

print all_ages_major_categories
print recent_grads_major_categories{'Arts': 1805865L, 'Psychology & Social Work': 1987278L, 'Business': 9858741L, 'Industrial Arts & Consumer Services': 1033798L, 'Computers & Mathematics': 1781378L, 'Agriculture & Natural Resources': 632437L, 'Interdisciplinary': 45199L, 'Humanities & Liberal Arts': 3738335L, 'Engineering': 3576013L, 'Biology & Life Science': 1338186L, 'Health': 2950859L, 'Law & Public Policy': 902926L, 'Physical Sciences': 1025318L, 'Education': 4700118L, 'Communications & Journalism': 1803822L, 'Social Science': 2654125L}
{'Arts': 357130L, 'Psychology & Social Work': 481007L, 'Business': 1302376L, 'Industrial Arts & Consumer Services': 229792L, 'Computers & Mathematics': 299008L, 'Agriculture & Natural Resources': 79981L, 'Interdisciplinary': 12296L, 'Humanities & Liberal Arts': 713468L, 'Engineering': 537583L, 'Biology & Life Science': 453862L, 'Health': 463230L, 'Law & Public Policy': 179107L, 'Physical Sciences': 185479L, 'Education': 559129L, 'Communications & Journalism': 392601L, 'Social Science': 529966L}

## 4: Low Wage Jobs Rates

The [press likes to talk a lot](http://bit.ly/1fNLmaT) about how many college grads are unable to get higher wage, skilled jobs and end up working lower wage, unskilled jobs instead. As a data person, it is your job to be skeptical of any broad claims and analyze relevant data to obtain a more nuanced view. Let's run some basic calculations to explore that idea further._____no_output_____

low_wage_percent = recent_grads["Low_wage_jobs"].astype(float).sum() / recent_grads["Total"].sum()

print low_wage_percent0.0985254607612

## 5: Comparing Datasets

Both `all_ages` and `recent_grads` datasets have 173 rows, corresponding to the 173 college major codes. This enables us to do some comparisons between the two datasets and perform some initial calculations to see how similar or different the statistics of recent college graduates are from those of the entire population._____no_output_____

# All majors, common to both DataFrames
majors = recent_grads['Major'].value_counts().index
recent_grads_lower_emp_count = 0
all_ages_lower_emp_count = 0

for major in majors:
  recent_unemp = recent_grads["Unemployment_rate"][recent_grads["Major"] == major].values[0]
  all_unemp = all_ages["Unemployment_rate"][all_ages["Major"] == major].values[0]
  if recent_unemp < all_unemp:
    recent_grads_lower_emp_count += 1
  elif recent_unemp > all_unemp:
    all_ages_lower_emp_count += 1

print "Recent grads fare better: ", recent_grads_lower_emp_count
print "All ages fare better: ", all_ages_lower_emp_countRecent grads fare better:  43
All ages fare better:  128


# Notebook from openworm/neuronal-analysis
Path: laboratory/PCA Next Steps.ipynb

###### This code is deliberately ad-hoc. Instead of making things difficult for myself by writing libraries and modules, etc, I'm writing code as a I go and only afterwards if need be will refactor by modularizing everything. My focus is on scientific insight, not on maintaining a codebase and the extra effort is simply not worth the mental energy and time spent organizing everything.

Additionally, letting loose with my coding practices is an important step in my development. I'm learning how to be a fearless experimenter by doing things against my nature. I've previously been really tight about modularizing and organizing my code and never writing ad-hoc scripts, instead crusting over my code with abstractions upon abstractions upon abstractions. No longer will this be! Sometimes you need to solve specific problems with specific code and rely on the power of the tools you work on and your own ability to craft new solutions with them at ease.

So this explains the erratic nature of the code in my project. It's me fearlessly experimenting with science and math.

I've come to appreciate how developments are made in science and math. Very intelligent and curious people perceive a void, a possibility, a need which leads them to asking questions as to how that void can be filled. They start rigorously testing solutions and looking at all the nuances of their solutions, refining and refining and testing and testing until they've arrived at something substancial and versatile.

For instance, Principal Component Analysis is the subject of much literature - because it is so nuanced. There are so many factors that come into play and the technique didn't just appear. It needed to be developed, tested and refined and be critiqued and worked on by brilliant people.

This leads me to my new goal to become a fearless experimenter. I need to get past that barrier of inexpertise and rigorously play around with the new mathematical, statistical, and scientific ideas I learn. That means playing around with PCA, with graph theory, etc. It means not being afraid to embrace complex new ideas by engaging with them at the deepest level possible. _____no_output_____

import sys
# Add the ptdraft folder path to the sys.path list
sys.path.append('../src/')

import data_config as dc
wormData = dc.kato.data()

# Importing the transform code
import transform as tf
import dimensions_kit as dk
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import gridspec/Users/akivalipshitz/Developer/neuronal-analysis/data/wbdata/
transformed = []
pca = []
predicted = []

for worm, data in wormData.iteritems():
    wd_pca, wd_transformed = tf.scikit_pca(data['deltaFOverF_deriv'], 3)

    #Storing this for later use
    transformed.append(wd_transformed)
    pca.append(wd_pca)
    """
    Using our PC's we can increase the dimensionality of
        our reduced data, from 3 to 107, thus getting a blurred
        dataset as predicted by PC's 1 and 2. It contains
        most of the information contained in the original
        data

    The operation is very simple - a transformation
        of 3 dimensional data to 107 dimensional data as the
        dot product of the components and transformed data
    """
    predicted.append(np.dot(wd_pca.components_.T, wd_transformed.T))

# How principle components are correlated with eachother
# Correlated: <>0, Not: ~ 0
# Tells you waht is common among the principle components_____no_output_____len(predicted)_____no_output_____def plot_all_predicted():
    fig = plt.figure(figsize=(10,10))
    gs = gridspec.GridSpec(3,2)
    for i in range(0,len(predicted)+1):
        # So we can layout our graphs
        # nicely in two dimensions
        # we need to map n <- R^1 to n'<-R^2
        x,y = dk.transform((3,2), i)
        ax = fig.add_subplot(gs[x,y])
        ax.set_title('Deriv-PCA Predicted ' + str(i))
        ax.pcolor(predicted[i-1])
    _____no_output_____"""
Here we will look at how the various datasets
compare with eachother.

This enables us to answer the question of whether the same neurons have
similar PC strengths in all datasets.
"""

# Here we plot the difference between two datasets
# The variance is fairly uniform except for at firing points
# where the pariance peaks
pred = predicted[0]
orig = wormData[0]['deltaFOverF_deriv']
diff = orig.T-pred
diff.shape

# This transforms a 2d array into a set of points
pts = np.array([[x,y,diff[x,y]] for x in range(diff.shape[0]) for y in range(diff.shape[1]) ])_____no_output_____"""
This shows you the variance between the predicted data
and the original derivative data
"""

%matplotlib inline
from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure()
ax = fig.add_subplot(111)
# ax.plot(pts[:,0], pts[:,1],pts[:,2])

ax.pcolormesh(diff*1000)
fig.show()/Library/Python/2.7/site-packages/matplotlib/figure.py:397: UserWarning: matplotlib is currently using a non-GUI backend, so cannot show the figure
  "matplotlib is currently using a non-GUI backend, "

This tells us that the variance is fairl uniform except for at critical points where
it scales by some large factor.
This makes sense - the PC would scale large points to make them outliers.

The next step suggested is:

## Comparing Adjacent PCs Across Datasets.

To do this, we need to line up all neurons shared in two datasets. We first need to ensure that doing this is worthwhile. _____no_output_____

# So that we can compare every worm with every worm,
# we use combinations. (Binomial Coefficient, etc.)
from itertools import combinations
groupings  = [ i for i in combinations(range(5), r=2)]

# Tells us which neurons are present in the nth dataset
def get_neurons_present():
    neuron_names = [
        { k[0]: idx
        for idx, k in enumerate(wormData[i]['NeuronIds'])
        if k != None and (len(k)>0) }
        # Forall worms
        for i in range(len(wormData)) ]
    return neuron_names

neurons_present = [
    filter(lambda x: x!=None,i[:][0])
    for i in wormData.T['NeuronIds'].as_matrix()]

neuron_names_present = [set(ns) for ns in neurons_present]

# I used this stackoverflow to get this way of finding
# the intersection of sets: http://bit.ly/1RiV6FE
all_shared_neurons = set.intersection(*neuron_names_present)

# similars[(i,j)] = neurons shared between ith and jth dataset

similars = {}
for i,j in groupings:
    ns_i = neurons_present[i]
    ns_j = neurons_present[j]

    # This makes accessing things easier.
    # You don't need to worry about the order.
    similars[(i,j)] = similars[(j,i)] = set.intersection(set(ns_i), set(ns_j))
20

We've just set up a framework for comparing neuron sharing across our datasets. Now let's use it to compute some preliminary metrics. _____no_output_____

all_possible_shared = list(set.union(*neuron_names_present))

print "Total number neurons present in > 2 datasets: " + str(len(all_possible_shared))

# Percentage of comparable neurons of the whole dataset
print str(int(float(len(all_possible_shared))/302*100)) + "%"64
Total number neurons present in > 2 datasets: 64
21%

Because there are so few shared neurons, I'm not sure what kind of insight we can get by comparing the principle components for each neuron.

Though after stepping back for a moment, I realized we can still get some insight if the shared neurons play important roles. Additionally, just comparing PCs for neurons across two datasets can provide some insight. So instead of comparing across all datasets, we can only compare across $x\geq2$ datasets.

If we take this route, how many comparable neurons are there? **65**

What percentage of the whole dataset are they? **21**

What if we don't need to compare neurons shared across the entire dataset, but only across two readings?_____no_output_____# Neurons Shared by Paired Datasets

Now I'm going to do something very cool using matplotlib_venn. I'm going to visualize the intersection of our datasets. I'm sure we can get a better metric than just percentage neurons shared. _____no_output_____

plt.clf()
%matplotlib inline
import numpy as np
from matplotlib_venn import *

import dimensions_kit as dk

dimensions = dk.dimensions(len(groupings))
f, axes = plt.subplots(*dimensions,figsize=(10,10))

f.suptitle('Overlapping neurons collected in datasets j and i', fontsize=20)
n=0
for k,i in groupings:

    names_k = neuron_names_present[k]
    names_i = neuron_names_present[i]

    x,y = dk.transform(dimensions,n)

    axis = axes[x][y]
    v = venn2(subsets=(names_k, names_i), ax=axis)

    intersection = float(len(names_k.intersection(names_i)))
    ttl = len(names_k)+len(names_i)-intersection
    ratio = int(intersection/ttl*100)

    axis.set_title('Dataset ${0}$ vs ${1}$: {2}% Overlap'.format(k,i,ratio ))
    plt.setp(axis.get_xticklabels(), visible=False)
    plt.setp(axis.get_yticklabels(), visible=False)

    n+=1

while n < dimensions[0]*dimensions[1]:
    x,y = dk.transform(dimensions,n)
    axes[x][y].axis('off')
    n+=1
_____no_output_____

## Starting to Compare The Two Datasets

Here is a list of ideas to consider. A set of very simple yet concrete questions we can go and answer.

Originally I was unsure of how to look at complex datasets such as these. Once I had my revelation related to mathematics I recognized progress is driven by asking larger questions and breaking them down into smaller questions. Essentially, I'm looking for relationships - ways of relating dynamics to structure or structural patterns. Before I get into any serious graph theory or dynamical systems, or dynamical graph theory, I'm seeking inspiration from empirical data. Thus I start the process of developing a higher order theory by making observations on reality itself.

My eventual goal is to start playing around with the pure mathematical ideas that arise from this experimental fishing expedition. I'll play around with the mathematical models I develop to build dynamical graphs with a goal in mind.

First, I need to understand the empirical data. So here are some smaller questions I need to answer.

- How many shared neurons play a significant role in a significant PC?
- Are these roles consistent across datasets?
- If any inconsistencies: why?
- Is classifying neurons consistent with their expected effect on global dynamics, i.e. muscle neurons, sensory neurons, etc.
- Correlation between neuron degree and PC weight?

> I also feel compelled to mention my hazy revelation relating to NKS - production rules, and predicting dynamics. Well - simple things do create very complex, seemingly random behavior, but can some properties of this behavior be quantified - as a discrete dynamical system? Just a thought.

Here are the components for all the datasets. _____no_output_____

components = [np.array(pc.components_) for pc in pca]
components_____no_output_____all_shared_neurons_____no_output_____

Now we are going to use K-Means clustering     _____no_output_____## Beyond $[$vanilla$]$ PCA

- Dynamical graph theory
- Global neural network
- Markov chain/manifold signature
- Derivative of dynamics to remove instance theory related problems

**Papers**:

- http://www.alainconnes.org/docs/reconstructionshort.pdf
- *PRINCIPAL MANIFOLDS AND GRAPHS IN PRACTICE: FROM MOLECULAR BIOLOGY TO DYNAMICAL SYSTEMS*: http://arxiv.org/pdf/1001.1122.pdf
- *Discontinuous Dynamical Systems*: http://arxiv.org/pdf/0901.3583v1.pdf
- *Data Analysis Techniques for Nonlinear Dynamical Systems*: http://www.diee.unica.it/DRIEI/tesi/22_camplani.pdf
- *A Learning Rule for the Emergence of Stable Dynamics and Timing in Recurrent Networks*: https://www.neurobio.ucla.edu/~dbuono/PDFs/BuonoJNphy05.pdf
- *Dynamical systems and time series*: https://openaccess.leidenuniv.nl/bitstream/handle/1887/14735/muskulus-chapter-2.pdf?sequence=16
- *A view of Neural Networks as dynamical systems.*: http://arxiv.org/pdf/0901.2203v2.pdf
- *An Introduction to Dynamical Systems and Chaos*: http://www.ldeo.columbia.edu/~mspieg/Complexity/Problems.pdf
- *Metastable behavior in Markov processes with internal states*: http://arxiv.org/pdf/1304.6957v2.pdf_____no_output_____3/17/16 At the NeuroKernel hackathon I've gotten lots of *sensory input* which brought me to conceptualize new approaches and guiding principles to what I'm doing.
t
I need to think about the whole picture and not get stuck with the little things that I'm doing–PCA, sticking to the Kato data, focusing solely on OpenWorm, etc. That is why speaking with people is so important–because it provides either confirmation or refutation of my own ideas but more importantly, exposure to new ones; super valuable insight. Collaboration is **sooo** key to research.

**That is why this Neuro-Arch project is so important ** Research exponentiates when you can ask questions and query data really quickly.

After speaking at length with Dave and revisiting graph theory, I've been thinking about how to connect what I have been doing now/recently with the **Kato** data. How will it bring published data under it's umbrella?

Two things just came to my mind, the first I'm recording for the sake of completeness and the second because it is relevant to the below:
**Urgency Fallacy** - doing less important things to feign progress to yourself when it would serve you, your learning, and your research better if you spend more time delving into the meaty issues instead of the less important issues.
**Empty Hypothesis Fallacy** when you invalidate all options without realizing that your hypothesis space was just too limited.

1. I've realized that when originally I sought to describe network level dynamics I didn't quite perceive what that meant in concrete terms. Now I understand you must sometimes look at each individual component and from that build an understanding of the entire system.
2. I'm going to see how static network properties map to dynamics and see if this holds across all 5 Kato datasets. This means comparing some static network properties

Then, I can create hypotheses from my learnings here and attempt to validate them.

One of the important things I learned at the conference from the researcher from Taiwan is that connectivity doesn't imply functionality. While his model was anatomically correct, it did not account for variation and specialization  It is really important to know the physiological mechanisms (that is what the experimentalists are for) which the theorists can then incorporate in their models either as pluginable parameters or ones central to the development of the model.

3/23/16 First of all, I've made significant progress with OpenWorm. They are now discussing creating a new openworm project for spike train analysis.

But getting back to the science, I've been wondering how best to compare these spike trains – which I'm hoping will lead me to some theories I can try to verify computationally, i.e. I'm now still in the exploration phase. I found [this](http://mathematical-neuroscience.springeropen.com/articles/10.1186/s13408-015-0030-9#CR29) paper and asked [this](http://cogsci.stackexchange.com/questions/13617/how-to-interpret-neuron-spiking-models-one-comes-across-in-literature) on binwise, probabalistic comparisions of neural spike trains beyond pairwise correlations. I'm going to spend more time digging through this research and reading up about the various methods Josh Merel told me about beyond vanilla PCA. _____no_output_____