# Notebook from Liyubov/einet
Path: code/Chapter 06 - Causal Emergence and the Emergence of Scale.ipynb

from ei_net import *
from ce_net import *
import matplotlib.pyplot as plt

import datetime as dt

%matplotlib inline_____no_output_____##########################################
############ PLOTTING SETUP ##############
EI_cmap = "Greys"
where_to_save_pngs = "../figs/pngs/"
where_to_save_pdfs = "../figs/pdfs/"
save = True
plt.rc('axes', axisbelow=True)
##########################################
##########################################_____no_output_____

# Uncertainty and causal emergence in complex networks_____no_output_____# Chapter 06 - Causal Emergence and the Emergence of Scale

## Network Macroscales
First we must introduce how to recast a network, $G$, at a higher scale. This is represented by a new network, $G_M$. Within $G_M$, a micro-node is a node that was present in the original $G$, whereas a macro-node is defined as a node, $v_M$, that represents a subgraph, $S_i$, from the original $G$ (replacing the subgraph within the network). Since the original network has been dimensionally reduced by grouping nodes together, $G_M$ will always have fewer nodes than $G$.

A macro-node $\mu$ is defined by some $W^{out}_{\mu}$, derived from the edge weights of the various nodes within the subgraph it represents. One can think of a macro-node as being a summary statistic of the underlying subgraph's behavior, a statistic that takes the form of a single node. Ultimately there are many ways of representing a subgraph, that is, building a macro-node, and some ways are more accurate than others in capturing the subgraph's behavior, depending on the connectivity. To decide whether or not a macro-node is an accurate summary of its underlying subgraph, we check whether random walkers behave identically on $G$ and $G_M$. We do this because many important analyses and algorithms---such as using PageRank for determining a node's centrality or InfoMap for community discovery---are based on random walking.

Specifically, we define the *inaccuracy* of a macroscale as the Kullback-Leibler divergence between the expected distribution of random walkers on $G$ vs. $G_M$, given some identical initial distribution on each. The  expected distribution over $G$ at some future time $t$ is $P_m(t)$, while the distribution over $G_M$ at some future time $t$ is $P_M(t)$. To compare the two, the distribution $P_m(t)$ is summed over the same nodes in the macroscale $G_M$, resulting in the distribution $P_{M|m}(t)$ (the microscale given the macroscale). We can then define the macroscale inaccuracy over some series of time steps $T$ as:

$$ \text{inaccuracy} = \sum_{t=0}^T \text{D}_{_{KL}}[P_{M}(t) || P_{M|m}(t)] $$

This measure addresses the extent to which a random dynamical process on the microscale topology will be recapitulated on a dimensionally-reduced topology.

What constitutes an accurate macroscale depends on the connectivity of the subgraph that gets grouped into a macro-node. The $W^{out}_{\mu}$ can be constructed based on the collective $W^{out}$ of the subgraph. For instance, in some cases one could just coarse-grain a subgraph by using its average $W^{out}$ as the $W^{out}_{\mu}$ of some new macro-node $\mu$. However, it may be that the subgraph has dependencies not captured by such a coarse-grain. Indeed, this is similar to the recent discovery that when constructing networks from data it is often necessary to explicitly model higher-order dependencies by using higher-order nodes so that the dynamics of random walks to stay true to the original data. We therefore introduce *higher-order macro-nodes* (HOMs), which draw on similar techniques to accurately represent subgraphs as single nodes.
_________________no_output_____

- Top: The original network, $G$ along with its adjacency matrix (left). The shaded oval indicates that subgraph $S$ member nodes $v_B$ and $v_C$ will be grouped together, forming a macro-node, ${\mu}$. All macro-nodes are some transformation of the original adjacency matrix via recasting it as a new adjacency matrix (right). The manner of this recasting depends on the type of macro-node.
- Bottom left: The simplest form of a macro-node is when $W^{out}_{\mu}$ is an average of the $W^{out}_{i}$ of each node in the subgraph.
- Bottom center left: A macro-node that represents some path-dependency, such as input from $A$. Here, in averaging to create the $W^{out}_{\mu}$ the out-weights of nodes $v_B$ and $v_C$ are weighted by their input from  $v_A$.
- Bottom center right: A macro-node that represents the subgraph's output over the network's stationary dynamics. Each node has some associated ${\pi}_{i}$, which is the probability of ${v}_{i}$ in the stationary distribution of the network. The $W^{out}_{\mu}$ of a $\mu | \pi$ macro-node is created by weighting each $W^{out}_{i}$ of the micro-nodes in the subgraph $S$ by $\frac{{\pi}_{i}}{\sum_{k \in S} {\pi}_{k}}$.
- Bottom right: A macro-node with a single timestep delay between input $\mu | j$ and its output $\mu | \pi$, each constructed using the same techniques as its components. However, $\mu | j$ always deterministically outputs to $\mu | \pi$. _____no_output_____Different subgraph connectivities require different types of HOMs to accurately represent. For instance, HOMs can be based on the input weights to the macro-node, which take the form $\mu | j$. In these cases the $W^{out}_{\mu|j}$ is a weighted average of each node's $W^{out}$ in the subgraph, where the weight is based on the input weight to each node in the subgraph. Another type of HOM that generally leads to accurate macro-nodes over time is when the $W^{out}_{\mu}$ is based on the stationary output from the subgraph to the rest of the network, which we represent as $\mu | \pi$. These types of HOMs may sometimes have minor inaccuracies given some initial state, but will almost always trend toward perfect accuracy as the network approaches its stationary dynamics.

Subgraphs with complex internal dynamics can require a more complex type of HOM in order to preserve the network's accuracy. For instance, in cases where subgraphs have a delay between their inputs and outputs, this can be represented by a combination of $\mu | j$ and $\mu | \pi$, which when combined captures that delay. In these cases the macro-node $\mu$ has two components, one of which acts as a buffer over a timestep. This means that macro-nodes can possess memory even when constructed from networks that are at the microscale memoryless, and in fact this type of HOM is sometimes necessary to accurately capture the microscale dynamics.

We present these types of macro-nodes not as an exhaustive list of all possible HOMs, but rather as examples of how to construct higher scales in a network by representing subgraphs as nodes, and also sometimes using higher-order dependencies to ensure those nodes are accurate. This approach offers a complete generalization of previous work on coarse-grains and also black boxes, while simultaneously solving the previously unresolved issue of macroscale accuracy by using higher-order dependencies. The types of macro-nodes formed by subgraphs also provides substantive information about the network, such as whether the macroscale of a network possesses memory or path-dependency.

## Causal emergence reveals the scale of networks

Causal emergence occurs when a recast network, $G_M$ (a macroscale), has more $EI$ than the original network, $G$ (the microscale). In general, networks with lower effectiveness (low $EI$ given their size) have a higher potential for causal emergence, since they can be recast to reduce their uncertainty. Searching across groupings allows the identification or approximation of a macroscale that maximizes the $EI$.

Checking all possible groupings is computationally intractable for all but the smallest networks. Therefore, in order to find macro-nodes which increase the $EI$, we use a greedy algorithm that groups nodes together and checks if the $EI$ increases. By choosing a node and then pairing it iteratively with its surrounding nodes we can grow macro-nodes until pairings no longer increase the $EI$, and then move on to a new node.

By generating undirected preferential attachment networks and varying the degree of preferential attachment, $\alpha$, we observe a crucial relationship between preferential attachment and causal emergence. One of the central results in network science has been the identification of "scale-free" networks. Our results show that networks that are not "scale-free" can be further separated into micro-, meso-, and macroscales depending on their connectivity. This scale can be identified based on their degree of causal emergence. In cases of sublinear preferential attachment ($\alpha < 1.0$) networks lack higher scales. Linear preferential attachment ($\alpha=1.0$) produces networks that are scale-free, which is the zone of preferential attachment right before the network develops higher scales. Such higher scales only exist in cases of superlinear preferential attachment ($\alpha > 1.0$). And past $\alpha > 3.0$ the network begins to converge to a macroscale where almost all the nodes are grouped into a single macro-node. The greatest degree of causal emergence is found in mesoscale networks, which is when $\alpha$ is between 1.5 and 3.0, when networks possess a rich array of macro-nodes.

Correspondingly the size of $G_M$ decreases as $\alpha$ increases and the network develops an informative higher scale, which can be seen in the ratio of macroscale network size, $N_M$, to the original network size, $N$. As discussed in previous sections, on the upper end of the spectrum of $\alpha$ the resulting network will approximate a hub-and-spoke, star-like network. Star-like networks have higher degeneracy and thus less $EI$, and because of this, we expect that there are more opportunities to increase the network's $EI$ through grouping nodes into macro-nodes. Indeed, the ideal grouping of a star network is when $N_M=2$ and $EI$ is 1 bit. This result is similar to recent advances in spectral coarse-graining that also observe that the ideal coarse-graining of a star network is to collapse it into a two-node network, grouping all the spokes into a single macro-node, which is what happens to star networks that are recast as macroscales.

Our results offer a principled and general approach to such community detection by asking when there is an informational gain from replacing a subgraph with a single node. Therefore we can define *causal communities* as being when a cluster of nodes, or some subgraph, forms a viable macro-node. Fundamentally causal communities represent noise at the microscale. The closer a subgraph is to complete noise, the greater the gain in $EI$ by replacing it with a macro-node. Minimizing the noise in a given network also identifies the optimal scale to represent that network. However, there must be some structure that can be revealed by noise minimization in the first place. In cases of random networks that form a single large component which lacks any such structure, causal emergence does not occur.
_________________no_output_____## 6.1 Causal Emergence in Preferential Attachment Networks_____no_output_____

def preferential_attachment_network(N, alpha=1.0, m=1):
    """
    Generates a network based off of a preferential attachment
    growth rule. Under this growth rule, new nodes place their
    $m$ edges to nodes already present in the graph, G, with
    a probability proportional to $k^\alpha$.

    Params
    ------
    N (int): the desired number of nodes in the final network
    alpha (float): the exponent of preferential attachment.
                   When alpha is less than 1.0, we describe it
                   as sublinear preferential attachment. At
                   alpha > 1.0, it is superlinear preferential
                   attachment. And at alpha=1.0, the network
                   was grown under linear preferential attachment,
                   as in the case of Barabasi-Albert networks.
    m (int): the number of new links that each new node joins
             the network with.

    Returns
    -------
    G (nx.Graph): a graph grown under preferential attachment.

    """
    G = nx.Graph()
    G = nx.complete_graph(m+1)

    for node_i in range(m+1,N):
        degrees = np.array(list(dict(G.degree()).values()))
        probs = (degrees**alpha) / sum(degrees**alpha)
        eijs = np.random.choice(G.number_of_nodes(),
                                size=(m,), replace=False, p=probs)
        for node_j in eijs:
            G.add_edge(node_i, node_j)

    return G_____no_output_____Nvals = sorted([30,60,90,120,150])
alphas= np.linspace(-1,5,25)
Niter = 2

m     = 1
pa_ce = {'alpha'  :[],
         'N_micro':[],
         'N_macro':[],
         'EI_micro':[],
         'EI_macro':[],
         'CE'      :[],
         'N_frac'  :[],
         'runtime' :[]}_____no_output_____

### Note: the following cell was run on a super-computing cluster. It is included as an example computation._____no_output_____

for N in Nvals:
    for alpha in alphas:
        for _ in range(Niter):
            G      = preferential_attachment_network(N,alpha,m)

            startT = dt.datetime.now()
            CE     = causal_emergence(G, printt=False)
            finisH = dt.datetime.now()

            diff   = finisH-startT
            diff   = diff.total_seconds()

            pa_ce['alpha'].append(alpha)
            pa_ce['N_micro'].append(N)
            pa_ce['N_macro'].append(CE['G_macro'].number_of_nodes())
            pa_ce['EI_micro'].append(CE['EI_micro'])
            pa_ce['EI_macro'].append(CE['EI_macro'])
            pa_ce['CE'].append(CE['EI_macro']-CE['EI_micro'])
            pa_ce['N_frac'].append(CE['G_macro'].number_of_nodes()/N)
            pa_ce['runtime'].append(diff)

NCE = pa_ce.copy()_____no_output_____# import cmocean as cmo
# colorz = cmo.cm.amp(np.linspace(0.2,0.9,len(Nvals)))
colorz = plt.cm.viridis(np.linspace(0,0.9,len(Nvals)))_____no_output_____mult=0.95

fig,ax=plt.subplots(1,1,figsize=(5.0*mult,4.5*mult))

plt.subplots_adjust(wspace=0.24, hspace=0.11)
ymax_so_far = 0
xmin_so_far = 0
xmax_so_far = 0

for i,Nn in enumerate(Nvals):
    col = colorz[i]
    means = [np.mean(NCE[Nn][i]['CE']) for i in NCE[Nn].keys()]
    stdvs = [np.std(NCE[Nn][i]['CE'])  for i in NCE[Nn].keys()]
    alphs = list(NCE[Nn].keys())

    alphs = np.array([(alphs[i]+alphs[i+1])/2
                      for i in range(0,len(alphs)-1,2)])
    means = np.array([(means[i]+means[i+1])/2
                      for i in range(0,len(means)-1,2)])
    stdvs = np.array([(stdvs[i]+stdvs[i+1])/2
                      for i in range(0,len(stdvs)-1,2)])

    xmin_so_far = min([xmin_so_far, min(alphs)])
    xmax_so_far = max([xmax_so_far, max(alphs)])
    ymax_so_far = max([ymax_so_far, max(means+stdvs)])

    ax.plot(alphs, means,
            markeredgecolor=col, color=col,
            markerfacecolor='w',
            markeredgewidth=1.5,markersize=5.0,
           linestyle='-',marker='o',linewidth=2.2,label='N = %i'%Nn)

    ax.fill_between(alphs, means-stdvs, means+stdvs,
                    facecolor=col, alpha=0.2,
                    edgecolors='w', linewidth=1)

cols = ["#a7d6ca","#dbb9d1","#d6cdae","#a5c9e3"]
ax.fill_between([-2,0.90],[-1,-1],[3,3],
                facecolor=cols[0],alpha=0.3,edgecolors='w',linewidth=0)
ax.fill_between([0.90,1.1],[-1,-1],[3,3],
                facecolor=cols[1],alpha=0.7,edgecolors='w',linewidth=0)
ax.fill_between([1.1,3.0],[-1,-1],[3,3],
                facecolor=cols[2],alpha=0.3,edgecolors='w',linewidth=0)
ax.fill_between([3.0,6],[-1,-1],[3,3],
                facecolor=cols[3],alpha=0.3,edgecolors='w',linewidth=0)

ax.text(-0.500, 2.65, '|', fontsize=14)
ax.text(0.9425, 2.65, '|', fontsize=14)
ax.text(0.9425, 2.72, '|', fontsize=14)
ax.text(0.9425, 2.79, '|', fontsize=14)
ax.text(2.4000, 2.65, '|', fontsize=14)
ax.text(4.2500, 2.65, '|', fontsize=14)

ax.text(-1.1, 2.81,'microscale',fontsize=12)
ax.text(0.35, 2.95,'scale-free',fontsize=12)
ax.text(1.70, 2.81,'mesoscale',fontsize=12)
ax.text(3.45, 2.81,'macroscale',fontsize=12)

ax.set_ylim(-0.025*ymax_so_far,ymax_so_far*1.05)
ax.set_xlim(-1.075,5*1.01)
ax.set_xlabel(r'$\alpha$',fontsize=14)
ax.set_ylabel('Causal emergence',fontsize=14, labelpad=10)
ax.legend(loc=6,framealpha=0.99)
ax.set_xticks(np.linspace(-1,5,7))
ax.set_xticklabels(["%i"%i for i in np.linspace(-1,5,7)])
ax.grid(linestyle='-', linewidth=2.0, color='#999999', alpha=0.3)

if save:
    plt.savefig(
        where_to_save_pngs+\
        'CE_pa_alpha_labs.png',
        dpi=425, bbox_inches='tight')
    plt.savefig(
        where_to_save_pdfs+\
        'CE_pa_alpha_labs.pdf',
        dpi=425, bbox_inches='tight')

plt.show()_____no_output_____mult=0.95

fig,ax=plt.subplots(1,1,figsize=(5.0*mult,4.5*mult))

plt.subplots_adjust(wspace=0.24, hspace=0.11)
ymax_so_far = 0
xmin_so_far = 0
xmax_so_far = 0

for i,Nn in enumerate(Nvals):
    col = colorz[i]
    means = [np.mean(NCE[Nn][i]['N_frac']) for i in NCE[Nn].keys()]
    stdvs = [np.std(NCE[Nn][i]['N_frac'])  for i in NCE[Nn].keys()]
    alphs = list(NCE[Nn].keys())

    alphs = np.array([(alphs[i]+alphs[i+1])/2
                      for i in range(0,len(alphs)-1,2)])
    means = np.array([(means[i]+means[i+1])/2
                      for i in range(0,len(means)-1,2)])
    stdvs = np.array([(stdvs[i]+stdvs[i+1])/2
                      for i in range(0,len(stdvs)-1,2)])

    xmin_so_far = min([xmin_so_far, min(alphs)])
    xmax_so_far = max([xmax_so_far, max(alphs)])
    ymax_so_far = max([ymax_so_far, max(means+stdvs)])

    ax.semilogy(alphs, means, markeredgecolor=col,
                color=col,markerfacecolor='w',
                markeredgewidth=1.5, markersize=5.0,
                linestyle='-',marker='o',linewidth=2.0,
                alpha=0.99,label='N = %i'%Nn)
    ax.fill_between(alphs, means-stdvs, means+stdvs,
                    facecolor=col,alpha=0.2,
                    edgecolors='w',linewidth=1)

cols = ["#a7d6ca","#dbb9d1","#d6cdae","#a5c9e3"]
ax.fill_between([-2,0.9],[-1,-1],[3,3],
                facecolor=cols[0],alpha=0.3,edgecolors='w',linewidth=0)
ax.fill_between([0.9,1.1],[-1,-1],[3,3],
                facecolor=cols[1],alpha=0.7,edgecolors='w',linewidth=0)
ax.fill_between([1.1,3.0],[-1,-1],[3,3],
                facecolor=cols[2],alpha=0.3,edgecolors='w',linewidth=0)
ax.fill_between([3.0,6],[-1,-1],[3,3],
                facecolor=cols[3],alpha=0.3,edgecolors='w',linewidth=0)

ax.text(-0.50, 1.036,'|', fontsize=14)
ax.text(0.935, 1.036,'|', fontsize=14)
ax.text(0.935, 1.170,'|', fontsize=14)
ax.text(0.935, 1.320,'|', fontsize=14)
ax.text(2.400, 1.036,'|', fontsize=14)
ax.text(4.250, 1.036,'|', fontsize=14)

ax.text(-1.1, 1.368, 'microscale', fontsize=12)
ax.text(0.35, 1.750, 'scale-free', fontsize=12)
ax.text(1.70, 1.368, 'mesoscale',  fontsize=12)
ax.text(3.45, 1.368, 'macroscale', fontsize=12)

ax.set_ylim(0.009*ymax_so_far,ymax_so_far*1.075)
ax.set_xlim(-1.075,5*1.01)
ax.set_xlabel(r'$\alpha$',fontsize=14)
ax.set_ylabel('Size ratio: macro to micro',
              fontsize=14, labelpad=2)
ax.legend(loc=6,framealpha=0.99)
ax.set_xticks(np.linspace(-1,5,7))
ax.set_xticklabels(["%i"%i for i in np.linspace(-1,5,7)])
ax.grid(linestyle='-', linewidth=2.0, color='#999999', alpha=0.3)

if save:
    plt.savefig(
        where_to_save_pngs+\
        'Nfrac_pa_alpha_labs.png',
        dpi=425, bbox_inches='tight')
    plt.savefig(
        where_to_save_pdfs+\
        'Nfrac_pa_alpha_labs.pdf',
        dpi=425, bbox_inches='tight')

plt.show()_____no_output_____

____________________no_output_____## 6.2 Causal Emergence of Random Networks_____no_output_____

Ns = [20,30,40,50]
ps = np.round(np.logspace(-3.25,-0.4,31),5)
Niter = 40

er_ce = {'p'  :[],
         'N_micro':[],
         'N_macro':[],
         'EI_micro':[],
         'EI_macro':[],
         'CE_mean' :[],
         'CE_stdv' :[],
         'N_frac'  :[],
         'runtime' :[]}
ER_CE = {N:er_ce for N in Ns}_____no_output_____

### Note: the following cell was run on a super-computing cluster. It is included as an example computation._____no_output_____

for N in Ns:
    print(N, dt.datetime.now())
    er_ce = {'p'  :[],
             'N_micro':[],
             'N_macro':[],
             'EI_micro':[],
             'EI_macro':[],
             'CE_mean' :[],
             'CE_stdv' :[],
             'N_frac'  :[],
             'runtime' :[]}

    for p in ps:
        print('\t',p)
        cee = []
        for rr in range(Niter):
            G      = nx.erdos_renyi_graph(N,p)
            startT = dt.datetime.now()
            CE     = causal_emergence(G,printt=False)
            finisH = dt.datetime.now()

            diff   = finisH-startT
            diff   = diff.total_seconds()
            ce = CE['EI_macro']-CE['EI_micro']
            cee.append(ce)

        er_ce['p'].append(p)
        er_ce['N_micro'].append(N)
        er_ce['N_macro'].append(CE['G_macro'].number_of_nodes())
        er_ce['EI_micro'].append(CE['EI_micro'])
        er_ce['EI_macro'].append(CE['EI_macro'])
        er_ce['CE_mean'].append(np.mean(cee))
        er_ce['CE_stdv'].append(np.std( cee))
        er_ce['runtime'].append(diff)

    ER_CE[N] = er_ce.copy()_____no_output_____# import cmocean as cmo
# colors = cmo.cm.thermal(np.linspace(0.1,0.95,len(Ns)))
colors = plt.cm.viridis(np.linspace(0.0,1,len(Ns)))

i = 0
ymax = 0
plt.vlines(100, -1, 1,
           label=r'$\langle k \rangle=1$', linestyle='--',
           color="#333333", linewidth=3.5, alpha=0.99)
for N in Ns:
    CE1 = np.array(ER_CE1[N]['CE_mean'].copy())
    CE2 = np.array(ER_CE2[N]['CE_mean'].copy())
    CE3 = np.array(ER_CE3[N]['CE_mean'].copy())
    CE4 = np.array(ER_CE4[N]['CE_mean'].copy())
    CE5 = np.array(ER_CE5[N]['CE_mean'].copy())
    CE6 = np.array(ER_CE6[N]['CE_mean'].copy())
    CEs = (CE1 + CE2 + CE3 + CE4 + CE5 + CE6)/6
    CEs = list(CEs)
    CEs = [(CEs[i] + CEs[i+1])/2 for i in range(0,len(CEs)-1)]
    CEs = [0] + CEs
    CEs.append(0)

    x1 = np.array(ER_CE1[N]['p'].copy())
    x2 = np.array(ER_CE2[N]['p'].copy())
    x3 = np.array(ER_CE3[N]['p'].copy())
    x4 = np.array(ER_CE4[N]['p'].copy())
    x5 = np.array(ER_CE5[N]['p'].copy())
    x6 = np.array(ER_CE6[N]['p'].copy())
    xx = (x1 + x2 + x3 + x4 + x5 + x6)/6
    xx = list(xx)
    xx = [(xx[i] + xx[i+1])/2 for i in range(0,len(xx)-1)]
    xx = [1e-4] + xx
    xx.append(1)

    std1 = np.array(ER_CE1[N]['CE_stdv'].copy())
    std2 = np.array(ER_CE2[N]['CE_stdv'].copy())
    std3 = np.array(ER_CE3[N]['CE_stdv'].copy())
    std4 = np.array(ER_CE4[N]['CE_stdv'].copy())
    std5 = np.array(ER_CE5[N]['CE_stdv'].copy())
    std6 = np.array(ER_CE6[N]['CE_stdv'].copy())
    stds = (std1 + std2 + std3 + std4 + std5 + std6)/6
    stds = list(stds)
    stds = [(stds[i] + stds[i+1])/2 for i in range(0,len(stds)-1)]
    stds = [0] + stds
    stds.append(0)

    ytop = np.array(CEs) + np.array(stds)
    ybot = np.array(CEs) - np.array(stds)
    ybot[ybot<0] = 0

    ymax = max([ymax, max(ytop)])

    plt.semilogx(xx, CEs, label='N=%i'%N,
                 color=colors[i], linewidth=4.0, alpha=0.95)
    plt.vlines(1/(N-1), -1, 1, linestyle='--',
               color=colors[i], linewidth=3.5, alpha=0.95)
    i += 1

plt.xlim(2.5e-4,max(xx))
plt.ylim(-0.0015, ymax*0.6)
plt.grid(linestyle='-', linewidth=2.5, alpha=0.3, color='#999999')
plt.ylabel('Causal emergence', fontsize=14)
plt.xlabel(r'$p$', fontsize=14)
plt.legend(fontsize=12)

if save:
    plt.savefig(
        where_to_save_pngs+\
        'CE_ER_p_N.png', dpi=425, bbox_inches='tight')
    plt.savefig(
        where_to_save_pdfs+\
        'CE_ER_p_N.pdf', dpi=425, bbox_inches='tight')
plt.show()/Users/brennan/anaconda3/lib/python3.7/site-packages/matplotlib/axes/_base.py:3099: UserWarning: Attempting to set identical left==right results
in singular transformations; automatically expanding.
left=100.0, right=100.0
  self.set_xlim(upper, lower, auto=None)
# import cmocean as cmo
# colors = cmo.cm.thermal(np.linspace(0.1,0.95,len(Ns)))
colors = plt.cm.viridis(np.linspace(0.0,1,len(Ns)))

i = 0
ymax = 0
plt.vlines(100, -1, 1, label=r'$\langle k \rangle=1$', linestyle='--',
           color="#333333", linewidth=3.5, alpha=0.99)
for N in Ns:
    CE1 = np.array(ER_CE1[N]['CE_mean'].copy())
    CE2 = np.array(ER_CE2[N]['CE_mean'].copy())
    CE3 = np.array(ER_CE3[N]['CE_mean'].copy())
    CE4 = np.array(ER_CE4[N]['CE_mean'].copy())
    CE5 = np.array(ER_CE5[N]['CE_mean'].copy())
    CE6 = np.array(ER_CE6[N]['CE_mean'].copy())
    CEs = (CE1 + CE2 + CE3 + CE4 + CE5 + CE6)/6
    CEs = list(CEs)
    CEs = [(CEs[i] + CEs[i+1])/2 for i in range(0,len(CEs)-1)]
    CEs = [0] + CEs
    CEs.append(0)

    x1 = np.array(ER_CE1[N]['p'].copy())
    x2 = np.array(ER_CE2[N]['p'].copy())
    x3 = np.array(ER_CE3[N]['p'].copy())
    x4 = np.array(ER_CE4[N]['p'].copy())
    x5 = np.array(ER_CE5[N]['p'].copy())
    x6 = np.array(ER_CE6[N]['p'].copy())
    xx = (x1 + x2 + x3 + x4 + x5 + x6)/6
    xx = list(xx)
    xx = [(xx[i] + xx[i+1])/2 for i in range(0,len(xx)-1)]
    xx = [1e-4] + xx
    xx.append(1)

    std1 = np.array(ER_CE1[N]['CE_stdv'].copy())
    std2 = np.array(ER_CE2[N]['CE_stdv'].copy())
    std3 = np.array(ER_CE3[N]['CE_stdv'].copy())
    std4 = np.array(ER_CE4[N]['CE_stdv'].copy())
    std5 = np.array(ER_CE5[N]['CE_stdv'].copy())
    std6 = np.array(ER_CE6[N]['CE_stdv'].copy())
    stds = (std1 + std2 + std3 + std4 + std5 + std6)/6
    stds = list(stds)
    stds = [(stds[i] + stds[i+1])/2 for i in range(0,len(stds)-1)]
    stds = [0] + stds
    stds.append(0)

    ytop = np.array(CEs) + np.array(stds)
    ybot = np.array(CEs) - np.array(stds)
    ybot[ybot<0] = 0

    ymax = max([ymax, max(ytop)])

    plt.semilogx(xx, CEs, label='N=%i'%N, color=colors[i],
                 linewidth=4.0, alpha=0.95)
    plt.fill_between(xx, ytop, ybot, facecolor=colors[i],
                     linewidth=2.0, alpha=0.35, edgecolor='w')
    plt.vlines(1/(N-1), -1, 1, linestyle='--',
               color=colors[i], linewidth=3.5, alpha=0.95)
    i += 1

plt.xlim(2.5e-4,max(xx))
plt.ylim(-0.0015, ymax)
plt.grid(linestyle='-', linewidth=2.5,
         alpha=0.3, color='#999999')
plt.ylabel('Causal emergence', fontsize=14)
plt.xlabel(r'$p$', fontsize=14)
plt.legend(fontsize=12)

if save:
    plt.savefig(
        where_to_save_pngs+\
        'CE_ER_p_N0.png', dpi=425, bbox_inches='tight')
    plt.savefig(
        where_to_save_pdfs+\
        'CE_ER_p_N0.pdf', dpi=425, bbox_inches='tight')
plt.show()/Users/brennan/anaconda3/lib/python3.7/site-packages/matplotlib/axes/_base.py:3099: UserWarning: Attempting to set identical left==right results
in singular transformations; automatically expanding.
left=100.0, right=100.0
  self.set_xlim(upper, lower, auto=None)
for n in ER_CE1.keys():
    ER_CE1[n]['k'] = np.array(ER_CE1[n]['p'])*n

for n in ER_CE2.keys():
    ER_CE2[n]['k'] = np.array(ER_CE2[n]['p'])*n

for n in ER_CE3.keys():
    ER_CE3[n]['k'] = np.array(ER_CE3[n]['p'])*n

for n in ER_CE4.keys():
    ER_CE4[n]['k'] = np.array(ER_CE4[n]['p'])*n

for n in ER_CE5.keys():
    ER_CE5[n]['k'] = np.array(ER_CE5[n]['p'])*n

for n in ER_CE6.keys():
    ER_CE6[n]['k'] = np.array(ER_CE6[n]['p'])*n    _____no_output_____# import cmocean as cmo
# colors = cmo.cm.thermal(np.linspace(0.1,0.95,len(Ns)))
colors = plt.cm.viridis(np.linspace(0.0,1,len(Ns)))

i = 0
ymax = 0

for N in Ns:
    CE1 = np.array(ER_CE1[N]['CE_mean'].copy())
    CE2 = np.array(ER_CE2[N]['CE_mean'].copy())
    CE3 = np.array(ER_CE3[N]['CE_mean'].copy())
    CE4 = np.array(ER_CE4[N]['CE_mean'].copy())
    CE5 = np.array(ER_CE5[N]['CE_mean'].copy())
    CE6 = np.array(ER_CE6[N]['CE_mean'].copy())
    CEs = (CE1 + CE2 + CE3 + CE4 + CE5 + CE6)/6
    CEs = list(CEs)
    CEs = [(CEs[i] + CEs[i+1])/2 for i in range(0,len(CEs)-1)]
    CEs = [0] + CEs

    x1 = np.array(ER_CE1[N]['k'].copy())
    x2 = np.array(ER_CE2[N]['k'].copy())
    x3 = np.array(ER_CE3[N]['k'].copy())
    x4 = np.array(ER_CE4[N]['k'].copy())
    x5 = np.array(ER_CE5[N]['k'].copy())
    x6 = np.array(ER_CE6[N]['k'].copy())
    xx = (x1 + x2 + x3 + x4 + x5 + x6)/6
    xx = list(xx)
    xx = [(xx[i] + xx[i+1])/2 for i in range(0,len(xx)-1)]
    xx = [1e-4] + xx

    std1 = np.array(ER_CE1[N]['CE_stdv'].copy())
    std2 = np.array(ER_CE2[N]['CE_stdv'].copy())
    std3 = np.array(ER_CE3[N]['CE_stdv'].copy())
    std4 = np.array(ER_CE4[N]['CE_stdv'].copy())
    std5 = np.array(ER_CE5[N]['CE_stdv'].copy())
    std6 = np.array(ER_CE6[N]['CE_stdv'].copy())
    stds = (std1 + std2 + std3 + std4 + std5 + std6)/6
    stds = list(stds)
    stds = [(stds[i] + stds[i+1])/2 for i in range(0,len(stds)-1)]
    stds = [0] + stds

    ytop = np.array(CEs) + np.array(stds)
    ybot = np.array(CEs) - np.array(stds)
    ybot[ybot<0] = 0

    ymax = max([ymax, max(ytop)])

    plt.semilogx(xx, CEs, label='N=%i'%N,
                 color=colors[i], linewidth=4.0, alpha=0.95)
    plt.fill_between(xx, ytop, ybot,
                     facecolor=colors[i],
                     linewidth=2.0, alpha=0.3, edgecolor='w')
    i += 1

plt.vlines(1, -1, 1, linestyle='--',label=r'$\langle k \rangle=1$',
           color='k', linewidth=3.0, alpha=0.95)

plt.xlim(1.0e-2,max(xx))
plt.ylim(-0.0015, ymax*1.01)
plt.grid(linestyle='-', linewidth=2.5, alpha=0.3, color='#999999')
plt.ylabel('Causal emergence', fontsize=14)
plt.xlabel(r'$\langle k \rangle$', fontsize=14)
plt.legend(fontsize=12)

if save:
    plt.savefig(
        where_to_save_pngs+\
        'CE_ER_k_N0.png', dpi=425, bbox_inches='tight')
    plt.savefig(
        where_to_save_pdfs+\
        'CE_ER_k_N0.pdf', dpi=425, bbox_inches='tight')
plt.show()_____no_output_____# import cmocean as cmo
# colors = cmo.cm.thermal(np.linspace(0.1,0.95,len(Ns)))
colors = plt.cm.viridis(np.linspace(0.0,1,len(Ns)))

i = 0
ymax = 0

for N in Ns:
    CE1 = np.array(ER_CE1[N]['CE_mean'].copy())
    CE2 = np.array(ER_CE2[N]['CE_mean'].copy())
    CE3 = np.array(ER_CE3[N]['CE_mean'].copy())
    CE4 = np.array(ER_CE4[N]['CE_mean'].copy())
    CE5 = np.array(ER_CE5[N]['CE_mean'].copy())
    CE6 = np.array(ER_CE6[N]['CE_mean'].copy())
    CEs = (CE1 + CE2 + CE3 + CE4 + CE5 + CE6)/6
    CEs = list(CEs)
    CEs = [(CEs[i] + CEs[i+1])/2 for i in range(0,len(CEs)-1)]
    CEs = [0] + CEs

    x1 = np.array(ER_CE1[N]['k'].copy())
    x2 = np.array(ER_CE2[N]['k'].copy())
    x3 = np.array(ER_CE3[N]['k'].copy())
    x4 = np.array(ER_CE4[N]['k'].copy())
    x5 = np.array(ER_CE5[N]['k'].copy())
    x6 = np.array(ER_CE6[N]['k'].copy())
    xx = (x1 + x2 + x3 + x4 + x5 + x6)/6
    xx = list(xx)
    xx = [(xx[i] + xx[i+1])/2 for i in range(0,len(xx)-1)]
    xx = [1e-4] + xx

    std1 = np.array(ER_CE1[N]['CE_stdv'].copy())
    std2 = np.array(ER_CE2[N]['CE_stdv'].copy())
    std3 = np.array(ER_CE3[N]['CE_stdv'].copy())
    std4 = np.array(ER_CE4[N]['CE_stdv'].copy())
    std5 = np.array(ER_CE5[N]['CE_stdv'].copy())
    std6 = np.array(ER_CE6[N]['CE_stdv'].copy())
    stds = (std1 + std2 + std3 + std4 + std5 + std6)/6
    stds = list(stds)
    stds = [(stds[i] + stds[i+1])/2 for i in range(0,len(stds)-1)]
    stds = [0] + stds

    ytop = np.array(CEs) + np.array(stds)
    ybot = np.array(CEs) - np.array(stds)
    ybot[ybot<0] = 0

    ymax = max([ymax, max(ytop)])

    plt.semilogx(xx, CEs, label='N=%i'%N,
                 color=colors[i],
                 linewidth=4.0, alpha=0.95)
    i += 1

plt.vlines(1, -1, 1, linestyle='--',
           label=r'$\langle k \rangle=1$',
           color='k', linewidth=3.0, alpha=0.95)

plt.xlim(1.0e-2,max(xx))
plt.ylim(-0.0015, ymax*0.6)
plt.grid(linestyle='-', linewidth=2.5,
         alpha=0.3, color='#999999')
plt.ylabel('Causal emergence', fontsize=14)
plt.xlabel(r'$\langle k \rangle$', fontsize=14)
plt.legend(fontsize=12)

if save:
    plt.savefig(
        where_to_save_pngs+'CE_ER_k.png',
        dpi=425, bbox_inches='tight')
    plt.savefig(
        where_to_save_pdfs+'CE_ER_k.pdf',
        dpi=425, bbox_inches='tight')
plt.show()_____no_output_____

## End of Chapter 06. In [Chapter 07](https://nbviewer.jupyter.org/github/jkbren/einet/blob/master/code/Chapter%2007%20-%20Estimating%20Causal%20Emergence%20in%20Real%20Networks.ipynb) we'll estimate causal emergence in real networks.

____________________no_output_____


# Notebook from whole-tale/tale_serialization_formats
Path: metadata_formats/ro/remote_data/data/workspace/LOSC_Event_tutorial.ipynb

# BINARY BLACK HOLE SIGNALS IN LIGO OPEN DATA

Version 1.63, 2017 Sept 11

Welcome! This IPython notebook (or associated python script LOSC_Event_tutorial.py ) will go through some typical signal processing tasks on strain time-series data associated with the LIGO Event data releases from the LIGO Open Science Center (LOSC):

* View the tutorial as a web page, for GW150914.
* After setting the desired "eventname" below, you can just run the full notebook.

Questions, comments, suggestions, corrections, etc: email losc@ligo.caltech.edu_____no_output_____## This tutorial is intended for educational purposes.  The code shown here is not used to produce results papers published by the LIGO Scientific Collaboration, which instead rely on special purpose analysis software packages.
### For publicly available, gravitational-wave software analysis packages that are used to produce LSC and Virgo Collaboration results papers, see https://losc.ligo.org/software/.
### For technical notes on this tutorial, see https://losc.ligo.org/bbh_tutorial_notes/._____no_output_____## Table of Contents
* Intro to signal processing
* Download the data
* Set the event name to choose event and the plot type
* Read in the data
* Plot the ASD
* Binary Neutron Star detection range
* Whitening
* Spectrograms
* Waveform Template
* Matched filtering to find the signal
* Make sound Files
* Data segments_____no_output_____## Intro to signal processing

This tutorial assumes that you are comfortable with python.

This tutorial also assumes that you know a bit about signal processing of digital time series data (or want to learn!). This includes power spectral densities, spectrograms, digital filtering, whitening, audio manipulation. This is a vast and complex set of topics, but we will cover many of the basics in this tutorial.

If you are a beginner, here are some resources from the web:
* http://101science.com/dsp.htm
* https://www.coursera.org/course/dsp
* https://georgemdallas.wordpress.com/2014/05/14/wavelets-4-dummies-signal-processing-fourier-transforms-and-heisenberg/
* https://en.wikipedia.org/wiki/Signal_processing
* https://en.wikipedia.org/wiki/Spectral_density
* https://en.wikipedia.org/wiki/Spectrogram
* http://greenteapress.com/thinkdsp/
* https://en.wikipedia.org/wiki/Digital_filter

And, well, lots more on the web!_____no_output_____## Set the event name to choose event and the plot type_____no_output_____

#-- SET ME   Tutorial should work with most binary black hole events
#-- Default is no event selection; you MUST select one to proceed.
eventname = ''
eventname = 'GW150914'
#eventname = 'GW151226'
#eventname = 'LVT151012'
#eventname = 'GW170104'

# want plots?
make_plots = 1
plottype = "png"
#plottype = "pdf"_____no_output_____# Standard python numerical analysis imports:
import numpy as np
from scipy import signal
from scipy.interpolate import interp1d
from scipy.signal import butter, filtfilt, iirdesign, zpk2tf, freqz
import h5py
import json

# the IPython magic below must be commented out in the .py file, since it doesn't work there.
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
import matplotlib.pyplot as plt
import matplotlib.mlab as mlab

# LIGO-specific readligo.py
import readligo as rl

# you might get a matplotlib warning here; you can ignore it._____no_output_____

### Read the event properties from a local json file (download in advance):_____no_output_____

# Read the event properties from a local json file
fnjson = "BBH_events_v3.json"
try:
    events = json.load(open(fnjson,"r"))
except IOError:
    print("Cannot find resource file "+fnjson)
    print("You can download it from https://losc.ligo.org/s/events/"+fnjson)
    print("Quitting.")
    quit()

# did the user select the eventname ?
try:
    events[eventname]
except:
    print('You must select an eventname that is in '+fnjson+'! Quitting.')
    quit()_____no_output_____# Extract the parameters for the desired event:
event = events[eventname]
fn_H1 = event['fn_H1']              # File name for H1 data
fn_L1 = event['fn_L1']              # File name for L1 data
fn_template = event['fn_template']  # File name for template waveform
fs = event['fs']                    # Set sampling rate
tevent = event['tevent']            # Set approximate event GPS time
fband = event['fband']              # frequency band for bandpassing signal
print("Reading in parameters for event " + event["name"])
print(event)_____no_output_____

## Read in the data
We will make use of the data, and waveform template, defined above._____no_output_____

#----------------------------------------------------------------
# Load LIGO data from a single file.
# FIRST, define the filenames fn_H1 and fn_L1, above.
#----------------------------------------------------------------
try:
    # read in data from H1 and L1, if available:
    strain_H1, time_H1, chan_dict_H1 = rl.loaddata(fn_H1, 'H1')
    strain_L1, time_L1, chan_dict_L1 = rl.loaddata(fn_L1, 'L1')
except:
    print("Cannot find data files!")
    print("You can download them from https://losc.ligo.org/s/events/"+eventname)
    print("Quitting.")
    quit()_____no_output_____

## Data Gaps
**NOTE** that in general, LIGO strain time series data has gaps (filled with NaNs) when the detectors are not taking valid ("science quality") data. Analyzing these data requires the user to
 loop over "segments" of valid data stretches.

**In this tutorial, for simplicity, we assume there are no data gaps - this will not work for all times!**  See the
notes on segments for details.
_____no_output_____## First look at the data from H1 and L1_____no_output_____

# both H1 and L1 will have the same time vector, so:
time = time_H1
# the time sample interval (uniformly sampled!)
dt = time[1] - time[0]

# Let's look at the data and print out some stuff:

print('time_H1: len, min, mean, max = ', \
    len(time_H1), time_H1.min(), time_H1.mean(), time_H1.max() )
print('strain_H1: len, min, mean, max = ', \
    len(strain_H1), strain_H1.min(),strain_H1.mean(),strain_H1.max())
print( 'strain_L1: len, min, mean, max = ', \
    len(strain_L1), strain_L1.min(),strain_L1.mean(),strain_L1.max())

#What's in chan_dict?  (See also https://losc.ligo.org/tutorials/)
bits = chan_dict_H1['DATA']
print("For H1, {0} out of {1} seconds contain usable DATA".format(bits.sum(), len(bits)))
bits = chan_dict_L1['DATA']
print("For L1, {0} out of {1} seconds contain usable DATA".format(bits.sum(), len(bits)))
 _____no_output_____# plot +- deltat seconds around the event:
# index into the strain time series for this time interval:
deltat = 5
indxt = np.where((time >= tevent-deltat) & (time < tevent+deltat))
print(tevent)

if make_plots:
    plt.figure()
    plt.plot(time[indxt]-tevent,strain_H1[indxt],'r',label='H1 strain')
    plt.plot(time[indxt]-tevent,strain_L1[indxt],'g',label='L1 strain')
    plt.xlabel('time (s) since '+str(tevent))
    plt.ylabel('strain')
    plt.legend(loc='lower right')
    plt.title('Advanced LIGO strain data near '+eventname)
_____no_output_____

The data are dominated by **low frequency noise**; there is no way to see a signal here, without some signal processing._____no_output_____## Plot the Amplitude Spectral Density (ASD)
Plotting these data in the Fourier domain gives us an idea of the frequency content of the data. A way to visualize the frequency content of the data is to plot the amplitude spectral density, ASD.

The ASDs are the square root of the power spectral densities (PSDs), which are averages of the square of the fast fourier transforms (FFTs) of the data.

They are an estimate of the "strain-equivalent noise" of the detectors versus frequency,
which limit the ability of the detectors to identify GW signals.

They are in units of strain/rt(Hz).
So, if you want to know the root-mean-square (rms) strain noise in a frequency band,
integrate (sum) the squares of the ASD over that band, then take the square-root.

There's a signal in these data!
For the moment, let's ignore that, and assume it's all noise._____no_output_____

make_psds = 1
if make_psds:
    # number of sample for the fast fourier transform:
    NFFT = 4*fs
    Pxx_H1, freqs = mlab.psd(strain_H1, Fs = fs, NFFT = NFFT)
    Pxx_L1, freqs = mlab.psd(strain_L1, Fs = fs, NFFT = NFFT)

    # We will use interpolations of the ASDs computed above for whitening:
    psd_H1 = interp1d(freqs, Pxx_H1)
    psd_L1 = interp1d(freqs, Pxx_L1)

    # Here is an approximate, smoothed PSD for H1 during O1, with no lines. We'll use it later.
    Pxx = (1.e-22*(18./(0.1+freqs))**2)**2+0.7e-23**2+((freqs/2000.)*4.e-23)**2
    psd_smooth = interp1d(freqs, Pxx)

if make_plots:
    # plot the ASDs, with the template overlaid:
    f_min = 20.
    f_max = 2000.
    plt.figure(figsize=(10,8))
    plt.loglog(freqs, np.sqrt(Pxx_L1),'g',label='L1 strain')
    plt.loglog(freqs, np.sqrt(Pxx_H1),'r',label='H1 strain')
    plt.loglog(freqs, np.sqrt(Pxx),'k',label='H1 strain, O1 smooth model')
    plt.axis([f_min, f_max, 1e-24, 1e-19])
    plt.grid('on')
    plt.ylabel('ASD (strain/rtHz)')
    plt.xlabel('Freq (Hz)')
    plt.legend(loc='upper center')
    plt.title('Advanced LIGO strain data near '+eventname)
_____no_output_____

NOTE that we only plot the data between f_min = 20 Hz and f_max = 2000 Hz.

Below f_min, the data **are not properly calibrated**. That's OK, because the noise is so high below f_min that LIGO cannot sense gravitational wave strain from astrophysical sources in that band.

The sample rate is fs = 4096 Hz (2^12 Hz), so the data cannot capture frequency content above the Nyquist frequency = fs/2 = 2048 Hz. That's OK, because our events only have detectable frequency content in the range given by fband, defined above; the upper end will (almost) always be below the Nyquist frequency. We set f_max = 2000, a bit below Nyquist.

You can see strong spectral lines in the data; they are all of instrumental origin. Some are engineered into the detectors (mirror suspension resonances at ~500 Hz and harmonics, calibration lines, control dither lines, etc) and some (60 Hz and harmonics) are unwanted. We'll return to these, later.

You can't see the signal in this plot, since it is relatively weak and less than a second long, while this plot averages over 32 seconds of data. So this plot is entirely dominated by instrumental noise.

The smooth model is hard-coded and tuned by eye; it won't be right for arbitrary times. We will only use it below for things that don't require much accuracy._____no_output_____## Binary Neutron Star (BNS) detection range

A standard metric that LIGO uses to evaluate the sensitivity of our detectors, based on the detector noise ASD, is the BNS range.

This is defined as the distance to which a LIGO detector can register a BNS signal with a single detector signal-to-noise ratio (SNR) of 8, averaged over source direction and orientation.  Here, SNR 8 is used as a nominal detection threshold, similar to typical CBC detection thresholds of SNR 6-8.

We take each neutron star in the BNS system to have a mass of 1.4 times the mass of the sun, and negligible spin.

GWs from BNS mergers are like "standard sirens"; we know their amplitude at the source from theoretical calculations. The amplitude falls off like 1/r, so their amplitude at the detectors on Earth tells us how far away they are. This is great, because it is hard, in general, to know the distance to astronomical sources.

The amplitude at the source is computed in the post-Newtonian "quadrupole approximation". This is valid for the inspiral phase only, and is approximate at best; there is no simple expression for the post-inspiral (merger and ringdown) phase. So this won't work for high-mass binary black holes like GW150914, which have a lot of signal strength in the post-inspiral phase.

But, in order to use them as standard sirens, we need to know the source direction and orientation relative to the detector and its "quadrupole antenna pattern" response to such signals. It is a standard (if non-trivial) computation to average over all source directions and orientations; the average amplitude is 1./2.2648 times the maximum value.

This calculation is described in Appendix D of:
FINDCHIRP: An algorithm for detection of gravitational waves from inspiraling compact binaries
B. Allen et al., PHYSICAL REVIEW D 85, 122006 (2012) ; http://arxiv.org/abs/gr-qc/0509116
_____no_output_____

BNS_range = 1
if BNS_range:
    #-- compute the binary neutron star (BNS) detectability range

    #-- choose a detector noise power spectrum:
    f = freqs.copy()
    # get frequency step size
    df = f[2]-f[1]

    #-- constants
    # speed of light:
    clight = 2.99792458e8                # m/s
    # Newton's gravitational constant
    G = 6.67259e-11                      # m^3/kg/s^2
    # one parsec, popular unit of astronomical distance (around 3.26 light years)
    parsec = 3.08568025e16               # m
    # solar mass
    MSol = 1.989e30                      # kg
    # solar mass in seconds (isn't relativity fun?):
    tSol = MSol*G/np.power(clight,3)     # s
    # Single-detector SNR for detection above noise background:
    SNRdet = 8.
    # conversion from maximum range (horizon) to average range:
    Favg = 2.2648
    # mass of a typical neutron star, in solar masses:
    mNS = 1.4

    # Masses in solar masses
    m1 = m2 = mNS
    mtot = m1+m2  # the total mass
    eta = (m1*m2)/mtot**2  # the symmetric mass ratio
    mchirp = mtot*eta**(3./5.)  # the chirp mass (FINDCHIRP, following Eqn 3.1b)

    # distance to a fiducial BNS source:
    dist = 1.0                           # in Mpc
    Dist =  dist * 1.0e6 * parsec /clight # from Mpc to seconds

    # We integrate the signal up to the frequency of the "Innermost stable circular orbit (ISCO)"
    R_isco = 6.      # Orbital separation at ISCO, in geometric units. 6M for PN ISCO; 2.8M for EOB
    # frequency at ISCO (end the chirp here; the merger and ringdown follow)
    f_isco = 1./(np.power(R_isco,1.5)*np.pi*tSol*mtot)
    # minimum frequency (below which, detector noise is too high to register any signal):
    f_min = 20. # Hz
    # select the range of frequencies between f_min and fisco
    fr = np.nonzero(np.logical_and(f > f_min , f < f_isco))
    # get the frequency and spectrum in that range:
    ffr = f[fr]

    # In stationary phase approx, this is htilde(f):
    # See FINDCHIRP Eqns 3.4, or 8.4-8.5
    htilde = (2.*tSol/Dist)*np.power(mchirp,5./6.)*np.sqrt(5./96./np.pi)*(np.pi*tSol)
    htilde *= np.power(np.pi*tSol*ffr,-7./6.)
    htilda2 = htilde**2

    # loop over the detectors
    dets = ['H1', 'L1']
    for det in dets:
        if det is 'L1': sspec = Pxx_L1.copy()
        else:           sspec = Pxx_H1.copy()
        sspecfr = sspec[fr]
        # compute "inspiral horizon distance" for optimally oriented binary; FINDCHIRP Eqn D2:
        D_BNS = np.sqrt(4.*np.sum(htilda2/sspecfr)*df)/SNRdet
        # and the "inspiral range", averaged over source direction and orientation:
        R_BNS = D_BNS/Favg
        print(det+' BNS inspiral horizon = {0:.1f} Mpc, BNS inspiral range   = {1:.1f} Mpc'.format(D_BNS,R_BNS))_____no_output_____

## BBH range is >> BNS range!

NOTE that, since mass is the source of gravity and thus also of gravitational waves, systems with higher masses (such as the binary black hole merger GW150914) are much "louder" and can be detected to much higher distances than the BNS range. We'll compute the BBH range, using a template with specific masses, below._____no_output_____## Whitening

From the ASD above, we can see that the data are very strongly "colored" - noise fluctuations are much larger at low and high frequencies and near spectral lines, reaching a roughly flat ("white") minimum in the band around 80 to 300 Hz.

We can "whiten" the data (dividing it by the noise amplitude spectrum, in the fourier domain), suppressing the extra noise at low frequencies and at the spectral lines, to better see the weak signals in the most sensitive band.

Whitening is always one of the first steps in astrophysical data analysis (searches, parameter estimation).
Whitening requires no prior knowledge of spectral lines, etc; only the data are needed.

To get rid of remaining high frequency noise, we will also bandpass the data.

The resulting time series is no longer in units of strain; now in units of "sigmas" away from the mean.

We will plot the whitened strain data, along with the signal template, after the matched filtering section, below._____no_output_____

# function to whiten data
def whiten(strain, interp_psd, dt):
    Nt = len(strain)
    freqs = np.fft.rfftfreq(Nt, dt)
    freqs1 = np.linspace(0,2048.,Nt/2+1)

    # whitening: transform to freq domain, divide by asd, then transform back,
    # taking care to get normalization right.
    hf = np.fft.rfft(strain)
    norm = 1./np.sqrt(1./(dt*2))
    white_hf = hf / np.sqrt(interp_psd(freqs)) * norm
    white_ht = np.fft.irfft(white_hf, n=Nt)
    return white_ht

whiten_data = 1
if whiten_data:
    # now whiten the data from H1 and L1, and the template (use H1 PSD):
    strain_H1_whiten = whiten(strain_H1,psd_H1,dt)
    strain_L1_whiten = whiten(strain_L1,psd_L1,dt)

    # We need to suppress the high frequency noise (no signal!) with some bandpassing:
    bb, ab = butter(4, [fband[0]*2./fs, fband[1]*2./fs], btype='band')
    normalization = np.sqrt((fband[1]-fband[0])/(fs/2))
    strain_H1_whitenbp = filtfilt(bb, ab, strain_H1_whiten) / normalization
    strain_L1_whitenbp = filtfilt(bb, ab, strain_L1_whiten) / normalization_____no_output_____

## Spectrograms

Now let's plot a short time-frequency spectrogram around our event:_____no_output_____

if make_plots:
    # index into the strain time series for this time interval:
    indxt = np.where((time >= tevent-deltat) & (time < tevent+deltat))

    # pick a shorter FTT time interval, like 1/8 of a second:
    NFFT = int(fs/8)
    # and with a lot of overlap, to resolve short-time features:
    NOVL = int(NFFT*15./16)
    # and choose a window that minimizes "spectral leakage"
    # (https://en.wikipedia.org/wiki/Spectral_leakage)
    window = np.blackman(NFFT)

    # the right colormap is all-important! See:
    # http://matplotlib.org/examples/color/colormaps_reference.html
    # viridis seems to be the best for our purposes, but it's new; if you don't have it, you can settle for ocean.
    #spec_cmap='viridis'
    spec_cmap='ocean'

    # Plot the H1 spectrogram:
    plt.figure(figsize=(10,6))
    spec_H1, freqs, bins, im = plt.specgram(strain_H1[indxt], NFFT=NFFT, Fs=fs, window=window,
                                            noverlap=NOVL, cmap=spec_cmap, xextent=[-deltat,deltat])
    plt.xlabel('time (s) since '+str(tevent))
    plt.ylabel('Frequency (Hz)')
    plt.colorbar()
    plt.axis([-deltat, deltat, 0, 2000])
    plt.title('aLIGO H1 strain data near '+eventname)

    # Plot the L1 spectrogram:
    plt.figure(figsize=(10,6))
    spec_H1, freqs, bins, im = plt.specgram(strain_L1[indxt], NFFT=NFFT, Fs=fs, window=window,
                                            noverlap=NOVL, cmap=spec_cmap, xextent=[-deltat,deltat])
    plt.xlabel('time (s) since '+str(tevent))
    plt.ylabel('Frequency (Hz)')
    plt.colorbar()
    plt.axis([-deltat, deltat, 0, 2000])
    plt.title('aLIGO L1 strain data near '+eventname)
_____no_output_____

In the above spectrograms, you may see lots of excess power below ~20 Hz, as well as strong spectral lines at 500, 1000, 1500 Hz (also evident in the ASDs above). The lines at multiples of 500 Hz are the harmonics of the "violin modes" of the fibers holding up the mirrors of the Advanced LIGO interferometers.

Now let's zoom in on where we think the signal is, using the whitened data, in the hope of seeing a chirp:_____no_output_____

if make_plots:
    #  plot the whitened data, zooming in on the signal region:

    # pick a shorter FTT time interval, like 1/16 of a second:
    NFFT = int(fs/16.0)
    # and with a lot of overlap, to resolve short-time features:
    NOVL = int(NFFT*15/16.0)
    # choose a window that minimizes "spectral leakage"
    # (https://en.wikipedia.org/wiki/Spectral_leakage)
    window = np.blackman(NFFT)

    # Plot the H1 whitened spectrogram around the signal
    plt.figure(figsize=(10,6))
    spec_H1, freqs, bins, im = plt.specgram(strain_H1_whiten[indxt], NFFT=NFFT, Fs=fs, window=window,
                                            noverlap=NOVL, cmap=spec_cmap, xextent=[-deltat,deltat])
    plt.xlabel('time (s) since '+str(tevent))
    plt.ylabel('Frequency (Hz)')
    plt.colorbar()
    plt.axis([-0.5, 0.5, 0, 500])
    plt.title('aLIGO H1 strain data near '+eventname)

    # Plot the L1 whitened spectrogram around the signal
    plt.figure(figsize=(10,6))
    spec_H1, freqs, bins, im = plt.specgram(strain_L1_whiten[indxt], NFFT=NFFT, Fs=fs, window=window,
                                            noverlap=NOVL, cmap=spec_cmap, xextent=[-deltat,deltat])
    plt.xlabel('time (s) since '+str(tevent))
    plt.ylabel('Frequency (Hz)')
    plt.colorbar()
    plt.axis([-0.5, 0.5, 0, 500])
    plt.title('aLIGO L1 strain data near '+eventname)
_____no_output_____

Loud (high SNR) signals may be visible in these spectrograms.  Compact object mergers show a characteristic "chirp" as the signal rises in frequency.  If you can't see anything, try
event GW150914, by changing the `eventname` variable in the first cell above._____no_output_____## Waveform Template

The results of a full LIGO-Virgo analysis of this BBH event include a set of parameters that are consistent with a range of parameterized waveform templates. Here we pick one for use in matched filtering.

As noted above, the results won't be identical to what is in the LIGO-Virgo papers, since we're skipping many subtleties, such as combining many consistent templates._____no_output_____

# read in the template (plus and cross) and parameters for the theoretical waveform
try:
    f_template = h5py.File(fn_template, "r")
except:
    print("Cannot find template file!")
    print("You can download it from https://losc.ligo.org/s/events/"+eventname+'/'+fn_template)
    print("Quitting.")
    quit()_____no_output_____# extract metadata from the template file:
template_p, template_c = f_template["template"][...]
t_m1 = f_template["/meta"].attrs['m1']
t_m2 = f_template["/meta"].attrs['m2']
t_a1 = f_template["/meta"].attrs['a1']
t_a2 = f_template["/meta"].attrs['a2']
t_approx = f_template["/meta"].attrs['approx']
f_template.close()
# the template extends to roughly 16s, zero-padded to the 32s data length. The merger will be roughly 16s in.
template_offset = 16.

# whiten the templates:
template_p_whiten = whiten(template_p,psd_H1,dt)
template_c_whiten = whiten(template_c,psd_H1,dt)
template_p_whitenbp = filtfilt(bb, ab, template_p_whiten) / normalization
template_c_whitenbp = filtfilt(bb, ab, template_c_whiten) / normalization

# Compute, print and plot some properties of the template:

# constants:
clight = 2.99792458e8                # m/s
G = 6.67259e-11                      # m^3/kg/s^2
MSol = 1.989e30                      # kg

# template parameters: masses in units of MSol:
t_mtot = t_m1+t_m2
# final BH mass is typically 95% of the total initial mass:
t_mfin = t_mtot*0.95
# Final BH radius, in km:
R_fin = 2*G*t_mfin*MSol/clight**2/1000.

# complex template:
template = (template_p + template_c*1.j)
ttime = time-time[0]-template_offset

# compute the instantaneous frequency of this chirp-like signal:
tphase = np.unwrap(np.angle(template))
fGW = np.gradient(tphase)*fs/(2.*np.pi)
# fix discontinuities at the very end:
# iffix = np.where(np.abs(np.gradient(fGW)) > 100.)[0]
iffix = np.where(np.abs(template) < np.abs(template).max()*0.001)[0]
fGW[iffix] = fGW[iffix[0]-1]
fGW[np.where(fGW < 1.)] = fGW[iffix[0]-1]

# compute v/c:
voverc = (G*t_mtot*MSol*np.pi*fGW/clight**3)**(1./3.)

# index where f_GW is in-band:
f_inband = fband[0]
iband = np.where(fGW > f_inband)[0][0]
# index at the peak of the waveform:
ipeak = np.argmax(np.abs(template))

# number of cycles between inband and peak:
Ncycles = (tphase[ipeak]-tphase[iband])/(2.*np.pi)

print('Properties of waveform template in {0}'.format(fn_template))
print("Waveform family = {0}".format(t_approx))
print("Masses = {0:.2f}, {1:.2f} Msun".format(t_m1,t_m2))
print('Mtot = {0:.2f} Msun, mfinal = {1:.2f} Msun '.format(t_mtot,t_mfin))
print("Spins = {0:.2f}, {1:.2f}".format(t_a1,t_a2))
print('Freq at inband, peak = {0:.2f}, {1:.2f} Hz'.format(fGW[iband],fGW[ipeak]))
print('Time at inband, peak = {0:.2f}, {1:.2f} s'.format(ttime[iband],ttime[ipeak]))
print('Duration (s) inband-peak = {0:.2f} s'.format(ttime[ipeak]-ttime[iband]))
print('N_cycles inband-peak = {0:.0f}'.format(Ncycles))
print('v/c at peak = {0:.2f}'.format(voverc[ipeak]))
print('Radius of final BH = {0:.0f} km'.format(R_fin))

if make_plots:
    plt.figure(figsize=(10,16))
    plt.subplot(4,1,1)
    plt.plot(ttime,template_p)
    plt.xlim([-template_offset,1.])
    plt.grid()
    plt.xlabel('time (s)')
    plt.ylabel('strain')
    plt.title(eventname+' template at D_eff = 1 Mpc')

    plt.subplot(4,1,2)
    plt.plot(ttime,template_p)
    plt.xlim([-1.1,0.1])
    plt.grid()
    plt.xlabel('time (s)')
    plt.ylabel('strain')
    #plt.title(eventname+' template at D_eff = 1 Mpc')

    plt.subplot(4,1,3)
    plt.plot(ttime,fGW)
    plt.xlim([-1.1,0.1])
    plt.grid()
    plt.xlabel('time (s)')
    plt.ylabel('f_GW')
    #plt.title(eventname+' template f_GW')

    plt.subplot(4,1,4)
    plt.plot(ttime,voverc)
    plt.xlim([-1.1,0.1])
    plt.grid()
    plt.xlabel('time (s)')
    plt.ylabel('v/c')
    #plt.title(eventname+' template v/c')
_____no_output_____

## Matched filtering to find the signal

Matched filtering is the optimal way to find a known signal buried in stationary, Gaussian noise. It is the standard technique used by the gravitational wave community to find GW signals from compact binary mergers in noisy detector data.

For some loud signals, it may be possible to see the signal in the whitened data or spectrograms. On the other hand, low signal-to-noise ratio (SNR) signals or signals which are of long duration in time may not be visible, even in the whitened data.  LIGO scientists use matched filtering to find such "hidden" signals. A matched filter works by compressing the entire signal into one time bin (by convention, the "end time" of the waveform).

LIGO uses a rather elaborate software suite to match the data against a family of such signal waveforms ("templates"), to find the best match. This procedure helps to "optimally" separate signals from instrumental noise, and to infer the parameters of the source (masses, spins, sky location, orbit orientation, etc) from the best match templates.

A blind search requires us to search over many compact binary merger templates (eg, 250,000) with different masses and spins, as well as over all times in all detectors, and then requiring triggers coincident in time and template between detectors. It's an extremely complex and computationally-intensive "search pipeline".

Here, we simplify things, using only one template (the one identified in the full search as being a good match to the data).

Assuming that the data around this event is fairly Gaussian and stationary, we'll use this simple method to identify the signal (matching the template) in our 32 second stretch of data. The peak in the SNR vs time is a "single-detector event trigger".

This calculation is described in section IV of:
FINDCHIRP: An algorithm for detection of gravitational waves from inspiraling compact binaries
B. Allen et al., PHYSICAL REVIEW D 85, 122006 (2012) ; http://arxiv.org/abs/gr-qc/0509116

The full search procedure is described in
GW150914: First results from the search for binary black hole coalescence with Advanced LIGO,
The LIGO Scientific Collaboration, the Virgo Collaboration, http://arxiv.org/abs/1602.03839_____no_output_____

# -- To calculate the PSD of the data, choose an overlap and a window (common to all detectors)
#   that minimizes "spectral leakage" https://en.wikipedia.org/wiki/Spectral_leakage
NFFT = 4*fs
psd_window = np.blackman(NFFT)
# and a 50% overlap:
NOVL = NFFT/2

# define the complex template, common to both detectors:
template = (template_p + template_c*1.j)
# We will record the time where the data match the END of the template.
etime = time+template_offset
# the length and sampling rate of the template MUST match that of the data.
datafreq = np.fft.fftfreq(template.size)*fs
df = np.abs(datafreq[1] - datafreq[0])

# to remove effects at the beginning and end of the data stretch, window the data
# https://en.wikipedia.org/wiki/Window_function#Tukey_window
try:   dwindow = signal.tukey(template.size, alpha=1./8)  # Tukey window preferred, but requires recent scipy version
except: dwindow = signal.blackman(template.size)          # Blackman window OK if Tukey is not available

# prepare the template fft.
template_fft = np.fft.fft(template*dwindow) / fs

# loop over the detectors
dets = ['H1', 'L1']
for det in dets:

    if det is 'L1': data = strain_L1.copy()
    else:           data = strain_H1.copy()

    # -- Calculate the PSD of the data.  Also use an overlap, and window:
    data_psd, freqs = mlab.psd(data, Fs = fs, NFFT = NFFT, window=psd_window, noverlap=NOVL)

    # Take the Fourier Transform (FFT) of the data and the template (with dwindow)
    data_fft = np.fft.fft(data*dwindow) / fs

    # -- Interpolate to get the PSD values at the needed frequencies
    power_vec = np.interp(np.abs(datafreq), freqs, data_psd)

    # -- Calculate the matched filter output in the time domain:
    # Multiply the Fourier Space template and data, and divide by the noise power in each frequency bin.
    # Taking the Inverse Fourier Transform (IFFT) of the filter output puts it back in the time domain,
    # so the result will be plotted as a function of time off-set between the template and the data:
    optimal = data_fft * template_fft.conjugate() / power_vec
    optimal_time = 2*np.fft.ifft(optimal)*fs

    # -- Normalize the matched filter output:
    # Normalize the matched filter output so that we expect a value of 1 at times of just noise.
    # Then, the peak of the matched filter output will tell us the signal-to-noise ratio (SNR) of the signal.
    sigmasq = 1*(template_fft * template_fft.conjugate() / power_vec).sum() * df
    sigma = np.sqrt(np.abs(sigmasq))
    SNR_complex = optimal_time/sigma

    # shift the SNR vector by the template length so that the peak is at the END of the template
    peaksample = int(data.size / 2)  # location of peak in the template
    SNR_complex = np.roll(SNR_complex,peaksample)
    SNR = abs(SNR_complex)

    # find the time and SNR value at maximum:
    indmax = np.argmax(SNR)
    timemax = time[indmax]
    SNRmax = SNR[indmax]

    # Calculate the "effective distance" (see FINDCHIRP paper for definition)
    # d_eff = (8. / SNRmax)*D_thresh
    d_eff = sigma / SNRmax
    # -- Calculate optimal horizon distnace
    horizon = sigma/8

    # Extract time offset and phase at peak
    phase = np.angle(SNR_complex[indmax])
    offset = (indmax-peaksample)

    # apply time offset, phase, and d_eff to template
    template_phaseshifted = np.real(template*np.exp(1j*phase))    # phase shift the template
    template_rolled = np.roll(template_phaseshifted,offset) / d_eff  # Apply time offset and scale amplitude

    # Whiten and band-pass the template for plotting
    template_whitened = whiten(template_rolled,interp1d(freqs, data_psd),dt)  # whiten the template
    template_match = filtfilt(bb, ab, template_whitened) / normalization # Band-pass the template

    print('For detector {0}, maximum at {1:.4f} with SNR = {2:.1f}, D_eff = {3:.2f}, horizon = {4:0.1f} Mpc'
          .format(det,timemax,SNRmax,d_eff,horizon))

    if make_plots:

        # plotting changes for the detectors:
        if det is 'L1':
            pcolor='g'
            strain_whitenbp = strain_L1_whitenbp
            template_L1 = template_match.copy()
        else:
            pcolor='r'
            strain_whitenbp = strain_H1_whitenbp
            template_H1 = template_match.copy()

        # -- Plot the result
        plt.figure(figsize=(10,8))
        plt.subplot(2,1,1)
        plt.plot(time-timemax, SNR, pcolor,label=det+' SNR(t)')
        #plt.ylim([0,25.])
        plt.grid('on')
        plt.ylabel('SNR')
        plt.xlabel('Time since {0:.4f}'.format(timemax))
        plt.legend(loc='upper left')
        plt.title(det+' matched filter SNR around event')

        # zoom in
        plt.subplot(2,1,2)
        plt.plot(time-timemax, SNR, pcolor,label=det+' SNR(t)')
        plt.grid('on')
        plt.ylabel('SNR')
        plt.xlim([-0.15,0.05])
        #plt.xlim([-0.3,+0.3])
        plt.grid('on')
        plt.xlabel('Time since {0:.4f}'.format(timemax))
        plt.legend(loc='upper left')

        plt.figure(figsize=(10,8))
        plt.subplot(2,1,1)
        plt.plot(time-tevent,strain_whitenbp,pcolor,label=det+' whitened h(t)')
        plt.plot(time-tevent,template_match,'k',label='Template(t)')
        plt.ylim([-10,10])
        plt.xlim([-0.15,0.05])
        plt.grid('on')
        plt.xlabel('Time since {0:.4f}'.format(timemax))
        plt.ylabel('whitened strain (units of noise stdev)')
        plt.legend(loc='upper left')
        plt.title(det+' whitened data around event')

        plt.subplot(2,1,2)
        plt.plot(time-tevent,strain_whitenbp-template_match,pcolor,label=det+' resid')
        plt.ylim([-10,10])
        plt.xlim([-0.15,0.05])
        plt.grid('on')
        plt.xlabel('Time since {0:.4f}'.format(timemax))
        plt.ylabel('whitened strain (units of noise stdev)')
        plt.legend(loc='upper left')
        plt.title(det+' Residual whitened data after subtracting template around event')

        # -- Display PSD and template
        # must multiply by sqrt(f) to plot template fft on top of ASD:
        plt.figure(figsize=(10,6))
        template_f = np.absolute(template_fft)*np.sqrt(np.abs(datafreq)) / d_eff
        plt.loglog(datafreq, template_f, 'k', label='template(f)*sqrt(f)')
        plt.loglog(freqs, np.sqrt(data_psd),pcolor, label=det+' ASD')
        plt.xlim(20, fs/2)
        plt.ylim(1e-24, 1e-20)
        plt.grid()
        plt.xlabel('frequency (Hz)')
        plt.ylabel('strain noise ASD (strain/rtHz), template h(f)*rt(f)')
        plt.legend(loc='upper left')
        plt.title(det+' ASD and template around event')
_____no_output_____

### Notes on these results

* We make use of only one template, with a simple ASD estimate. The full analysis produces a Bayesian posterior result using many nearby templates. It does a more careful job estimating the ASD, and includes effects of uncertain calibration.
* As a result, our parameters (SNR, masses, spins, D_eff) are somewhat different from what you will see in our papers.
* We compute an "effective distance" D_eff. Is is NOT an estimate of the actual (luminosity) distance, which depends also on the source location and orbit orientation.
* These distances are at non-zero redshift, so cosmological effects must be taken into account (neglected here). Since we estimate the BH masses using the phase evolution of the waveform, which has been redshifted, our masses are themselves "redshifted". The true source masses must be corrected for this effect; they are smaller by a factor (1+z).
_____no_output_____## Make sound files

Make wav (sound) files from the filtered, downsampled data, +-2s around the event._____no_output_____

# make wav (sound) files from the whitened data, +-2s around the event.

from scipy.io import wavfile

# function to keep the data within integer limits, and write to wavfile:
def write_wavfile(filename,fs,data):
    d = np.int16(data/np.max(np.abs(data)) * 32767 * 0.9)
    wavfile.write(filename,int(fs), d)

deltat_sound = 2.                     # seconds around the event

# index into the strain time series for this time interval:
indxd = np.where((time >= tevent-deltat_sound) & (time < tevent+deltat_sound))

# write the files:
write_wavfile("../"+eventname+"_H1_whitenbp.wav",int(fs), strain_H1_whitenbp[indxd])
write_wavfile("../"+eventname+"_L1_whitenbp.wav",int(fs), strain_L1_whitenbp[indxd])

# re-whiten the template using the smoothed PSD; it sounds better!
template_p_smooth = whiten(template_p,psd_smooth,dt)

# and the template, sooming in on [-3,+1] seconds around the merger:
indxt = np.where((time >= (time[0]+template_offset-deltat_sound)) & (time < (time[0]+template_offset+deltat_sound)))
write_wavfile("../"+eventname+"_template_whiten.wav",int(fs), template_p_smooth[indxt])
_____no_output_____

### Listen to the whitened template and data

With good headphones, you may be able to hear a faint thump in the middle; that's our signal!_____no_output_____

from IPython.display import Audio

fna = "../"+eventname+"_template_whiten.wav"
print(fna)
Audio(fna)_____no_output_____fna = "../"+eventname+"_H1_whitenbp.wav"
print(fna)
Audio(fna)_____no_output_____

### Frequency shift the audio files
We can enhance this by increasing the frequency;
this is the "audio" equivalent of the enhanced visuals that NASA employs on telescope images with "false color".

The code below will shift the data up by 400 Hz (by taking an FFT, shifting/rolling the frequency series, then inverse fft-ing). The resulting sound file will be noticibly more high-pitched, and the signal will be easier to hear._____no_output_____

# function that shifts frequency of a band-passed signal
def reqshift(data,fshift=100,sample_rate=4096):
    """Frequency shift the signal by constant
    """
    x = np.fft.rfft(data)
    T = len(data)/float(sample_rate)
    df = 1.0/T
    nbins = int(fshift/df)
    # print T,df,nbins,x.real.shape
    y = np.roll(x.real,nbins) + 1j*np.roll(x.imag,nbins)
    y[0:nbins]=0.
    z = np.fft.irfft(y)
    return z

# parameters for frequency shift
fs = 4096
fshift = 400.
speedup = 1.
fss = int(float(fs)*float(speedup))

# shift frequency of the data
strain_H1_shifted = reqshift(strain_H1_whitenbp,fshift=fshift,sample_rate=fs)
strain_L1_shifted = reqshift(strain_L1_whitenbp,fshift=fshift,sample_rate=fs)

# write the files:
write_wavfile("../"+eventname+"_H1_shifted.wav",int(fs), strain_H1_shifted[indxd])
write_wavfile("../"+eventname+"_L1_shifted.wav",int(fs), strain_L1_shifted[indxd])

# and the template:
template_p_shifted = reqshift(template_p_smooth,fshift=fshift,sample_rate=fs)
write_wavfile("../"+eventname+"_template_shifted.wav",int(fs), template_p_shifted[indxt])_____no_output_____

### Listen to the frequency-shifted template and data_____no_output_____

fna = "../"+eventname+"_template_shifted.wav"
print(fna)
Audio(fna)_____no_output_____fna = "../"+eventname+"_H1_shifted.wav"
print(fna)
Audio(fna)_____no_output_____

## Data segments

As mentioned above, LIGO strain time series data has gaps (filled with NaNs) when the detectors are not taking valid ("science quality") data. Analyzing these data requires the user to loop over "segments" of valid data stretches.  For details, see the notes on segments or introduction to LIGO data files.

In the code below, we can check times around this event for gaps in the L1 data.  You are welcome to repeat this with H1 data, with files containing 4096 seconds of data, and with data sampled at 16384 Hz. All of the relevant files can be downloaded from LOSC event pages.

We also unpack the DQ and HW injection bits to check what their values are._____no_output_____

data_segments = 1
if data_segments:
    # read in the data at 4096 Hz:
    # fn = 'L-L1_LOSC_4_V1-1126259446-32.hdf5'
    strain, time, chan_dict = rl.loaddata(fn_L1, 'H1')

    print("Contents of all the key, value pairs in chan_dict")
    for keys,values in chan_dict.items():
        print(keys)
        print(values)

    print('Total number of non-NaNs in these data = ',np.sum(~np.isnan(strain)))
    print('GPS start, GPS stop and length of  all data in this file = ',time[0], time[-1],len(strain))

    # select the level of data quality; default is "DATA" but "CBC_CAT3" is a conservative choice:
    DQflag = 'CBC_CAT3'
    # readligo.py method for computing segments (start and stop times with continuous valid data):
    segment_list = rl.dq_channel_to_seglist(chan_dict[DQflag])
    print('Number of segments with DQflag',DQflag,' = ',len(segment_list))

    # loop over seconds and print out start, stop and length:
    iseg = 0
    for segment in segment_list:
        time_seg = time[segment]
        seg_strain = strain[segment]
        print('GPS start, GPS stop and length of segment',iseg, \
            'in this file = ',time_seg[0], time_seg[-1], len(seg_strain))
        iseg = iseg+1
        # here is where you would insert code to analyze the data in this segment.

    # now look at segments with no CBC hardware injections:
    DQflag = 'NO_CBC_HW_INJ'
    segment_list = rl.dq_channel_to_seglist(chan_dict['NO_CBC_HW_INJ'])
    print('Number of segments with DQflag',DQflag,' = ',len(segment_list))
    iseg = 0
    for segment in segment_list:
        time_seg = time[segment]
        seg_strain = strain[segment]
        print('GPS start, GPS stop and length of segment',iseg, \
            'in this file = ',time_seg[0], time_seg[-1], len(seg_strain))
        iseg = iseg+1
_____no_output_____

## Comments on sampling rate

LIGO data are acquired at 16384 Hz (2^14 Hz). Here, we have been working with data downsampled to 4096 Hz, to save on download time, disk space, and memory requirements.

This is entirely sufficient for signals with no frequency content above f_Nyquist = fs/2 = 2048 Hz, such as signals from higher-mass binary black hole systems; the frequency at which the merger begins (at the innermost stable circular orbit) for equal-mass, spinless black holes is roughly 1557 Hz * (2.8/M_tot), where 2.8 solar masses is the total mass of a canonical binary neutron star system.

If, however, you are interested in signals with frequency content above 2048 Hz, you need the data sampled at the full rate of 16384 Hz. _____no_output_____## Construct a csv file containing the whitened data and template_____no_output_____

# time vector around event
times = time-tevent
# zoom in on [-0.2,0.05] seconds around event
irange = np.nonzero((times >= -0.2) & (times < 0.05))
# construct a data structure for a csv file:
dat = [times[irange], strain_H1_whitenbp[irange],strain_L1_whitenbp[irange],
      template_H1[irange],template_L1[irange] ]
datcsv = np.array(dat).transpose()
# make a csv filename, header, and format
fncsv = "../"+eventname+'_data.csv'
headcsv = eventname+' time-'+str(tevent)+ \
    ' (s),H1_data_whitened,L1_data_whitened,H1_template_whitened,L1_template_whitened'
fmtcsv = ",".join(["%10.6f"] * 5)
np.savetxt(fncsv, datcsv, fmt=fmtcsv, header=headcsv)

print("Wrote whitened data to file {0}".format(fncsv))_____no_output_____


# Notebook from nama-aman/qiskit-textbook
Path: content/ch-gates/proving-universality.ipynb

# Proving Universality_____no_output_____## Contents

1. [Introduction](#intro)
2. [Fun With Matrices](#fun)
    2.1 [Matrices as outer products](#outer)
    2.2 [Unitary and Hermitian matrices](#u-and-h)
    2.3 [Pauli Decomposition](#pauli)
3. [Defining Universality](#defining)
4. [Basic Gate Sets](#basic)
    4.1 [Clifford Gates](#big-red)
    4.2 [Non-Clifford Gates](#non-clifford)
    4.3 [Expanding the Gate Set](#expanding)
5. [Proving Universality](#proving)
6. [Universal Sets of Quantum Gates](#gate-sets)_____no_output_____## 1. Introduction

What does it mean for a computer to be able to do everything that it could possibly do? This was a question tackled by Alan Turing before we even had a good idea of what a computer was, or how to build one.

To ask this question for our classical computers, and specifically for our standard digital computers, we need to strip away all the screens, speakers and fancy input devices. What we are left with is simply a machine that converts input bit strings into output bit strings. If a device can perform any such conversion, taking any arbitrary set of inputs and converting them to an arbitrarily chosen set of corresponding outputs, we call it *universal*.

Quantum computers similarly take input states and convert them into output states. We will therefore be able to define universality in a similar way. To be more precise, and to be able to prove when universality can and cannot be achieved, it is useful to use the matrix representation of our quantum gates. But first we'll need to brush up on a few techniques._____no_output_____## 2. Fun With Matrices

### 2.1 Matrices as outer products

In previous sections we have calculated many inner products, such as $\langle0|0\rangle =1$. These combine a bra and a ket to give us a single number. We can also combine them in a way that gives us a matrix, simply by putting them in the opposite order. This is called an outer product, and works by standard matrix multiplication. For example

$$
|0\rangle\langle0|= \begin{pmatrix} 1 \\\\ 0 \end{pmatrix} \begin{pmatrix} 1 & 0 \end{pmatrix} = \begin{pmatrix} 1&0 \\\\ 0&0 \end{pmatrix},\\\\
|0\rangle\langle1| = \begin{pmatrix} 1 \\\\ 0 \end{pmatrix} \begin{pmatrix} 0 & 1 \end{pmatrix} = \begin{pmatrix} 0&1 \\\\ 0&0 \end{pmatrix},\\\\
|1\rangle\langle0| = \begin{pmatrix} 0 \\\\ 1 \end{pmatrix} \begin{pmatrix} 1 & 0 \end{pmatrix} = \begin{pmatrix} 0&0 \\\\ 1&0 \end{pmatrix},\\\\
|1\rangle\langle1| = \begin{pmatrix} 0 \\\\ 1 \end{pmatrix} \begin{pmatrix} 0 & 1 \end{pmatrix} = \begin{pmatrix} 0&0 \\\\ 0&1 \end{pmatrix}.\\\\
$$

This also means that we can write any matrix purely in terms of outer products. In the examples above, we constructed the four matrices that cover each of the single elements in a single-qubit matrix, so we can write any other single-qubit matrix in terms of them.

$$
M= \begin{pmatrix} m_{0,0}&m_{0,1} \\\\ m_{1,0}&m_{1,1} \end{pmatrix} = m_{0,0} |0\rangle\langle0|+ m_{0,1} |0\rangle\langle1|+ m_{1,0} |1\rangle\langle0|+ m_{1,1} |1\rangle\langle1|
$$

This property also extends to matrices for any number of qubits, $n$. We simply use the outer products of the corresponding $n$-bit strings.

### 2.2 Unitary and Hermitian matrices

The Hermitian conjugate $M^\dagger$ of a matrix $M$ is the combination of the transpose \(replace the bottom left element with the top right, and so on\) and the complex conjugate of each element. Two families of matrices that are very important to quantum computing are defined by their relationship with the Hermitian conjugate. One is the family of unitary matrices, for which

$$
U U^\dagger = U^\dagger U = 1.
$$

This means that the Hermitian conjugate of a unitary is its inverse: another unitary $U^\dagger$ with the power to undo the effects of $U$. All gates in quantum computing, with the exception of measurement and reset operations, can be represented by unitary matrices.

Another consequence of unitarity is that it preserves the inner product between two arbitrary states. Specifically, take two states $\left| \psi_0 \right\rangle$ and $\left| \psi_1 \right\rangle$. The inner product of these is $\left\langle \psi_0 | \psi_1 \right\rangle$. If we apply the same unitary $U$ to both, the inner product of the resulting states is exactly the same,

$$
\left( \left\langle \psi_0 \right| U^\dagger \right) \left( U \left| \psi_1 \right\rangle \right) = \left\langle \psi_0 |U^\dagger U| \psi_1 \right\rangle = \left\langle \psi_0 | \psi_1 \right\rangle.
$$

This property provides us with a useful way of thinking about these gates. It means that for any set of states $\{ \left| \psi_j \right\rangle \}$ that provide an orthonormal basis for our system, the set of states $\{ \left| \phi_j \right\rangle = U \left| \psi_j \right\rangle \}$ will also be an orthonormal basis. The unitary can then be thought of as a rotation between these bases, and can be written accordingly as

$$
U = \sum_j \left| \phi_j \right\rangle \left\langle \psi_j \right|.
$$

This is essentially the quantum version of the 'truth tables' that describe the action of classical Boolean gates.

The other important family of matrices are the Hermitian matrices. These are those that are unaffected by the Hermitian conjugate

$$
H = H^\dagger.
$$

The matrices $X$, $Y$, $Z$ and $H$ are examples of Hermitian matrices that you've already seen \(coincidentally, they are also all unitary since they are their own inverses\).

All unitary matrices and Hermitian matrices have the property of being diagonalizable. This means that they can be written in the form

$$
M = \sum_j \lambda_j |h_j\rangle\langle h_j|,
$$

where the $\lambda_j$ are the eigenvalues of the matrix and $|h_j\rangle$ are the corresponding eigenstates.

For unitaries, applying the $U U^\dagger=1$ condition in this diagonal form imples that $\lambda_j \lambda_j^* = 1$. The eigenvalues are therefore always complex numbers of magnitude 1, and so can be expressed $e^{ih_j}$ for some real value $h_j$. For Hermitian matrices the condition $H = H^\dagger$ implies $\lambda_j = \lambda_j^*$, and hence that all eigenvalues are real.

These two types of matrices therefore differ only in that one must have real numbers for eigenvalues, and the other must have complex exponentials of real numbers. This means that, for every unitary, we can define a corresponding Hermitian matrix. For this we simply reuse the same eigenstates, and use the $h_j$ from each $e^{ih_j}$ as the corresponding eigenvalue.

Similarly, for each Hermitian matrix there exists a unitary. We simply reuse the same eigenstates, and exponentiate the $h_j$ to create the eigenvalues $e^{ih_j}$. This can be expressed as

$$
U = e^{iH}
$$

Here we have used the standard definition of how to exponentiate a matrix, which has exactly the properties we require: preserving the eigenstates and exponentiating the eigenvalues.

### 2.3 Pauli decomposition

As we saw above, it is possible to write matrices entirely in terms of outer products.

$$
M= \begin{pmatrix} m_{0,0}&m_{0,1} \\\\ m_{1,0}&m_{1,1} \end{pmatrix} = m_{0,0} |0\rangle\langle0|+ m_{0,1} |0\rangle\langle1|+ m_{1,0} |1\rangle\langle0|+ m_{1,1} |1\rangle\langle1|
$$

Now we will see that it is also possible to write them completely in terms of Pauli operators. For this, the key thing to note is that

$$
\frac{1+Z}{2} = \frac{1}{2}\left[ \begin{pmatrix} 1&0 \\\\0&1 \end{pmatrix}+\begin{pmatrix} 1&0 \\\\0&-1 \end{pmatrix}\right] = |0\rangle\langle0|,\\\\\frac{1-Z}{2} = \frac{1}{2}\left[ \begin{pmatrix} 1&0 \\\\0&1 \end{pmatrix}-\begin{pmatrix} 1&0 \\\\0&-1 \end{pmatrix}\right] = |1\rangle\langle1|
$$

This shows that $|0\rangle\langle0|$ and $|1\rangle\langle1|$ can be expressed using the identity matrix and $Z$. Now, using the property that $X|0\rangle = |1\rangle$, we can also produce

$$
|0\rangle\langle1| = |0\rangle\langle0|X = \frac{1}{2}(1+Z)~X = \frac{X+iY}{2},\\\\
|1\rangle\langle0| = X|0\rangle\langle0| = X~\frac{1}{2}(1+Z) = \frac{X-iY}{2}.
$$

Since we have all the outer products, we can now use this to write the matrix in terms of Pauli matrices:

$$
M = \frac{m_{0,0}+m_{1,1}}{2}~1~+~\frac{m_{0,1}+m_{1,0}}{2}~X~+~i\frac{m_{0,1}-m_{1,0}}{2}~Y~+~\frac{m_{0,0}-m_{1,1}}{2}~Z.
$$

This example was for a general single-qubit matrix, but the corresponding result is true also for matrices for any number of qubits. We simply start from the observation that

$$
\left(\frac{1+Z}{2}\right)\otimes\left(\frac{1+Z}{2}\right)\otimes\ldots\otimes\left(\frac{1+Z}{2}\right) = |00\ldots0\rangle\langle00\ldots0|,
$$

and can then proceed in the same manner as above. In the end it can be shown that any matrix can be expressed in terms of tensor products of Pauli matrices:

$$
M = \sum_{P_{n-1},\ldots,P_0 \in \{1,X,Y,Z\}} C_{P_{n-1}\ldots,P_0}~~P_{n-1} \otimes P_{n-2}\otimes\ldots\otimes P_0.
$$

For Hermitian matrices, note that the coefficients $C_{P_{n-1}\ldots,P_0}$ here will all be real._____no_output_____## 3. Defining Universality

Just as each quantum gate can be represented by a unitary, so too can we describe an entire quantum computation by a (very large) unitary operation. The effect of this is to rotate the input state to the output state.

One possible special case of this is that the input and output states describe bit strings expressed in quantum form. The mapping of each input $x$ to its output $f(x)$ could be described by some (reversible) classical computation. Any such computation could therefore be expressed as a unitary.

$$
U = \sum_j \left| f(x) \right\rangle \left\langle x \right|.
$$

If we were able to implement any possible unitary, it would therefore mean we could acheive universality in the sense of standard digital computers.

Another special case is that the input and output states could describe a physical system, and the computation we perform is to simulate the dynamics of that system. This is an important problem that is impractical for classical computers, but is a natural application of quantum computers. The time evolution of the system in this case corresponds to the unitary that we apply, and the associated Hermitian matrix is the Hamiltonian of the system. Achieving any unitary would therefore correspond to simulating any time evolution, and engineering the effects of any Hamiltonian.

Combining these insights we can define what it means for quantum computers to be universal. It is simply the ability to achieve any desired unitary on any arbitrary number of qubits. If we have this, we know that we can reproduce anything a digital computer can do, simulate any quantum system, and do everything else that is possible for a quantum computer._____no_output_____## 4. Basic Gate Sets

Whether or not we can build any unitary from an set of basic gates depends greatly on what basic gates we have access to. For every possible realization of fault-tolerant quantum computing, there is a set of quantum operations that are most straightforward to realize. Often these consist of single and two qubit gates, most of which correspond to the set of so-called *Clifford gates*. This is a very important set of operations, which do a lot of the heavy-lifting in any quantum algorithm._____no_output_____### 4.1 Clifford Gates

To understand Clifford gates, let's start with an example that you have already seen many times: the Hadamard.

$$
H = |+\rangle\langle0|~+~ |-\rangle\langle1| = |0\rangle\langle+|~+~ |1\rangle\langle-|.
$$

This gate is expressed above using outer products, as described above. When expressed in this form, its famous effect becomes obvious: it takes $|0\rangle$, and rotates it to $|+\rangle$. More generally, we can say it rotates the basis states of the z measurement, $\{ |0\rangle,|1\rangle \}$, to the basis states of the x measurement, $\{ |+\rangle,|-\rangle \}$, and vice versa.

In this way, the effect of the Hadamard is to move information around a qubit. It swaps any information that would previously be accessed by an x measurement with that accessed by a z measurement.

The Hadamard can be combined with other gates to perform different operations, for example:

$$
H X H = Z,\\\\
H Z H = X.
$$

By doing a Hadamard before and after an $X$, we cause the action it previously applied to the z basis states to be transferred to the x basis states instead. The combined effect is then identical to that of a $Z$. Similarly, we can create an $X$ from Hadamards and a $Z$.

Similar behavior can be seen for the $S$ gate and its Hermitian conjugate,

$$
S X S^{\dagger} = Y,\\\\
S Y S^{\dagger} = -X,\\\\
S Z S^{\dagger} = Z.
$$

This has a similar effect to the Hadamard, except that it swaps $X$ and $Y$ instead of $X$ and $Z$. In combination with the Hadamard, we could then make a composite gate that shifts information between y and z.

This property of transforming Paulis into other Paulis is the defining feature of Clifford gates. Stated explicitly for the single-qubit case: if $U$ is a Clifford and $P$ is a Pauli, $U P U^{\dagger}$ will also be a Pauli. For Hermitian gates, like the Hadamard, we can simply use $U P U$.

Further examples of single-qubit Clifford gates are the Paulis themselves. These do not transform the Pauli they act on. Instead, they simply assign a phase of $-1$ to the two that they anticommute with. For example,

$$
Z X Z = -X,\\\\
Z Y Z = -Y,\\\\
Z Z Z= ~~~~Z.
$$

You may have noticed that a similar phase also arose in the effect of the $S$ gate. By combining this with a Pauli, we could make a composite gate that would cancel this phase, and swap $X$ and $Y$ in a way more similar to the Hadamard's swap of $X$ and $Z$.

For multiple-qubit Clifford gates, the defining property is that they transform tensor products of Paulis to other tensor products of Paulis. For example, the most prominent two-qubit Clifford gate is the CNOT. The property of this that we will make use of in this chapter is

$$
{ CX}_{j,k}~ (X \otimes 1)~{ CX}_{j,k} = X \otimes X.
$$

This effectively 'copies' an $X$ from the control qubit over to the target.

The process of sandwiching a matrix between a unitary and its Hermitian conjugate is known as conjugation by that unitary. This process transforms the eigenstates of the matrix, but leaves the eigenvalues unchanged. The reason why conjugation by Cliffords can transform between Paulis is because all Paulis share the same set of eigenvalues._____no_output_____### 4.2 Non-Clifford Gates

The Clifford gates are very important, but they are not powerful on their own. In order to do any quantum computation, we need gates that are not Cliffords. Three important examples are arbitrary rotations around the three axes of the qubit, $R_x(\theta)$, $R_y(\theta)$ and $R_z(\theta)$.

Let's focus on $R_x(\theta)$. As we saw above, any unitary can be expressed in an exponential form using a Hermitian matrix. For this gate, we find

$$
R_x(\theta) = e^{i \frac{\theta}{2} X}.
$$

The last section also showed us that the unitary and its corresponding Hermitian matrix have the same eigenstates. In this section, we've seen that conjugation by a unitary transforms eigenstates and leaves eigenvalues unchanged. With this in mind, it can be shown that

$$
U R_x(\theta)U^\dagger = e^{i \frac{\theta}{2} ~U X U^\dagger}.
$$

By conjugating this rotation by a Clifford, we can therefore transform it to the same rotation around another axis. So even if we didn't have a direct way to perform $R_y(\theta)$ and $R_z(\theta)$, we could do it with $R_x(\theta)$ combined with Clifford gates. This technique of boosting the power of non-Clifford gates by combining them with Clifford gates is one that we make great use of in quantum computing._____no_output_____### 4.3 Expanding the Gate Set

As another example of combining $R_x(\theta)$ with Cliffords, let's conjugate it with a CNOT.

$$
CX_{j,k} ~(R_x(\theta) \otimes 1)~ CX_{j,k} = CX_{j,k} ~ e^{i \frac{\theta}{2} ~ (X\otimes 1)}~ CX_{j,k} = e^{i \frac{\theta}{2} ~CX_{j,k} ~ (X\otimes 1)~ CX_{j,k}} = e^{i \frac{\theta}{2} ~ X\otimes X}
$$

This transforms our simple, single-qubit rotation into a much more powerful two-qubit gate. This is not just equivalent to performing the same rotation independently on both qubits. Instead, it is a gate capable of generating and manipulating entangled states.

We needn't stop there. We can use the same trick to extend the operation to any number of qubits. All that's needed is more conjugates by the CNOT to keep copying the $X$ over to new qubits.

Furthermore, we can use single-qubit Cliffords to transform the Pauli on different qubits. For example, in our two-qubit example we could conjugate by $S$ on the qubit on the right to turn the $X$ there into a $Y$:

$$
\left( I \otimes S \right)  ~e^{i \frac{\theta}{2} ~ X\otimes X}~\left( I \otimes S^\dagger \right) = e^{i \frac{\theta}{2} ~ X\otimes Y}.
$$

With these techniques, we can make complex entangling operations that act on any arbitrary number of qubits, of the form

$$
U = e^{i\frac{\theta}{2} ~ P_{n-1}\otimes P_{n-2}\otimes...\otimes P_0}, ~~~ P_j \in \{I,X,Y,Z\}.
$$

This all goes to show that combining the single and two-qubit Clifford gates with rotations around the x axis gives us a powerful set of possibilities. What's left to demonstrate is that we can use them to do anything._____no_output_____## 5. Proving Universality

As for classical computers, we will need to split this big job up into manageable chunks. We'll need to find a basic set of gates that will allow us to achieve this. As we'll see, the single- and two-qubit gates of the last section are sufficient for the task.

Suppose we wish to implement the unitary

$$
U = e^{i(aX + bZ)},
$$

but the only gates we have are $R_x(\theta) = e^{i \frac{\theta}{2} X}$ and $R_z(\theta) = e^{i \frac{\theta}{2} Z}$. The best way to solve this problem would be to use Euler angles. But let's instead consider a different method.

The Hermitian matrix in the exponential for $U$ is simply the sum of those for the $R_x(\theta)$ and $R_z(\theta)$ rotations. This suggests a naive approach to solving our problem: we could apply $R_z(2b) = e^{i bZ}$ followed by $R_x(2a) = e^{i a X}$. Unfortunately, because we are exponentiating matrices that do not commute, this approach will not work.

$$
e^{i a X} e^{i b Z} \neq e^{i(aX + bZ)}
$$

However, we could use the following modified version:

$$
U = \lim_{n\rightarrow\infty} ~ \left(e^{iaX/n}e^{ibZ/n}\right)^n.
$$

Here we split $U$ up into $n$ small slices. For each slice, it is a good approximation to say that

$$
e^{iaX/n}e^{ibZ/n} = e^{i(aX + bZ)/n}
$$

The error in this approximation scales as $1/n^2$. When we combine the $n$ slices, we get an approximation of our target unitary whose error scales as $1/n$. So by simply increasing the number of slices, we can get as close to $U$ as we need. Other methods of creating the sequence are also possible to get even more accurate versions of our target unitary.

The power of this method is that it can be used in complex cases than just a single qubit. For example, consider the unitary

$$
U = e^{i(aX\otimes X\otimes X + bZ\otimes Z\otimes Z)}.
$$

We know how to create the unitary $e^{i\frac{\theta}{2} X\otimes X\otimes X}$ from a single qubit $R_x(\theta)$ and two controlled-NOTs.

```python
qc.cx(0,2)
qc.cx(0,1)
qc.rx(theta,0)
qc.cx(0,1)
qc.cx(0,2)
```

With a few Hadamards, we can do the same for $e^{i\frac{\theta}{2} Z\otimes Z\otimes Z}$.

```python
qc.h(0)
qc.h(1)
qc.h(2)
qc.cx(0,2)
qc.cx(0,1)
qc.rx(theta,0)
qc.cx(0,1)
qc.cx(0,2)
qc.h(2)
qc.h(1)
qc.h(0)
```

This gives us the ability to reproduce a small slice of our new, three-qubit $U$:

$$
e^{iaX\otimes X\otimes X/n}e^{ibZ\otimes Z\otimes Z/n} = e^{i(aX\otimes X\otimes X + bZ\otimes Z\otimes Z)/n}.
$$

As before, we can then combine the slices together to get an arbitrarily accurate approximation of $U$.

This method continues to work as we increase the number of qubits, and also the number of terms that need simulating. Care must be taken to ensure that the approximation remains accurate, but this can be done in ways that require reasonable resources. Adding extra terms to simulate, or increasing the desired accuracy, only require the complexity of the method to increase polynomially.

Methods of this form can reproduce any unitary $U = e^{iH}$ for which $H$ can be expressed as a sum of tensor products of Paulis. Since we have shown previously that all matrices can be expressed in this way, this is sufficient to show that we can reproduce all unitaries. Though other methods may be better in practice, the main concept to take away from this chapter is that there is certainly a way to reproduce all multi-qubit unitaries using only the basic operations found in Qiskit. Quantum universality can be achieved!

This gate set is not the only one that can acheive universality. For example it can be shown that just the Hadamard and Toffoli are sufficient for universality. Multiple other gates sets have also been considered and been proven universal, each motivated by different routes toward acheiving the gates fault-tolerantly.

Everything we have discussed in this book follows the circuit model of computation. However, the circuit model is not the only universal model of quantum computation. There exists other forms of quantum computing such as the adiabatic quantum computing or the measurement based quantum computer. The fact that they are universal means that it has been proven that there is a polynomial time and resource mapping from the circuit model to these models of computing. These often leverage other quantum mechanical properties also in order to perform their computation. It is important to be aware that while these other forms of quantum computation exist, the benefits of each are hardware related. Since a universal model of quantum computation can perform any quantum algorithm, we need only stick with the circuit model and can ignore other universal models for our discussion.

There are other forms of quantum computation that are not universal, but are applicable to specific applications. For example quantum annealing is very useful to optimization and sampling problems. Annealing is the process of heating a metal to a high temperature and then allowing it to cool down slowly. This causes molten metal to flow over the surface of the metal piece and redistribute itself. This process changes many properties of the metal in question. Quantum annealing is analogous to this in some sense. It involves encoding problems into an energy landscape of sorts and then letting a quantum state explore the landscape.While normal waves may get trapped in troughs which are lower than their surroundings (local minima), quantum effects increase the speed at which the quantum states find the true lowest point on the landscape (global minima)._____no_output_____

import qiskit
qiskit.__qiskit_version_______no_output_____