# Notebook from ajrazander/qiskit-community-tutorials
Path: terra/qis_adv/single_qubit_geometric_gates.ipynb

![image.png](attachment:image.png)_____no_output_____# Single-qubit geometric gates_____no_output_____In this notebook we create a single-qubit geometric gate between the $|0\rangle$ and $|2\rangle$ state of the transmon. The ideal unitary operator describing the single-qubit geometric gate in the $\{|0\rangle, |2\rangle\}$ basis is
\begin{align}
U_g = \begin{pmatrix}
\cos\theta & e^{i\phi}\sin\theta \\
e^{-i\phi}\sin\theta & -\cos\theta
\end{pmatrix}
\end{align}
where $\theta$ and $\phi$ are controllable rotation angles.
The rotation that a geometric gate implements depends only on the path that the quantum system takes in Hilbert space during the time evolution. This gate is implemented by simultaneously driving a $2\pi$ rotation between the $|0\rangle$ to $|1\rangle$ and the $|1\rangle$ to $|2\rangle$ transitions which differ in frequency by the anharmonicity of the transmon. We therefore require properly calibrated $X$ gates between $|0\rangle$ to $|1\rangle$ and between $|1\rangle$ to $|2\rangle$ which we label $X_{01}$ and $X_{12}$, respectively. The backend provides us with a properly calibrated $X_{01}$ gate. By modulating the pulse that implements $X_{01}$ with the anharmonicity of the transmon and by scaling its amplitude we can implement a calibrated $X_{12}$ gate that has the same duration as $X_{01}$. This modulation requires a bandwidth comparable to the anharmonicity, which is typically -400 ~ -300 MHz for transmon devices. To avoid calibrating $2\pi$ rotations we implement the geometric gate by two back-to-back simultaneously driven $X_{01}$ and $X_{12}$ gates. The pulse envelope of the $X_{01}$ and $X_{12}$ gates are scaled by the complex parameters $a$ and $b$ which control the rotation angles $\theta$ and $\phi$ on the Bloch sphere spanned by $|0\rangle$ and $|2\rangle$ according to
\begin{align}
e^{i\phi}\tan\frac{\theta}{2}=\frac{a}{b}.
\end{align}

To implement the geometric gate we need to do the following steps.
1. Identify the frequency of the $|1\rangle$ to $|2\rangle$ transition using spectroscopy.
2. Identify the amplitude of the $\pi$-pulse between the $|1\rangle$ to $|2\rangle$ transition.
3. Build a discriminator using Qiskit-Ignis to discriminate the $|0\rangle$, $|1\rangle$, and $|2\rangle$.
4. Implement and measure a single-qubit geometric gate.

References:
- Abdumalikov *et al.*, Nature **496**, 482-485 (2013)
- Egger *et al*, Phys. Rev. Applied **11**, 014017 (2019)_____no_output_____

import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit

import qiskit.pulse as pulse
from qiskit.compiler import assemble

# Qiskit pulse
from qiskit.pulse import (MemorySlot, Acquire, DriveChannel, Schedule, SamplePulse,
                          InstructionScheduleMap, Play, MeasureChannel, AcquireChannel)
from qiskit.pulse.pulse_lib import Gaussian
from qiskit.qobj.utils import MeasLevel, MeasReturnType
from qiskit.scheduler import measure_all
from qiskit.visualization.pulse.qcstyle import SchedStyle

# Ignis discriminator fitter tools
from sklearn.svm import SVC
from qiskit.ignis.mitigation.measurement import CompleteMeasFitter, MeasurementFilter
from qiskit.ignis.measurement.discriminator.filters import DiscriminationFilter
from qiskit.ignis.measurement.discriminator.iq_discriminators import SklearnIQDiscriminator

%matplotlib inline_____no_output_____

### Load a high resolution pulse backend_____no_output_____

import warnings
warnings.filterwarnings('ignore')
from qiskit.tools.jupyter import *

from qiskit import IBMQ
IBMQ.load_account()
hub='update-here'
group='update-here'
project='update-here'

provider = IBMQ.get_provider(hub=hub, group=group, project=project)
backend = provider.get_backend('ibmq_armonk')_____no_output_____backend_____no_output_____config = backend.configuration()
defaults = backend.defaults()
inst_map = defaults.circuit_instruction_map
style = SchedStyle(figsize=(16, 5))  # schedule style plotting_____no_output_____

Extract the default Xp and measurement pulse for the qubit that we are going to use._____no_output_____

xp = inst_map.get('x', (0,)).instructions[0][1].pulse
d0 = DriveChannel(0)
dt = config.dt
shots = 1024_____no_output_____def add_modulation(pulse, freq: float, dt: float, scale: float = 1.) -> SamplePulse:
    """
    Add a modulation to the pulse.

    Args:
        samples: A list of AWG samples to modulate.
        freq: The frequency of the modulation to add to samples.
        dt: The cycle time.
        scale: a factor to scale the samples.

    Returns: SamplePulse with the added modulation.
    """

    if not isinstance(pulse, SamplePulse):
        samples = pulse.get_sample_pulse().samples
    else:
        samples = pulse.samples

    modulated_samples = []
    for i, amp in enumerate(samples):
        modulated_samples.append(scale * amp * np.exp(2.0j*np.pi*freq*i*dt))

    return SamplePulse(modulated_samples)

def get_job_data(job, average: bool, qubit: int, scale_factor=1):
    """Retrieve data from a job that has already run.

    Args:
        job (Job): The job whose data you want.
        average: If True, gets the data assuming data is an average.
                 If False, gets the data assuming it is for single shots.
    Return:
        list: List containing job result data.
    """
    job_results = job.result(timeout=120) # timeout parameter set to 120 s
    result_data = []
    for i in range(len(job_results.results)):
        if average: # get avg data
            result_data.append(job_results.get_memory(i)[qubit]*scale_factor)
        else: # get single data
            result_data.append(job_results.get_memory(i)[:, qubit]*scale_factor)
    return result_data

def lorenz(x, a, q_freq, b, c):
     return (a / np.pi) * (b / ((x - q_freq)**2 + b**2)) + c_____no_output_____

## Calibration of an X gate between $|1\rangle$ and $|2\rangle$_____no_output_____### Find the $|1\rangle$ to $|2\rangle$ transition

We use spectroscopy to find the transition between $|1\rangle$ and $|2\rangle$.
First, we apply a pi-pulse to the qubit to prepare the $|1\rangle$ state.
Next, we apply a longer and weaker spectroscopic pulse with a frequency detunning $\delta f$ from the $|0\rangle$ to $|1\rangle$ transition and measure.
By repeating this experiment for different values of $\delta f$ we can identify the frequency detunning of the transition between $|1\rangle$ and $|2\rangle$ which is called the anharmonicity $\alpha$._____no_output_____**Note**: Here we will drive a transition at +400 MHz which corresponds to an image (created by the IQ mixer) of the anharmonicity. Since the LO is +50 MHz above the $|0\rangle$ to $|1\rangle$ transition frequency of the qubit and that the true qubit anharmonicity is approximately -350 MHz the +400 MHz thus corresponds to an image of the anharmonicity. This image transition of the anharmonicity has proven superior at creating a $\pi$-pulse between the $|1\rangle$ and $|2\rangle$ states._____no_output_____

exps = 74

# Frequency range to scan in GHz
# Anharmonicity is around -349 MHz and its image is around +400 MHz.
frequency_offsets = np.linspace(0.396, 0.404, exps)

schedules = []
for freq in frequency_offsets:
    sched = Schedule(name='Frequency %f MHz' % (freq/1e3))

    spec_pulse = add_modulation(Gaussian(3200, 0.1, 800), freq*1e9, dt)
    #spec_pulse = add_modulation(xp, freq*1e9, dt, scale=0.88)

    sched += Play(xp, d0)
    sched += Play(spec_pulse, d0)
    sched += measure_all(backend) << sched.duration

    schedules.append(sched)_____no_output_____schedules[32].draw(plot_range=[0, 7500], style=style, channels=[DriveChannel(0), MeasureChannel(0)])_____no_output_____qobj = assemble(schedules, backend, meas_level=1,
                meas_return=MeasReturnType.AVERAGE,
                shots=shots)_____no_output_____job = backend.run(qobj)
job.job_id()_____no_output_____job.status()_____no_output_____result = get_job_data(job, True, 0, 1e-15)
spec_signal = abs(np.array(result))
spec_signal = np.average(spec_signal) - spec_signal_____no_output_____popt, pcov = curve_fit(lorenz, frequency_offsets, spec_signal, [0.015, 0.4, 0.0045, 0.1])
anharmonicity = popt[1]*1e9_____no_output_____anharmonicity_____no_output_____fig = plt.figure()
plt.rcParams['font.size'] = 20
plt.plot(frequency_offsets, lorenz(frequency_offsets, *popt))
plt.plot(frequency_offsets, spec_signal, 'ok')
plt.xlabel('Frequency detunning (MHz)')
plt.ylabel('Signal (arb units.)')
plt.title('Anharmonicity %.2f MHz' % (anharmonicity/1e6));_____no_output_____

## Calibrate a $\pi$-pulse between 1 and 2

Since we now know the anharmonicity $\alpha$ of the transmon, we can apply a frequency modulation $e^{i\alpha t}$ to the pulse envelope $\Omega(t)$ that implements an $X$ gate between $|0\rangle$ and $|1\rangle$ to drive the $|1\rangle$ to $|2\rangle$ transition.
However, the resulting pulse will not have the correct amplitude to implement a $\pi$-pulse between $|1\rangle$ and $|2\rangle$.
To calibrate this pulse we scan a scaling factor $\eta$ of the pulse envelope $\eta\,\Omega(t)\,e^{i\alpha t}$ to measure a Rabi oscillation between $|1\rangle$ and $|2\rangle$._____no_output_____

xp12_scales = np.linspace(0., 1.6, 74)

schedules = []
for amp in xp12_scales:
    sched = Schedule(name='Amplitude {:.3f} (\\% AWG output)'.format(amp))

    xp12 = add_modulation(xp, anharmonicity, dt, scale=amp)

    sched += Play(xp, d0)
    sched += Play(xp12, d0)
    sched += measure_all(backend) << sched.duration

    schedules.append(sched)_____no_output_____schedules[73].draw(plot_range=[0, 2000], style=style, channels=[d0, MeasureChannel(0)])_____no_output_____qobj = assemble(schedules, backend, meas_level=1,
                meas_return=MeasReturnType.AVERAGE,
                shots=1024)_____no_output_____job = backend.run(qobj)
job.job_id()_____no_output_____job.status()_____no_output_____def amp_func(x, a, b, c):
    """Function used to fit the amplitude scan."""
    return a*np.cos(np.pi*x/b)+c_____no_output_____signal = np.real(np.array(get_job_data(job, True, 0, 1e-15)))

popt, pcov = curve_fit(amp_func, xp12_scales, signal, [0.8, 2.0, 0.5])
xp12_scale = abs(popt[1])_____no_output_____fig = plt.figure()
plt.plot(xp12_scales, amp_func(xp12_scales, *popt))
plt.plot(xp12_scales, signal, 'ok')
plt.plot([xp12_scale]*2, [min(signal), max(signal)])
plt.xlabel('Amplitude of xp12 relative to xp')
plt.ylabel('Signal (arb. units)')
plt.title('Relative amplitude %.2f %%' % (xp12_scale*100));_____no_output_____

## Create a 0, 1, 2 discriminator with Qiskit Ignis

We should now have a calibrated $\pi$-pulse between the $|1\rangle$ and $|2\rangle$ states.
We check this by preparing calibration schedules to see if we can distinguish between the $|0\rangle$, $|1\rangle$ and $|2\rangle$ states in the IQ plane.
This measurement is then used to build a 0-1-2 discriminator with Qiskit-Ignis.

Note that we also added a small modulation to the readout pulse to improve the efficiency of the 0-1-2 discriminator._____no_output_____

# The measurement pulse may not be setup to distinguish between 0, 1, and 2
# This line of code allows us to adjust the frequency and amplitude
# of the measurement pulse
meas = measure_all(backend).instructions[1][1].pulse
meas_freq_shift = -0.075e6
meas_pulse = add_modulation(meas, meas_freq_shift, dt, scale=0.9)

meas_sched = Schedule(name='meas %f' % meas_freq_shift)
meas_sched += Play(meas_pulse, MeasureChannel(0))
meas_sched += Acquire(len(meas.samples), AcquireChannel(0), MemorySlot(0))

schedules = []

cal0 = Schedule(name='cal_0'.format(amp))
cal0 += meas_sched << cal0.duration
schedules.append(cal0)

cal1 = Schedule(name='cal_1'.format(amp))
cal1 += Play(xp, d0)
cal1 += meas_sched << cal1.duration
schedules.append(cal1)

# Create the calibrated pi-pulse between |1> and |2>
xp12 = add_modulation(xp, anharmonicity, dt, scale=xp12_scale)

cal2 = Schedule(name='cal_2'.format(amp))
cal2 += Play(xp, d0)
cal2 += Play(xp12, d0)
cal2 += meas_sched << cal2.duration
schedules.append(cal2)_____no_output_____schedules[2].draw(style=style, channels=[d0, MeasureChannel(0), AcquireChannel(0)])_____no_output_____qobj = assemble(schedules, backend, meas_level=1,
                meas_return=MeasReturnType.SINGLE,
                shots=1024)_____no_output_____job = backend.run(qobj)
job.job_id()_____no_output_____job.status()_____no_output_____cal_result_disc = job.result(timeout=3600)_____no_output_____svc = SVC(C=0.01, kernel="rbf", gamma="scale")

svc_discriminator = SklearnIQDiscriminator(svc, cal_result_disc, [0], ['0', '1', '2'])
filter012 = DiscriminationFilter(svc_discriminator)_____no_output_____

The data in the figure below should show three clearly seperated clusters in the IQ plan which correspond to $|0\rangle$, $|1\rangle$, and $|2\rangle$. Since the discriminator must discriminate three states we use the support vector machine provided by Scikit learn._____no_output_____

fig, ax = plt.subplots(1, 1, figsize=(8,5))
svc_discriminator.plot(ax, flag_misclassified=False, show_boundary=True);_____no_output_____

This measurement can also be used to correct for readout errors using a readout error mitigation tools provided by Qiskit Ignis. In the code below the discriminator returns the states encoded in binary, therefore 0, 1, and 2 are represented by the strings '0', '1', and '10', respectively._____no_output_____

cal_counts = filter012.apply(cal_result_disc).get_counts()
cal_matrix = np.array([[cal_counts[0].get('0',0), cal_counts[1].get('0',0), cal_counts[2].get('0',0)],
                       [cal_counts[0].get('1',0), cal_counts[1].get('1',0), cal_counts[2].get('1',0)],
                       [cal_counts[0].get('10',0), cal_counts[1].get('10',0), cal_counts[2].get('10',0)]])/shots
meas_filter = MeasurementFilter(cal_matrix, ['0', '1', '10'])_____no_output_____plt.imshow(cal_matrix, cmap='gray')
plt.xlabel('Prepared state')
plt.ylabel('Measured state');_____no_output_____

## Geometric single-qubit gates

We now have all the tools needed to implement a single-qubit geometric gate between the $|0\rangle$ and $|2\rangle$ states. This is done by *simultaneously* applying two back-to-back $\pi$-pulses (which corresponds to a $2\pi$ rotation) between the $|0\rangle$ and $|1\rangle$ states and the $|1\rangle$ and $|2\rangle$ states. By using two $\pi$-pulses we save ourselves the task of having to calibrate the $2\pi$-pulses. The pulses that implement the single-qubit geometric gate have the envelope

\begin{align}
a\,\Omega(t)+b\,\eta\,\Omega(t)\,e^{i\alpha t}.
\end{align}

The complex scalars $a$ and $b$ define the rotation angle $\theta$ and phase $\phi$ of the unitary gate that will be implemented on the Bloch sphere created by $|0\rangle$ and $|2\rangle$. $\theta$ and $\phi$ are related to $a$ and $b$ by

\begin{align}
e^{i\phi}\tan\frac{\theta}{2}=\frac{a}{b}.
\end{align}

Note that the pulses that create the single-qubit geometric gate cannot be implement with the `SetFrequency` instruction._____no_output_____

def geometric(a: complex, b: complex, xp: SamplePulse, xp12: SamplePulse, name='') -> Schedule:
    """
    Create a single-qubit geometric gate between the 0 and 2 state.
    In principle the geometric gate is built from a 2pi rotation between 0 and
    1 and a 2pi rotation between 1 and 2. Since we have not calibrated the 2pi
    rotations we will emulate them with two pi back-to-back pi rotations.

    The geometric gate creates a rotation on the Bloch Sphere between |0> and |2>.
    The rotation angle theta and phase phi on the (0,2) Bloch sphere are
    given by e^(i*phi)*tan(theta/2)=a/b.

    Args:
        a: Parameter to control the geometric rotation.
        b: Parameter to control the geometric rotation.
        xp: Calibrated X gate between |0> and |1>.
        xp12: Calibrated X gate between |1> and |2>.

    Returns: A schedules with the geometric gate.
    """

    if abs(abs(a)**2 + abs(b)**2 - 1.0) > 1e-4:
        raise ValueError('a and b must satisfy |a|**2+|b|**2')

    geom_samples = a*xp.samples + b*xp12.samples
    geom = SamplePulse(geom_samples)

    sched = Schedule(name='Geometric (%f, %f) ' % (a, b)+name)
    sched += Play(geom, d0)
    sched += Play(geom, d0)
    sched += meas_sched << sched.duration

    return sched_____no_output_____schedules = [cal0, cal1, cal2]
angles = np.linspace(0., np.pi, 70)
for angle in angles:
    a = np.sin(angle/2)
    schedules.append(geometric(a, np.sqrt(1.0-a*a), xp, xp12))_____no_output_____

The schedule below shows the single-qubit geometric gate, implemented as two back-to-back superpositions of $X_{01}$ and $X_{12}$. These pulses can only be implemented on high resolution devices and cannot be implemented with the `SetFrequency` instruction._____no_output_____

schedules[65].draw(plot_range=[0, 1500], style=style, channels=[d0, MeasureChannel(0)])_____no_output_____qobj = assemble(schedules, backend, meas_level=1,
                meas_return=MeasReturnType.SINGLE,
                shots=1024)_____no_output_____job = backend.run(qobj)
job.job_id()_____no_output_____job.status()_____no_output_____geom_result = job.result(timeout=3600)_____no_output_____

To analyse the data we first apply the the 012 discriminator that we build with Qiskit-Ignis. Next, we employ our measurement error mitigation filter to correct the readout._____no_output_____

# Apply discriminator
geom_discriminated = filter012.apply(geom_result)

# Mitigate readout errors
geom_mitigated = meas_filter.apply(geom_discriminated)

# Compute the population counts
counts = geom_mitigated.get_counts()
pop0 = [cnt.get('0',0)/shots for cnt in counts[3:]]
pop1 = [cnt.get('1',0)/shots for cnt in counts[3:]]
pop2 = [cnt.get('10',0)/shots for cnt in counts[3:]]_____no_output_____fig, ax = plt.subplots(1, 2, figsize=(14,5), tight_layout=True)

svc_discriminator.plot(ax[0], flag_misclassified=False, show_boundary=True, show_fitting_data=False);
for idx in range(len(angles)):
    iqs = geom_result.get_memory(idx+3)
    ax[0].scatter(np.real(iqs), np.imag(iqs), s=3, color='k',alpha=0.05)

ax[0].set_title('')

ax[1].plot(np.linspace(0, np.pi, 100), np.cos(np.linspace(0, np.pi, 100))**2, '--k')
ax[1].plot(np.linspace(0, np.pi, 100), np.sin(np.linspace(0, np.pi, 100))**2, '--k', label='Ideal')
ax[1].plot(angles, pop0, 'oC0', label='$|0\\rangle$')
ax[1].plot(angles, pop1, 'oC2', label='$|1\\rangle$')
ax[1].plot(angles, pop2, 'oC1', label='$|2\\rangle$')
ax[1].set_xlabel('Rotation angle (rad)')
ax[1].set_ylabel('Transmon population')
ax[1].set_xticks([0, np.pi/2, np.pi])
ax[1].set_xticklabels(['0', '$\pi/2$', '$\pi$'])
ax[1].set_ylim([0, 1.3])
ax[1].set_yticks([0, 0.25, 0.50, 0.75, 1.0])
ax[1].legend(ncol=4, loc=1, fontsize=16, columnspacing=0);_____no_output_____import qiskit.tools.jupyter
%qiskit_version_table
%qiskit_copyright_____no_output_____


# Notebook from alberwan/spark-nlp-workshop
Path: jupyter/enterprise/healthcare/Disambiguation.ipynb

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/jupyter/enterprise/healthcare/Disambiguation.ipynb)_____no_output_____

<code>
import json

with open('keys.json') as f:
    license_keys = json.load(f)

license_keys.keys()
_____no_output_____import os

# Install java
! apt-get install -y openjdk-8-jdk-headless -qq > /dev/null
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["PATH"] = os.environ["JAVA_HOME"] + "/bin:" + os.environ["PATH"]
! java -version

secret = license_keys['secret']
os.environ['SPARK_NLP_LICENSE'] = license_keys['SPARK_NLP_LICENSE']
os.environ['JSL_OCR_LICENSE'] = license_keys['JSL_OCR_LICENSE']
os.environ['AWS_ACCESS_KEY_ID']= license_keys['AWS_ACCESS_KEY_ID']
os.environ['AWS_SECRET_ACCESS_KEY'] = license_keys['AWS_SECRET_ACCESS_KEY']
version = license_keys['version']

! python -m pip install --upgrade spark-nlp-jsl==$version  --extra-index-url https://pypi.johnsnowlabs.com/$secret

import sparknlp

print (sparknlp.version())

import json
import os
from pyspark.ml import Pipeline
from pyspark.sql import SparkSession

from sparknlp.annotator import *
from sparknlp_jsl.annotator import *
from sparknlp.base import *
import sparknlp_jsl

def start(secret):
    builder = SparkSession.builder \
        .appName("Spark NLP Licensed") \
        .master("local[*]") \
        .config("spark.driver.memory", "16G") \
        .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer") \
        .config("spark.kryoserializer.buffer.max", "2000M") \
        .config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.11:"+version) \
        .config("spark.jars", "https://pypi.johnsnowlabs.com/"+secret+"/spark-nlp-jsl-"+version+".jar")

    return builder.getOrCreate()

spark = start(secret) # if you want to start the session with custom params as in start function above
# sparknlp_jsl.start(secret)openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-8u252-b09-1~18.04-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)
Looking in indexes: https://pypi.org/simple, https://pypi.johnsnowlabs.com/8zvTuUjWPt
Collecting spark-nlp-jsl==2.5.2
  Downloading https://pypi.johnsnowlabs.com/8zvTuUjWPt/spark-nlp-jsl/spark_nlp_jsl-2.5.2-py3-none-any.whl
Collecting spark-nlp==2.5.2
  Downloading https://files.pythonhosted.org/packages/3e/b0/c272273674b5810c0909b369c57669197907a15d84bbdf058007bb909c99/spark_nlp-2.5.2-py2.py3-none-any.whl (122kB)
     |████████████████████████████████| 133kB 4.9MB/s
Collecting pyspark==2.4.4
  Downloading https://files.pythonhosted.org/packages/87/21/f05c186f4ddb01d15d0ddc36ef4b7e3cedbeb6412274a41f26b55a650ee5/pyspark-2.4.4.tar.gz (215.7MB)
     |████████████████████████████████| 215.7MB 56kB/s
Collecting py4j==0.10.7
  Downloading https://files.pythonhosted.org/packages/e3/53/c737818eb9a7dc32a7cd4f1396e787bd94200c3997c72c1dbe028587bd76/py4j-0.10.7-py2.py3-none-any.whl (197kB)
     |████████████████████████████████| 204kB 44.5MB/s
Building wheels for collected packages: pyspark
  Building wheel for pyspark (setup.py) ... done
  Created wheel for pyspark: filename=pyspark-2.4.4-py2.py3-none-any.whl size=216130388 sha256=8ca2df845b5d966fe4f2882a58b65d75d79b2afd191d8f5380ab96b8271e9c05
  Stored in directory: /root/.cache/pip/wheels/ab/09/4d/0d184230058e654eb1b04467dbc1292f00eaa186544604b471
Successfully built pyspark
Installing collected packages: spark-nlp, py4j, pyspark, spark-nlp-jsl
Successfully installed py4j-0.10.7 pyspark-2.4.4 spark-nlp-2.5.2 spark-nlp-jsl-2.5.2
2.5.2
# Sample data
text = "The show also had a contestant named Donald Trump " \
               + "who later defeated Christina Aguilera on the way to become Female Vocalist Champion in the 1989 edition of Star Search in the United States. "
data = spark.createDataFrame([
            [text]]) \
            .toDF("text").cache()
_____no_output__________no_output_____# Preprocessing pipeline
da = DocumentAssembler().setInputCol("text").setOutputCol("document")
sd = SentenceDetector().setInputCols("document").setOutputCol("sentence")
tk = Tokenizer().setInputCols("sentence").setOutputCol("token")
emb = WordEmbeddingsModel.pretrained().setOutputCol("embs")
semb = SentenceEmbeddings().setInputCols("sentence","embs").setOutputCol("sentence_embeddings")
ner = NerDLModel.pretrained().setInputCols("sentence","token","embs").setOutputCol("ner")
nc = NerConverter().setInputCols("sentence","token","ner").setOutputCol("ner_chunk").setWhiteList(["PER"])
disambiguator = NerDisambiguator() \
    .setS3KnowledgeBaseName("i-per") \
    .setInputCols("ner_chunk", "sentence_embeddings") \
    .setOutputCol("disambiguation") \
    .setNumFirstChars(5)
pl = Pipeline().setStages([da,sd,tk,emb,semb,ner,nc,disambiguator])
data = pl.fit(data).transform(data)
data.selectExpr("explode(disambiguation)").show(10, False)glove_100d download started this may take some time.
Approximate size to download 145.3 MB
[OK!]
ner_dl download started this may take some time.
Approximate size to download 13.6 MB
[OK!]
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|col                                                                                |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|[disambiguation, 37, 48, http://en.wikipedia.org/?curid=4848272, http://en.wikipedia.org/?curid=31698421, http://en.wikipedia.org/?curid=55907961, [chunk -> Donald Trump, titles -> donald trump ::::: donald crump ::::: donald frump, links -> http://en.wikipedia.org/?curid=4848272 ::::: http://en.wikipedia.org/?curid=31698421 ::::: http://en.wikipedia.org/?curid=55907961, beginInText -> 37, scores -> 0.9637175040283449, 0.9555978783336097, 0.10186673596888873, categories -> Businesspeople, Politicians, Businesspeople, Businesspeople, Politicians, ids -> 4848272, 31698421, 55907961, endInText -> 48], []]|
|[disambiguation, 69, 86, http://en.wikipedia.org/?curid=144171, http://en.wikipedia.org/?curid=6636454, [chunk -> Christina Aguilera, titles -> christina aguilera ::::: christina aguilar, links -> http://en.wikipedia.org/?curid=144171 ::::: http://en.wikipedia.org/?curid=6636454, beginInText -> 69, scores -> 0.975820790095742, 0.9726838470180229, categories -> Musicians, Singers, Actors, Businesspeople, Musicians, Singers, ids -> 144171, 6636454, endInText -> 86], []]                                                                                |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

_____no_output_____
</code>


# Notebook from dustlrdk/noise2self
Path: notebooks/Single Cell - Figure.ipynb

# Single-Cell Gene Expression_____no_output_____

<code>
%load_ext autoreload
%autoreload 2
%matplotlib inline

import sys
sys.path.append("..")_____no_output_____dir = '../figs/singlecell/'_____no_output_____import numpy as np
import matplotlib.pyplot as plt
import scprep
import numpy as np
import pandas as pd
from sklearn.utils.extmath import randomized_svd_____no_output_____def mse(x, y):
    return ((x - y)**2).mean()
def norm(x):
    return (x**2).mean()_____no_output_____data_raw = scprep.io.load_csv('https://github.com/KrishnaswamyLab/PHATE/raw/master/data/BMMC_myeloid.csv.gz')
#data_raw.to_pickle('data/BMMC_myeloid.gzip')
#data_raw = pd.read_pickle('data/BMMC_myeloid.gzip')_____no_output_____data_raw.shape_____no_output_____scprep.plot.plot_library_size(data_raw, cutoff=1000)_____no_output_____data_raw = scprep.filter.filter_library_size(data_raw, cutoff=1000)
data_raw = scprep.filter.remove_rare_genes(data_raw, min_cells=10)_____no_output_____data_raw.shape_____no_output_____data = scprep.normalize.library_size_normalize(data_raw)
data = scprep.transform.sqrt(data)_____no_output_____# Input p is percent to set aside for validation

def split_data(df, p):
    X_target = np.random.binomial(df.values, p)
    X_masked = df.values - X_target
    df_masked = pd.DataFrame(data = X_masked,
                                 index = df.index,
                                 columns = df.columns)
    df_target = pd.DataFrame(data = X_target,
                                 index = df.index,
                                 columns = df.columns)
    return df_masked, df_target

data_masked, data_target = split_data(data_raw, 0.5)

data_masked = scprep.normalize.library_size_normalize(data_masked)
#data_masked = scprep.transform.log(data_masked)
data_masked = scprep.transform.sqrt(data_masked)

data_target = scprep.normalize.library_size_normalize(data_target)
#data_target = scprep.transform.log(data_masked)
data_target = scprep.transform.sqrt(data_target)_____no_output_____X1 = data_masked.values
X2 = data_target.values_____no_output_____from sklearn import linear_model
from sklearn.model_selection import train_test_split_____no_output_____X_train, X_test, Y_train, Y_test = train_test_split(X1, X2)_____no_output_____U, S, V = randomized_svd(X_train, n_components = 100)_____no_output_____mses = []
rank_range = range(6, 30)
for k in rank_range:
    prediction = X_test.dot(V[:k,:].T.dot(np.diag(1/S[:k]).dot(U[:,:k].T).dot(Y_train)))
    current_mse = mse(prediction, Y_test)
    mses.append(current_mse)_____no_output_____opt = rank_range[np.argmin(mses)]
opt_____no_output_____plt.plot(rank_range, mses)
plt.axvline(opt, color='k', linestyle='--')
plt.title("Calibrating PC Regression")
plt.xlabel("Rank")
plt.ylabel("N2S Loss")_____no_output_____plt.plot(rank_range, mses)
plt.axvline(opt, color='k', linestyle='--')

plt.yticks([0.0872, 0.0878])

# Hide the right and top spines
plt.gca().spines['right'].set_visible(False)
plt.gca().spines['top'].set_visible(False)
# plt.savefig(dir + 'pc_regression.pdf')_____no_output_____k = opt
U, S, V = randomized_svd(X1, n_components = 100)
prediction = X1.dot(V[:k,:].T.dot(np.diag(1/S[:k]).dot(U[:,:k].T).dot(X2)))_____no_output_____reconstructed_data = pd.DataFrame(prediction, index = data.index, columns=data.columns)_____no_output_____plt.figure(figsize=(8,6))

plt.scatter(data['Mpo'], data['Klf1'], c=data['Ifitm1'], cmap='inferno', s=1)
plt.xlabel('Mpo')
plt.ylabel('Klf1')
plt.title('Raw Data')

plt.colorbar(label='Ifitm1')

plt.tight_layout()
# plt.savefig(dir + 'raw_data_with_labels.pdf')_____no_output_____plt.figure(figsize=(8,6))

plt.scatter(data['Mpo'], data['Klf1'], c=data['Ifitm1'], cmap='inferno', s=1)
plt.colorbar()

plt.yticks([0.0, 1.0, 2.0, 3.0])
plt.xticks([0, 6, 12])

# Hide the right and top spines
plt.gca().spines['right'].set_visible(False)
plt.gca().spines['top'].set_visible(False)
plt.tight_layout()

# plt.savefig(dir + 'raw_data_with_colorbar.pdf')_____no_output_____plt.figure(figsize=(6,6))

plt.scatter(data['Mpo'], data['Klf1'], c=data['Ifitm1'], cmap='inferno', s=1)

plt.yticks([])
plt.xticks([])

# Hide the right and top spines
plt.gca().spines['right'].set_visible(False)
plt.gca().spines['top'].set_visible(False)
plt.gca().spines['bottom'].set_visible(False)
plt.gca().spines['left'].set_visible(False)

plt.tight_layout()

# plt.savefig(dir + 'raw_data_noaxis.png')_____no_output_____plt.figure(figsize=(6,6))

plt.scatter(data['Mpo'], data['Klf1'], c=data['Ifitm1'], cmap='inferno', s=1)

plt.yticks([0.0, 1.0, 2.0, 3.0])
plt.xticks([0, 6, 12])

# Hide the right and top spines
plt.gca().spines['right'].set_visible(False)
plt.gca().spines['top'].set_visible(False)
plt.tight_layout()

# plt.savefig(dir + 'raw_data.png')_____no_output_____U, S, V = randomized_svd(X1, n_components = 200)_____no_output_____k = 17
prediction = X1.dot(V[:k,:].T.dot(np.diag(1/S[:k]).dot(U[:,:k].T).dot(X2)))
reconstructed_data = pd.DataFrame(prediction, index = data.index, columns=data.columns)_____no_output_____plt.figure(figsize=(6,6))

plt.scatter(reconstructed_data['Mpo'], reconstructed_data['Klf1'], c=reconstructed_data['Ifitm1'], cmap='inferno', s=1)

plt.yticks([0.0, 1.0])
plt.xticks([0, 4, 8])

# Hide the right and top spines
plt.gca().spines['right'].set_visible(False)
plt.gca().spines['top'].set_visible(False)
plt.tight_layout()

# plt.savefig(dir + 'denoised_data.pdf')_____no_output_____plt.figure(figsize=(6,6))

plt.scatter(reconstructed_data['Mpo'], reconstructed_data['Klf1'], c=reconstructed_data['Ifitm1'], cmap='inferno', s=1)

plt.yticks([])
plt.xticks([])

# Hide the right and top spines
plt.gca().spines['right'].set_visible(False)
plt.gca().spines['top'].set_visible(False)
plt.gca().spines['bottom'].set_visible(False)
plt.gca().spines['left'].set_visible(False)

# Hide the right and top spines
plt.gca().spines['right'].set_visible(False)
plt.gca().spines['top'].set_visible(False)
plt.tight_layout()

# plt.savefig(dir + 'denoised_data_noaxis.pdf')_____no_output_____k = 3
prediction = X1.dot(V[:k,:].T.dot(np.diag(1/S[:k]).dot(U[:,:k].T).dot(X2)))
reconstructed_data = pd.DataFrame(prediction, index = data.index, columns=data.columns)_____no_output_____plt.figure(figsize=(6,6))

plt.scatter(reconstructed_data['Mpo'], reconstructed_data['Klf1'], c=reconstructed_data['Ifitm1'], cmap='inferno', s=1)

plt.yticks([0.0, 1.0])
plt.xticks([0, 4, 8])

plt.tight_layout()

# Hide the right and top spines
plt.gca().spines['right'].set_visible(False)
plt.gca().spines['top'].set_visible(False)
# plt.savefig(dir + 'under_corrected_data.pdf')_____no_output_____plt.figure(figsize=(6,6))

plt.scatter(reconstructed_data['Mpo'], reconstructed_data['Klf1'], c=reconstructed_data['Ifitm1'], cmap='inferno', s=1)

plt.yticks([])
plt.xticks([])

# Hide the right and top spines
plt.gca().spines['right'].set_visible(False)
plt.gca().spines['top'].set_visible(False)
plt.gca().spines['bottom'].set_visible(False)
plt.gca().spines['left'].set_visible(False)

plt.tight_layout()

# Hide the right and top spines
plt.gca().spines['right'].set_visible(False)
plt.gca().spines['top'].set_visible(False)
# plt.savefig(dir + 'under_corrected_data_noaxis.pdf')_____no_output_____k = 300
prediction = X1.dot(V[:k,:].T.dot(np.diag(1/S[:k]).dot(U[:,:k].T).dot(X2)))
reconstructed_data = pd.DataFrame(prediction, index = data.index, columns=data.columns)_____no_output_____plt.figure(figsize=(6,6))

plt.scatter(reconstructed_data['Mpo'], reconstructed_data['Klf1'], c=reconstructed_data['Ifitm1'], cmap='inferno', s=1)

plt.yticks([0.0, 1.0])
plt.xticks([0, 4, 8])

plt.tight_layout()

# Hide the right and top spines
plt.gca().spines['right'].set_visible(False)
plt.gca().spines['top'].set_visible(False)
# plt.savefig(dir + 'overcorrected_data.pdf')_____no_output_____plt.figure(figsize=(6,6))

plt.scatter(reconstructed_data['Mpo'], reconstructed_data['Klf1'], c=reconstructed_data['Ifitm1'], cmap='inferno', s=1)

plt.yticks([])
plt.xticks([])

# Hide the right and top spines
plt.gca().spines['right'].set_visible(False)
plt.gca().spines['top'].set_visible(False)
plt.gca().spines['bottom'].set_visible(False)
plt.gca().spines['left'].set_visible(False)

plt.tight_layout()

# Hide the right and top spines
plt.gca().spines['right'].set_visible(False)
plt.gca().spines['top'].set_visible(False)
# plt.savefig(dir + 'overcorrected_data_noaxis.pdf')_____no_output_____
</code>


# Notebook from PavanK8/DataQuest-Guided-Projects
Path: 5_Visualizing_the_gender_gap_in_college_degrees.ipynb

# Visualizing the gender gap in college degrees
In this project, we'll visualize the gender gap across college degrees. We will see, how over the time period of 1970 - 2012, how many graduates are Women in various disciplenes of `STEM`, `Liberal Arts` and `Other categories`.

[The Department of Education Statistics](http://nces.ed.gov/programs/digest/2013menu_tables.asp) releases a data set annually containing the percentage of bachelor's degrees granted to women from 1970 to 2012. The data set is broken up into 17 categories of degrees, with each column as a separate category.

Randal Olson, a data scientist at University of Pennsylvania, has cleaned the data set and made it available on his personal website. You can download the dataset Randal compiled [here](http://www.randalolson.com/wp-content/uploads/percent-bachelors-degrees-women-usa.csv)._____no_output_____

%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt

women_degrees = pd.read_csv('percent-bachelors-degrees-women-usa.csv')
cb_dark_blue = (0/255,107/255,164/255)
cb_orange = (255/255, 128/255, 14/255)
stem_cats = ['Engineering', 'Computer Science', 'Psychology', 'Biology', 'Physical Sciences', 'Math and Statistics']

fig = plt.figure(figsize=(18, 3))

for sp in range(0,6):
    ax = fig.add_subplot(1,6,sp+1)
    ax.plot(women_degrees['Year'], women_degrees[stem_cats[sp]], c=cb_dark_blue, label='Women', linewidth=3)
    ax.plot(women_degrees['Year'], 100-women_degrees[stem_cats[sp]], c=cb_orange, label='Men', linewidth=3)
    ax.spines["right"].set_visible(False)
    ax.spines["left"].set_visible(False)
    ax.spines["top"].set_visible(False)
    ax.spines["bottom"].set_visible(False)
    ax.set_xlim(1968, 2011)
    ax.set_ylim(0,100)
    ax.set_title(stem_cats[sp])
    ax.tick_params(bottom="off", top="off", left="off", right="off")

    if sp == 0:
        ax.text(2005, 87, 'Men')
        ax.text(2002, 8, 'Women')
    elif sp == 5:
        ax.text(2005, 62, 'Men')
        ax.text(2001, 35, 'Women')
plt.show()_____no_output_____

As mentioned earlier, the data set contains 17 degrees. For better understanding it can be divided into three categories, `STEM`, `liberal arts` and `Other`. _____no_output_____

stem_cats = ['Psychology', 'Biology', 'Math and Statistics', 'Physical Sciences', 'Computer Science', 'Engineering']
lib_arts_cats = ['Foreign Languages', 'English', 'Communications and Journalism', 'Art and Performance', 'Social Sciences and History']
other_cats = ['Health Professions', 'Public Administration', 'Education', 'Agriculture','Business', 'Architecture']_____no_output_____fig, axs = plt.subplots(nrows = 6, ncols = 3, figsize = (10,18))

def plot_in_col(degrees, col):
    '''
    Function to plot a list of degrees within a column
    '''
    for idx, degree in enumerate(degrees):
        axs[idx, col].plot(women_degrees['Year'], women_degrees[degree], c=cb_dark_blue, label='Women', linewidth=3)
        axs[idx, col].plot(women_degrees['Year'], 100 - women_degrees[degree], c=cb_orange, label='Men', linewidth=3)
        axs[idx, col].set_title(degree, color = 'grey', fontweight = 3)
        axs[idx, col].tick_params(bottom="off", top="off", left="off", right="off", labelbottom='off')
        axs[idx, col].set_yticks([0,100])
        axs[idx, col].set_yticklabels(['0','100'],color= 'grey')
        axs[idx, col].axhline(50, c=(171/255, 171/255, 171/255), alpha=0.3)
        axs[idx, col].spines["right"].set_visible(False)
        axs[idx, col].spines["top"].set_visible(False)
        axs[idx, col].spines["left"].set_color(c=(171/255, 171/255, 171/255, 0.3))
        axs[idx, col].spines["bottom"].set_color(c=(171/255, 171/255, 171/255, 0.3))

plot_in_col(stem_cats, 0)
plot_in_col(lib_arts_cats, 1)
plot_in_col(other_cats, 2)
fig.delaxes(axs[5][1])

axs[0][0].text(2000, 70, 'Women', color = 'grey')
axs[0][0].text(2005, 28, 'Men', color = 'grey')

axs[0][1].text(2000, 62, 'Women', color = 'grey')
axs[0][1].text(2005, 35, 'Men', color = 'grey')

axs[0][2].text(2000, 78, 'Women', color = 'grey')
axs[0][2].text(2005, 20, 'Men', color = 'grey')

axs[5][0].text(2000, 72, 'Men', color = 'grey')
axs[5][0].text(2005, 22, 'Women', color = 'grey')

axs[5][2].text(2000, 65, 'Men', color = 'grey')
axs[5][2].text(2005, 35, 'Women', color = 'grey')

def clean_axes(row, col):
    '''
    Function to do repeated cleaning of axes
    '''
    axs[row][col].tick_params(labelbottom = 'on')
    axs[row][col].set_xticks(range(1970, 2016, 15))
    axs[row][col].set_xticklabels(range(1970, 2016, 15), color = 'grey')

clean_axes(5, 0)
clean_axes(4, 1)
clean_axes(5, 2)

plt.subplots_adjust(hspace=0.35)
plt.savefig("gender_degrees.png", bbox_inches = 'tight', pad_inches=0.2)_____no_output_____

### Observations : [The Gender Gap in college degrees](#observation)_____no_output_____![Comparision of gender gap among various college degrees](gender_degrees.png)_____no_output_____#### Gender Gap: Present scenario
In total of 17 degrees, as % of participants
- overwhelmingly, women lead in 9 degrees namely, `Psychology`, `Foreign Languages`, `Health Professions`, `Biology`, `English`, `Public Administration`, `Communications and Journalism`, `Education` and `Arts and Performance`
- Men in 3 fields, `Computer science`, `Engineering` and `Physical Sciences` which are incidentally also the fields in which Men have dominated historically
- 5 fields in which gap is very small or non existant in `Maths and statistics`, `Agriculture`, `Social Sciences and history`, `Business` and `Architecture`

#### Gender Gap: Historically
Following are some of the observations from the visualization of gender gap in college degrees
- Historically Women dominated degrees: There are some fields in which % women graduates were very high i.e fields like `Health prefessions`, `education`, `foreign languages`, `english`
- Historically Men dominated degrees: `Computer science`, `Engineering` and `Physical Sciences`
- `Psychology` used to be a Men dominated field in the early 1970s but Women quickly captured and startted dominating ever since 70s.
- `Agriculture` and `Business`: Practically there were no Women in 1970s, Women have closed the gap and the numbers are almost equal now_____no_output_____


# Notebook from dustywhite7/pythonMikkeli
Path: Notebooks/5 - Regular Expression in Python.ipynb

# Regular Expression: Analyzing Text in Python_____no_output_____## What is Regular Expression?

Working with text is HARD. Do we consider "dog" to be different from "dogs"? How do we explain to a computer if we want those words to be considered the same word? In written language, most rules are not really rules, but patterns within text. Regular expression is a paradigm for describing patterns in text in order to accomodate the flexibility of written language and our attempts to find information within larger bodies of text.

For example, I might want to find a series of keywords within a body of text in order to determine whether or not the topic of the text has been correctly identified (or even in order to determine what the topic of the text actually is!). Regular expression has the ability to help express patterns in text that we may want to examine, and the flexibility to determine exactly how we want to capture it. Some common uses of regular expression:

- Validating emails, phone numbers, addresses, and other text blocks
- Finding subsets of a document
- Determining the presence or count of particular words, phrases or patterns
- Web Scraping
- Grammar suggestions

There are, of course many more ways to use regular expression. Let's start looking at how we can use it in our personal data analysis._____no_output_____## Basics of Regular Expression

Remember, regular expression is way to describe patterns. Below, we will explore how to create various kinds of patterns. To get started, we need to import our regular expression library, called `re`. We can import `re` just like any other library by including the following line of code:

```python
import re
```

The first function that we will utilize is `re.search`. This function will allow us to search through a string of text in order to try and find patterns of interest. Let's begin by importing some text. We will utilize two chapters (44 and 45) from the Jane Austen novel "Pride and Prejudice" (one of my favorites). Below is the code to import these chapters as a (very long) string:_____no_output_____

<code>
import requests
url = "https://raw.githubusercontent.com/dustywhite7/pythonMikkeli/master/exampleData/prideAndPrejudiceChapters.txt"
document = requests.get(url).text

print(document)Chapter 44

Elizabeth had settled it that Mr. Darcy would bring his sister to visit
her the very day after her reaching Pemberley; and was consequently
resolved not to be out of sight of the inn the whole of that morning.
But her conclusion was false; for on the very morning after their
arrival at Lambton, these visitors came. They had been walking about the
place with some of their new friends, and were just returning to the inn
to dress themselves for dining with the same family, when the sound of a
carriage drew them to a window, and they saw a gentleman and a lady in
a curricle driving up the street. Elizabeth immediately recognizing
the livery, guessed what it meant, and imparted no small degree of her
surprise to her relations by acquainting them with the honour which she
expected. Her uncle and aunt were all amazement; and the embarrassment
of her manner as she spoke, joined to the circumstance itself, and many
of the circumstances of the preceding day, opened to them a new idea on
the business. Nothing had ever suggested it before, but they felt that
there was no other way of accounting for such attentions from such a
quarter than by supposing a partiality for their niece. While these
newly-born notions were passing in their heads, the perturbation of
Elizabeth's feelings was at every moment increasing. She was quite
amazed at her own discomposure; but amongst other causes of disquiet,
she dreaded lest the partiality of the brother should have said too much
in her favour; and, more than commonly anxious to please, she naturally
suspected that every power of pleasing would fail her.

She retreated from the window, fearful of being seen; and as she walked
up and down the room, endeavouring to compose herself, saw such looks of
inquiring surprise in her uncle and aunt as made everything worse.

Miss Darcy and her brother appeared, and this formidable introduction
took place. With astonishment did Elizabeth see that her new
acquaintance was at least as much embarrassed as herself. Since her
being at Lambton, she had heard that Miss Darcy was exceedingly proud;
but the observation of a very few minutes convinced her that she was
only exceedingly shy. She found it difficult to obtain even a word from
her beyond a monosyllable.

Miss Darcy was tall, and on a larger scale than Elizabeth; and, though
little more than sixteen, her figure was formed, and her appearance
womanly and graceful. She was less handsome than her brother; but there
was sense and good humour in her face, and her manners were perfectly
unassuming and gentle. Elizabeth, who had expected to find in her as
acute and unembarrassed an observer as ever Mr. Darcy had been, was much
relieved by discerning such different feelings.

They had not long been together before Mr. Darcy told her that Bingley
was also coming to wait on her; and she had barely time to express her
satisfaction, and prepare for such a visitor, when Bingley's quick
step was heard on the stairs, and in a moment he entered the room. All
Elizabeth's anger against him had been long done away; but had she still
felt any, it could hardly have stood its ground against the unaffected
cordiality with which he expressed himself on seeing her again. He
inquired in a friendly, though general way, after her family, and looked
and spoke with the same good-humoured ease that he had ever done.

To Mr. and Mrs. Gardiner he was scarcely a less interesting personage
than to herself. They had long wished to see him. The whole party before
them, indeed, excited a lively attention. The suspicions which had just
arisen of Mr. Darcy and their niece directed their observation towards
each with an earnest though guarded inquiry; and they soon drew from
those inquiries the full conviction that one of them at least knew
what it was to love. Of the lady's sensations they remained a little
in doubt; but that the gentleman was overflowing with admiration was
evident enough.

Elizabeth, on her side, had much to do. She wanted to ascertain the
feelings of each of her visitors; she wanted to compose her own, and
to make herself agreeable to all; and in the latter object, where she
feared most to fail, she was most sure of success, for those to whom she
endeavoured to give pleasure were prepossessed in her favour. Bingley
was ready, Georgiana was eager, and Darcy determined, to be pleased.

In seeing Bingley, her thoughts naturally flew to her sister; and, oh!
how ardently did she long to know whether any of his were directed in
a like manner. Sometimes she could fancy that he talked less than on
former occasions, and once or twice pleased herself with the notion
that, as he looked at her, he was trying to trace a resemblance. But,
though this might be imaginary, she could not be deceived as to his
behaviour to Miss Darcy, who had been set up as a rival to Jane. No look
appeared on either side that spoke particular regard. Nothing occurred
between them that could justify the hopes of his sister. On this point
she was soon satisfied; and two or three little circumstances occurred
ere they parted, which, in her anxious interpretation, denoted a
recollection of Jane not untinctured by tenderness, and a wish of saying
more that might lead to the mention of her, had he dared. He observed
to her, at a moment when the others were talking together, and in a tone
which had something of real regret, that it “was a very long time since
he had had the pleasure of seeing her;” and, before she could reply,
he added, “It is above eight months. We have not met since the 26th of
November, when we were all dancing together at Netherfield.”

Elizabeth was pleased to find his memory so exact; and he afterwards
took occasion to ask her, when unattended to by any of the rest, whether
_all_ her sisters were at Longbourn. There was not much in the question,
nor in the preceding remark; but there was a look and a manner which
gave them meaning.

It was not often that she could turn her eyes on Mr. Darcy himself;
but, whenever she did catch a glimpse, she saw an expression of general
complaisance, and in all that he said she heard an accent so removed
from _hauteur_ or disdain of his companions, as convinced her that
the improvement of manners which she had yesterday witnessed however
temporary its existence might prove, had at least outlived one day. When
she saw him thus seeking the acquaintance and courting the good opinion
of people with whom any intercourse a few months ago would have been a
disgrace--when she saw him thus civil, not only to herself, but to the
very relations whom he had openly disdained, and recollected their last
lively scene in Hunsford Parsonage--the difference, the change was
so great, and struck so forcibly on her mind, that she could hardly
restrain her astonishment from being visible. Never, even in the company
of his dear friends at Netherfield, or his dignified relations
at Rosings, had she seen him so desirous to please, so free from
self-consequence or unbending reserve, as now, when no importance
could result from the success of his endeavours, and when even the
acquaintance of those to whom his attentions were addressed would draw
down the ridicule and censure of the ladies both of Netherfield and
Rosings.

Their visitors stayed with them above half-an-hour; and when they arose
to depart, Mr. Darcy called on his sister to join him in expressing
their wish of seeing Mr. and Mrs. Gardiner, and Miss Bennet, to dinner
at Pemberley, before they left the country. Miss Darcy, though with a
diffidence which marked her little in the habit of giving invitations,
readily obeyed. Mrs. Gardiner looked at her niece, desirous of knowing
how _she_, whom the invitation most concerned, felt disposed as to its
acceptance, but Elizabeth had turned away her head. Presuming however,
that this studied avoidance spoke rather a momentary embarrassment than
any dislike of the proposal, and seeing in her husband, who was fond of
society, a perfect willingness to accept it, she ventured to engage for
her attendance, and the day after the next was fixed on.

Bingley expressed great pleasure in the certainty of seeing Elizabeth
again, having still a great deal to say to her, and many inquiries to
make after all their Hertfordshire friends. Elizabeth, construing all
this into a wish of hearing her speak of her sister, was pleased, and on
this account, as well as some others, found herself, when their
visitors left them, capable of considering the last half-hour with some
satisfaction, though while it was passing, the enjoyment of it had been
little. Eager to be alone, and fearful of inquiries or hints from her
uncle and aunt, she stayed with them only long enough to hear their
favourable opinion of Bingley, and then hurried away to dress.

But she had no reason to fear Mr. and Mrs. Gardiner's curiosity; it was
not their wish to force her communication. It was evident that she was
much better acquainted with Mr. Darcy than they had before any idea of;
it was evident that he was very much in love with her. They saw much to
interest, but nothing to justify inquiry.

Of Mr. Darcy it was now a matter of anxiety to think well; and, as far
as their acquaintance reached, there was no fault to find. They could
not be untouched by his politeness; and had they drawn his character
from their own feelings and his servant's report, without any reference
to any other account, the circle in Hertfordshire to which he was known
would not have recognized it for Mr. Darcy. There was now an interest,
however, in believing the housekeeper; and they soon became sensible
that the authority of a servant who had known him since he was four
years old, and whose own manners indicated respectability, was not to be
hastily rejected. Neither had anything occurred in the intelligence of
their Lambton friends that could materially lessen its weight. They had
nothing to accuse him of but pride; pride he probably had, and if not,
it would certainly be imputed by the inhabitants of a small market-town
where the family did not visit. It was acknowledged, however, that he
was a liberal man, and did much good among the poor.

With respect to Wickham, the travellers soon found that he was not held
there in much estimation; for though the chief of his concerns with the
son of his patron were imperfectly understood, it was yet a well-known
fact that, on his quitting Derbyshire, he had left many debts behind
him, which Mr. Darcy afterwards discharged.

As for Elizabeth, her thoughts were at Pemberley this evening more than
the last; and the evening, though as it passed it seemed long, was not
long enough to determine her feelings towards _one_ in that mansion;
and she lay awake two whole hours endeavouring to make them out. She
certainly did not hate him. No; hatred had vanished long ago, and she
had almost as long been ashamed of ever feeling a dislike against him,
that could be so called. The respect created by the conviction of his
valuable qualities, though at first unwillingly admitted, had for some
time ceased to be repugnant to her feeling; and it was now heightened
into somewhat of a friendlier nature, by the testimony so highly in
his favour, and bringing forward his disposition in so amiable a light,
which yesterday had produced. But above all, above respect and esteem,
there was a motive within her of goodwill which could not be overlooked.
It was gratitude; gratitude, not merely for having once loved her,
but for loving her still well enough to forgive all the petulance and
acrimony of her manner in rejecting him, and all the unjust accusations
accompanying her rejection. He who, she had been persuaded, would avoid
her as his greatest enemy, seemed, on this accidental meeting, most
eager to preserve the acquaintance, and without any indelicate display
of regard, or any peculiarity of manner, where their two selves only
were concerned, was soliciting the good opinion of her friends, and bent
on making her known to his sister. Such a change in a man of so much
pride exciting not only astonishment but gratitude--for to love, ardent
love, it must be attributed; and as such its impression on her was of a
sort to be encouraged, as by no means unpleasing, though it could not be
exactly defined. She respected, she esteemed, she was grateful to him,
she felt a real interest in his welfare; and she only wanted to know how
far she wished that welfare to depend upon herself, and how far it would
be for the happiness of both that she should employ the power, which her
fancy told her she still possessed, of bringing on her the renewal of
his addresses.

It had been settled in the evening between the aunt and the niece, that
such a striking civility as Miss Darcy's in coming to see them on the
very day of her arrival at Pemberley, for she had reached it only to a
late breakfast, ought to be imitated, though it could not be equalled,
by some exertion of politeness on their side; and, consequently, that
it would be highly expedient to wait on her at Pemberley the following
morning. They were, therefore, to go. Elizabeth was pleased; though when
she asked herself the reason, she had very little to say in reply.

Mr. Gardiner left them soon after breakfast. The fishing scheme had been
renewed the day before, and a positive engagement made of his meeting
some of the gentlemen at Pemberley before noon.

Chapter 45

Convinced as Elizabeth now was that Miss Bingley's dislike of her had
originated in jealousy, she could not help feeling how unwelcome her
appearance at Pemberley must be to her, and was curious to know with how
much civility on that lady's side the acquaintance would now be renewed.

On reaching the house, they were shown through the hall into the saloon,
whose northern aspect rendered it delightful for summer. Its windows
opening to the ground, admitted a most refreshing view of the high woody
hills behind the house, and of the beautiful oaks and Spanish chestnuts
which were scattered over the intermediate lawn.

In this house they were received by Miss Darcy, who was sitting there
with Mrs. Hurst and Miss Bingley, and the lady with whom she lived in
London. Georgiana's reception of them was very civil, but attended with
all the embarrassment which, though proceeding from shyness and the fear
of doing wrong, would easily give to those who felt themselves inferior
the belief of her being proud and reserved. Mrs. Gardiner and her niece,
however, did her justice, and pitied her.

By Mrs. Hurst and Miss Bingley they were noticed only by a curtsey; and,
on their being seated, a pause, awkward as such pauses must always be,
succeeded for a few moments. It was first broken by Mrs. Annesley, a
genteel, agreeable-looking woman, whose endeavour to introduce some kind
of discourse proved her to be more truly well-bred than either of the
others; and between her and Mrs. Gardiner, with occasional help from
Elizabeth, the conversation was carried on. Miss Darcy looked as if she
wished for courage enough to join in it; and sometimes did venture a
short sentence when there was least danger of its being heard.

Elizabeth soon saw that she was herself closely watched by Miss Bingley,
and that she could not speak a word, especially to Miss Darcy, without
calling her attention. This observation would not have prevented her
from trying to talk to the latter, had they not been seated at an
inconvenient distance; but she was not sorry to be spared the necessity
of saying much. Her own thoughts were employing her. She expected every
moment that some of the gentlemen would enter the room. She wished, she
feared that the master of the house might be amongst them; and whether
she wished or feared it most, she could scarcely determine. After
sitting in this manner a quarter of an hour without hearing Miss
Bingley's voice, Elizabeth was roused by receiving from her a cold
inquiry after the health of her family. She answered with equal
indifference and brevity, and the other said no more.

The next variation which their visit afforded was produced by the
entrance of servants with cold meat, cake, and a variety of all the
finest fruits in season; but this did not take place till after many
a significant look and smile from Mrs. Annesley to Miss Darcy had been
given, to remind her of her post. There was now employment for the whole
party--for though they could not all talk, they could all eat; and the
beautiful pyramids of grapes, nectarines, and peaches soon collected
them round the table.

While thus engaged, Elizabeth had a fair opportunity of deciding whether
she most feared or wished for the appearance of Mr. Darcy, by the
feelings which prevailed on his entering the room; and then, though but
a moment before she had believed her wishes to predominate, she began to
regret that he came.

He had been some time with Mr. Gardiner, who, with two or three other
gentlemen from the house, was engaged by the river, and had left him
only on learning that the ladies of the family intended a visit to
Georgiana that morning. No sooner did he appear than Elizabeth wisely
resolved to be perfectly easy and unembarrassed; a resolution the more
necessary to be made, but perhaps not the more easily kept, because she
saw that the suspicions of the whole party were awakened against them,
and that there was scarcely an eye which did not watch his behaviour
when he first came into the room. In no countenance was attentive
curiosity so strongly marked as in Miss Bingley's, in spite of the
smiles which overspread her face whenever she spoke to one of its
objects; for jealousy had not yet made her desperate, and her attentions
to Mr. Darcy were by no means over. Miss Darcy, on her brother's
entrance, exerted herself much more to talk, and Elizabeth saw that he
was anxious for his sister and herself to get acquainted, and forwarded
as much as possible, every attempt at conversation on either side. Miss
Bingley saw all this likewise; and, in the imprudence of anger, took the
first opportunity of saying, with sneering civility:

“Pray, Miss Eliza, are not the ----shire Militia removed from Meryton?
They must be a great loss to _your_ family.”

In Darcy's presence she dared not mention Wickham's name; but Elizabeth
instantly comprehended that he was uppermost in her thoughts; and the
various recollections connected with him gave her a moment's distress;
but exerting herself vigorously to repel the ill-natured attack, she
presently answered the question in a tolerably detached tone. While
she spoke, an involuntary glance showed her Darcy, with a heightened
complexion, earnestly looking at her, and his sister overcome with
confusion, and unable to lift up her eyes. Had Miss Bingley known what
pain she was then giving her beloved friend, she undoubtedly would
have refrained from the hint; but she had merely intended to discompose
Elizabeth by bringing forward the idea of a man to whom she believed
her partial, to make her betray a sensibility which might injure her in
Darcy's opinion, and, perhaps, to remind the latter of all the follies
and absurdities by which some part of her family were connected
with that corps. Not a syllable had ever reached her of Miss Darcy's
meditated elopement. To no creature had it been revealed, where secrecy
was possible, except to Elizabeth; and from all Bingley's connections
her brother was particularly anxious to conceal it, from the very
wish which Elizabeth had long ago attributed to him, of their becoming
hereafter her own. He had certainly formed such a plan, and without
meaning that it should affect his endeavour to separate him from Miss
Bennet, it is probable that it might add something to his lively concern
for the welfare of his friend.

Elizabeth's collected behaviour, however, soon quieted his emotion; and
as Miss Bingley, vexed and disappointed, dared not approach nearer to
Wickham, Georgiana also recovered in time, though not enough to be able
to speak any more. Her brother, whose eye she feared to meet, scarcely
recollected her interest in the affair, and the very circumstance which
had been designed to turn his thoughts from Elizabeth seemed to have
fixed them on her more and more cheerfully.

Their visit did not continue long after the question and answer above
mentioned; and while Mr. Darcy was attending them to their carriage Miss
Bingley was venting her feelings in criticisms on Elizabeth's person,
behaviour, and dress. But Georgiana would not join her. Her brother's
recommendation was enough to ensure her favour; his judgement could not
err. And he had spoken in such terms of Elizabeth as to leave Georgiana
without the power of finding her otherwise than lovely and amiable. When
Darcy returned to the saloon, Miss Bingley could not help repeating to
him some part of what she had been saying to his sister.

“How very ill Miss Eliza Bennet looks this morning, Mr. Darcy,” she
cried; “I never in my life saw anyone so much altered as she is since
the winter. She is grown so brown and coarse! Louisa and I were agreeing
that we should not have known her again.”

However little Mr. Darcy might have liked such an address, he contented
himself with coolly replying that he perceived no other alteration than
her being rather tanned, no miraculous consequence of travelling in the
summer.

“For my own part,” she rejoined, “I must confess that I never could
see any beauty in her. Her face is too thin; her complexion has no
brilliancy; and her features are not at all handsome. Her nose
wants character--there is nothing marked in its lines. Her teeth are
tolerable, but not out of the common way; and as for her eyes,
which have sometimes been called so fine, I could never see anything
extraordinary in them. They have a sharp, shrewish look, which I do
not like at all; and in her air altogether there is a self-sufficiency
without fashion, which is intolerable.”

Persuaded as Miss Bingley was that Darcy admired Elizabeth, this was not
the best method of recommending herself; but angry people are not always
wise; and in seeing him at last look somewhat nettled, she had all the
success she expected. He was resolutely silent, however, and, from a
determination of making him speak, she continued:

“I remember, when we first knew her in Hertfordshire, how amazed we all
were to find that she was a reputed beauty; and I particularly recollect
your saying one night, after they had been dining at Netherfield, '_She_
a beauty!--I should as soon call her mother a wit.' But afterwards she
seemed to improve on you, and I believe you thought her rather pretty at
one time.”

“Yes,” replied Darcy, who could contain himself no longer, “but _that_
was only when I first saw her, for it is many months since I have
considered her as one of the handsomest women of my acquaintance.”

He then went away, and Miss Bingley was left to all the satisfaction of
having forced him to say what gave no one any pain but herself.

Mrs. Gardiner and Elizabeth talked of all that had occurred during their
visit, as they returned, except what had particularly interested them
both. The look and behaviour of everybody they had seen were discussed,
except of the person who had mostly engaged their attention. They talked
of his sister, his friends, his house, his fruit--of everything but
himself; yet Elizabeth was longing to know what Mrs. Gardiner thought of
him, and Mrs. Gardiner would have been highly gratified by her niece's
beginning the subject.

</code>

### Finding text

Now that we have our text imported, we can begin to explore it. We will start by looking for the word "Elizabeth" (she is the main character of the novel, and is also sometimes called "Lizzy" or "Eliza"). In order to look for the word "Elizabeth", we will use the `re.search` function mentioned earlier. This function takes two arguments: a regular expression, and a string in which to search for matches to that pattern. Regular expression patterns will be stored as strings, as well, but we will use **raw strings**. To create a raw string, we prepend the string with a single letter `r`, so that our string looks something like this: `r''`.

By using raw strings, we allow ourselves to use escape characters (character sequences beginning with `\` such as `\n` or `\r`) within our string to describe possible patterns that we would like to match.

Once we have created our regular expression pattern, we can call the `re.search` function. It will look for the first match to your pattern. Here is what happens when we look for a match to "Elizabeth":_____no_output_____

<code>
import re

re.search(r'Elizabeth', document)_____no_output_____
</code>

This is a successful search! The document does mention Elizabeth, and we see that a match to the expression `r'Elizabeth'` happened. The result of the function is an `re.Match` object, which contains information about where the match was found, as well as the text that matched our stated pattern. When we include a string like `r'Elizabeth'` as our pattern, we are telling regular expression to find a **capital** "E", followed by lower case "l", followed by lower case "i", followed by lower case "z", followed by lower case "a", followed by lower case "b", followed by lower case "e", followed by lower case "t", followed by lower case "h". No other text will match this pattern.

We will often want to be more expressive than this. For example, if we read the document, we can see that some characters mention Elizabeth as Eliza. We can of course search for Eliza separately, or we can try to combine Eliza into our expression pattern, so that we can detect all matches of either Elizabeth or Eliza. To see the difference, we will use the `re.finditer` function to look for all matches to our pattern:_____no_output_____

<code>
# Number of results for only Elizabeth
print("Number of results for only Elizabeth")
print(len([i for i in re.finditer(r'Elizabeth', document)]))

# Number of results for Elizabeth OR Eliza
print("Number of results for only Elizabeth OR Eliza")
print(len([i for i in re.finditer(r'Eliza(beth)?', document)]))Number of results for only Elizabeth
32
Number of results for only Elizabeth OR Eliza
34

</code>

What did we just do? A couple things. First, we used the `re.finditer` function, which generates an **iterable**. In order to get back all of our results, we need to create a list by iterating over the results. We used a list comprehension to do that. (Remember that list comprehensions are statements like `[i for i in some_iterable]`). At that point, we had a list of results. We then checked the length of the list for each of two different regular expressions. One for just Elizabeth, and one for Eliza/Elizabeth.

Now let's break down the difference in the regular expressions. There are two differences between our first and second expressions. First, we used `()` to encapsulate a portion of the word Elizabeth that may not be present in some cases. Parens (`()`) create **groups** in regular expression. Sometimes we will want to capture groups from within an expression, and other times we will only want to indicate that a group may or may not be present. In this case, we indicate that the sequence `beth` is not strictly required for a match by using a `?` character after the group. This doesn't mean that our text needs to include a `?`. The `?` symbol says "I want either 1 or 0 matches to the "beth" sequence.

In other words, "beth" may be present (have one match) or not present (have zero matches) while still allowing us to have a successful match to our regular expression pattern. There are several similar expressions:

| Symbol | Meaning |
| --- | :---: |
| `?` | 1 or 0 repetitions|
| `+` | 1 or more repetitions |
| `*` | 0 or more repetitions |
|`{n}` | exactly `n` repetitions|
|`{m,n}` | between `m` and `n` repetitions|

We can use these expressions with groups, or with single characters. What if we wanted to find the end of sentences in our text? Let's try and express that:_____no_output_____

<code>
[i for i in re.finditer(r'[.?!][ ]*', document)]_____no_output_____
</code>

This is a good start! We know that sentences can end with "?", "!", or ".", so we create a set of characters that can be matched: `[.?!]`. This expression says that a character can be a period, question mark, or exclamation mark and still match our pattern. We then use a character set including only the space character followed by a star (`[ ]*`) to indicate that there should be 0 or more spaces following each period.

We can use character sets where spelling is uncertain, or where there is a possibility of letters being upper or lower case. Let's look for the word "on", both with and without a character set to accomodate case ambiguity:_____no_output_____

<code>
# Number of results for "on"
print("Number of results for 'on'")
print(len([i for i in re.finditer(r'on', document)]))

# Number of results for "on" or "On"
print("Number of results for 'on' or 'On'")
print(len([i for i in re.finditer(r'[Oo]n', document)]))Number of results for 'on'
219
Number of results for 'on' or 'On'
221

</code>

If we weren't careful, we would miss some matches to the word we are looking for!_____no_output_____### Finding numbers

We won't always be looking for words. Sometimes, we want to find or validate numbers. In the United States, phone numbers are commonly expressed with a three-digit area code, a hyphen (`-`), and a seven-digit phone number with the first three numbers separated from the final four by another hyphen. Let's try to detect this pattern in text that is provided by the user. Before we get started, let's talk about how we can identify numbers.

When we look for numbers, we typically are not looking for a single specific number (like 42). We are instead looking for **some** number. It might be 2, it might be 142, or it might be 1000. Let's look for a number in the following sentence:_____no_output_____

<code>
sentence = "My favorite number is 7"

re.search(r'7', sentence)_____no_output_____
</code>

Okay, we can find an exact number. Can you think of a way that we have discussed above to find **any** single-digit number?

**Solve it (2 points)**:

The user will enter their favorite single-digit number. You need to make sure that there is a number in the sentence. Store the results of your `re.search` function in the variable `hasNumber` in the cell below:_____no_output_____

<code>
import re

# Ask user for favorite number (DON'T DELETE)
myNumber = input("Please enter your favorite single-digit number: ")

# Create the sentence (DON'T DELETE)
sentence = "My favorite number is {}".format(int(myNumber))

# Write a regular expression to determine if there is a number in the sentence
hasNumber = re.search(r'\b\d\b', sentence)

print(bool(hasNumber))Please enter your favorite single-digit number: 10
False

</code>

When we want to find a number, we could use character sets: `[0123456789]`. This character set specifies that we will accept any number character as a match to our pattern. If we want to specify that it be a number that has more than one character, we could provide a pattern such as `[0123456789]+`. It will get really old, really fast to keep typing `[0123456789]`. Instead, we can use the shorthand `\d` representation for a **digit** or number. Now, if we want to find a number character, we can just write `\d` in place of `[0123456789]`. For one or more number characters, we write `\d+`.

Now, let's get back to trying to break down phone number validation. Using what we know about digit representation, combined with our knowledge of ways to express repeated patterns from before, we can express a phone number! If we rewrite our goal as words, we would end up with the following:

- Three numbers
- A hyphen
- Three numbers
- A hyphen
- Four numbers

Let's write this as a regular expression:_____no_output_____

<code>
# Ask user for phone number
myNumber = input("Please enter your phone number: ")

# Check if the phone number is valid
valid = re.search(r'\d{3}-\d{3}-\d{4}', myNumber)

# Print a boolean value indicating whether or not the string contains a valid phone number
print(bool(valid))Please enter your phone number: 123-456-7890 turtles
True

</code>

Awesome! Try different valid and invalid phone numbers in the cell above to see if they work with our code._____no_output_____### More general patterns

What other patterns might come in handy? Here is a short table of helpers for making your expressions easier to write:

| Pattern | Meaning | Pattern | Meaning |
| --- | --- | --- | --- |
| `\s` | any whitespace character | `\S` | any non-whitespace character |
| `\d` | any digit character | `\D` | any non-digit character |
| `\w` | any word character (a number, a letter, or a `_`) | `\W` | non-word characters|
| `\b` | word boundaries (the start or end of a sequence of word characters) ||
| `^` | start of a string | `$` | end of a string

Going back to our phone number exercise, what happens if you provide the string `123-456-7890 turtles` as a phone number? Is it considered valid? **It is!** We can use our new patterns to protect against this:

_____no_output_____

<code>
# Add start and end of string characters
valid = re.search(r'^\d{3}-\d{3}-\d{4}$', "123-456-7890 turtles")

# Print an improved boolean value indicating whether or not the string contains a valid phone number
print(bool(valid))True

</code>

Now we won't be fooled so easily! Our number has to be formatted correctly, AND our string cannot contain any other characters. Play with the code above and check for yourself!

When we think about how to create an expression to validate text or numbers, we should be particularly careful to think of any **possible** ways that a value might be invalid. It is not sufficient to think only of ways in which text is **likely** to be invalid. If we are not careful, our validation will not truly validate, and any work that we do with the "validated" text may be broken by cases we have not accounted for._____no_output_____### Looking Behind

Speaking of not accounting for all the possibilities, we made this mistake earlier when we were looking for the ends of sentences earlier. We looked for all periods (`.`), question marks (`?`), and exclamation marks (`!`). Unfortunately, not all periods come at the end of sentences! In *Pride and Prejudice*, many characters are referred to as "Mr." or "Mrs.". Those periods are followed by spaces, and so would have counted as the end of a sentence.

#### Negative Lookbehind

In order to solve this problem, we need a new tool: the **negative lookbehind**. A negative lookbehind is a pattern that must be matched while simultaneously NOT being preceded by another pattern. In this case, we would want `[.?!]` wherever that character set is NOT preceded by `Mr` or `Mrs`. First, let's make sure that we can find `Mr` and `Mrs` with a single expression:_____no_output_____

<code>
[i for i in re.finditer(r'Mrs*', document)]_____no_output_____
</code>

So far, so good! Now that we can find all of the cases that we **don't** want to count as ends of sentences, let's use a negative lookbehind. The expression we want to avoid will be put in a group (between `()`), and we will put the character sequence `?<!` at the start of the group:_____no_output_____

<code>
# Count of sentences without negative lookbehind (incorrect)
print("Count of sentences without negative lookbehind (incorrect)")
print(len([i for i in re.finditer(r'[.!?]', document)]))

# Count of sentences with negative lookbehind (correct)
print("Count of sentences with negative lookbehind (correct)")
print(len([i for i in re.finditer(r'(?<!Mr)(?<!Mrs)[.!?]', document)]))
Count of sentences without negative lookbehind (incorrect)
161
Count of sentences with negative lookbehind (correct)
128

</code>

We have to use **two** negative lookbehinds to solve this problem, because we cannot allow for `Mr` or `Mrs` to precede our punctuation. You might have tried `(?<!Mrs*)` (as I did at first). Unfortunately, we cannot use this structure because we are not permitted to use a negative lookbehind that is of uncertain length (as is any pattern with `*`, `+`, `?`, or `{}`). Instead, we simply check for the first string, then check for the second string._____no_output_____#### Positive Lookbehind

We can also use a positive lookbehind to **require** that our pattern be preceded by another pattern. For example, we may wish to determine which chapters are covered in a text document (we know it is Chapters 44 and 45, but we can use this as an exercise anyway). A positive lookbehind is indicated within parens (`()`) just like the negative lookbehind, but uses the character sequence `?<=`. Let's try to find numbers that are preceded by the pattern `Chapter `:_____no_output_____

<code>
[i for i in re.finditer(r'(?<=Chapter )\d+', document)]_____no_output_____
</code>

Excellent! Just as we know *should* be the case, we can see that we found the beginnings of both chapters 44 and 45._____no_output_____## Options

When we are creating patterns, we sometimes will encounter multiple acceptable patterns that we wish to account for. With phone numbers, area codes in the United States are often written in parens rather than with hypens. Thus, we might need to account for two patterns:
- `xxx-xxx-xxxx`
- `(xxx) xxx-xxxx`

Let's try to write expressions for the second option (since we can already accomodate the first):_____no_output_____

<code>
# Ask user for phone number
myNumber = input("Please enter your phone number: ")

# Check if the phone number is valid
valid = re.search(r'^[(]\d{3}[)][ ]\d{3}-\d{4}$', myNumber)

# Print a boolean value indicating whether or not the string contains a valid phone number
print(bool(valid))Please enter your phone number: (425) 443-1880
True

</code>

Our new rule is written `[(]\d{3}[)][ ]`. There are a lot of square brackets! In order to look for the `()` characters, we have to wrap them in a character set, so that our program knows that they are not there to indicate a group, but are instead literal characters. In between the parens, we have our `\d{3}` three-digit number set. Finally, at the end, we have a single space character `[ ]` that must separate the area code `(xxx)` from the rest of the phone number `xxx-xxxx`.

You can try various inputs to see if the code continues to validate phone numbers correctly. It should work on our new format with parens, but not on our original phone number format. Luckily, now that we have a validity check for both phone number formats, we can incorportate them into a single expression to accept either valid format:_____no_output_____

<code>
# Ask user for phone number
myNumber = input("Please enter your phone number: ")

# Check if the phone number is valid
valid = re.search(r'^([(]\d{3}[)][ ]|\d{3}-)\d{3}-\d{4}$', myNumber)

# Print a boolean value indicating whether or not the string contains a valid phone number
print(bool(valid))Please enter your phone number: (123) 456-7890
True

</code>

In this case, we have taken the original area code rule (`\d{3}-`) as well as the new area code rule (`[(]\d{3}[)][ ]`), and we have put them inside of a group using parens `()`. They are separated inside this group by the `|` character. Just like it does in our boolean conditions, the `|` character represents the logical statement "or". Essentially, we are saying that our pattern must match `\d{3}-` OR `[(]\d{3}[)][ ]`, followed by `\d{3}-\d{4}`.

For good measure, we have wrapped the whole expression in string start and end characters (`^` and `$`) in order to disallow any other text in our valid phone numbers._____no_output_____## Extracting results

When we use the `re.search` or `re.finditer` functions, matches are returned to us through an `re.Match` object. We can extract the text from a match object by using the `.string` attribute. _____no_output_____

<code>
valid.string_____no_output_____
</code>

Also useful is the `.span` function of the `re.Match` object, which returns to us the start and end points (within the original string) of the match to our pattern:_____no_output_____

<code>
valid.span()_____no_output_____
</code>

Finally, we also have the ability to transform an `re.Match` object into a `bool`-type object, which is useful in various contexts such as text validation. To evaluate an `re.Match` object, we can simply convert the variable into which it is stored:_____no_output_____

<code>
bool(valid)_____no_output_____
</code>

If, on the other hand, we fail to find a match to our pattern, our `re.search` or `re.finditer` function will return a `None` object, and it's boolean value will be `False`:_____no_output_____

<code>
value = re.search(r'happy', "I am sad")

print(value)
print(bool(value))None
False

</code>

**Solve it (2 points)**:

Write a function called `dateCheck` that will accept a string as an argument to determine whether or not that string contains ONLY a valid date. The date should be of the format DD-MM-YYYY. In other words, the day of the month should come first, followed by a hyphen (`-`), followed by the month itself, followed by another hyphen (`-`), and finally the year. Days and months should always have two digits, and years should have four digits.

For the purpose of this exercise, pretend that all months have 31 days, and remember that there are twelve months in any year.

When the string is a correctly formatted date, your function should return `True`. Where the string is not a valid date, your function should return `False`.

The blue cell below should contain ALL NECESSARY CODE, including import statements, for your function to work properly._____no_output_____

<code>
import re

def dateCheck(date):
    check = re.search(r'^(3[01]|[012]\d)-(0\d|1[012])-\d{4}$', date)
    return bool(check)_____no_output_____
</code>

**Solve it (2 points)**:

Write a function called `emailCheck` that will accept a string as an argument, and determine whether or not that string is a valid email. Emails take the following form: a string that contains numbers, letters, and periods followed by the `@` symbol and a domain. We will accept domains that contain numbers and letters, ending in `.com`, `.edu`, or `.org`.

When the string is a correctly formatted email address, your function should return `True`. Where the string is not a valid email address, your function should return `False`.

The blue cell below should contain ALL NECESSARY CODE, including import statements, for your function to work properly._____no_output_____

<code>
import re

def emailCheck(email):
    check = re.search(r'^[a-zA-Z0-9.]+[@][a-zA-Z0-9]+([.]com|[.]edu|[.]org)$', email)
    return bool(check)_____no_output_____
</code>