# Notebook from adamreevesman/msds692
Path: notes/pdf.ipynb

# Extracting text from PDFs

[PDF](https://techterms.com/definition/pdf) stands for "Portable Document Format" and PDF files contain images, text, and page layout information.  PDF files are actually programs in a very simple programming language and, hence, can display just about anything. Much of what you see inside a PDF file is text, however, and we can grab that text without the layout information using [poppler](https://poppler.freedesktop.org/). (I used to use `pdfminer` but somehow no longer works on OS X.) Install it with:

```bash
brew install poppler
```

or

```bash
brew upgrade poppler
```

Then use `pdftotext` as a command from the commandline, which will extract out the text  and save in a text file. First download a sample PDF, such as [Tesla model S](https://www.tesla.com/sites/default/files/tesla-model-s.pdf), which we can easily do from the command line using `curl` (which you might have to install):_____no_output_____

! curl https://www.tesla.com/sites/default/files/tesla-model-s.pdf > /tmp/tsla.pdf  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 9438k    0 9438k    0     0  4375k      0 --:--:--  0:00:02 --:--:-- 4375k

That command downloads the file and because of the redirection operator, `>`, the output gets written to `tsla.pdf` up in `/tmp` directory.

Once we have the data, we can pass the filename to `pdftotext` to extract the text:_____no_output_____

! pdftotext /tmp/tsla.pdf # saves into /tmp/tsla.txtSyntax Warning: Invalid Font Weight
Syntax Warning: Invalid Font Weight
Syntax Warning: Invalid Font Weight
Syntax Warning: Invalid Font Weight
Syntax Warning: Invalid Font Weight

(Don't worry about those warnings.)_____no_output_____

! head -10 /tmp/tsla.txtModel S
Premium Electric Sedan

An evolution
in automobile
engineering
Tesla’s advanced electric powertrain
delivers exhilarating performance.
Unlike a gasoline internal combustion
engine with hundreds of moving

Once you have text output, you can perform whatever analysis you'd like without having to worry about the data coming in PDF form. For example, you might want to run some analysis on financial documents but they are all in PDF. First, convert to text and then perform your analysis._____no_output_____### Exercise

Read that text file with a Python script and split the document into a list of words. Print out the first 100 words. It should look like:

```
['Model', 'S', 'Premium', 'Electric', 'Sedan', 'An', ...]
```_____no_output_____

with open('/tmp/tsla.txt') as f:
    print(f.read().split()[:100])['Model', 'S', 'Premium', 'Electric', 'Sedan', 'An', 'evolution', 'in', 'automobile', 'engineering', 'Tesla’s', 'advanced', 'electric', 'powertrain', 'delivers', 'exhilarating', 'performance.', 'Unlike', 'a', 'gasoline', 'internal', 'combustion', 'engine', 'with', 'hundreds', 'of', 'moving', 'parts,', 'Tesla', 'electric', 'motors', 'have', 'only', 'one', 'moving', 'piece:', 'the', 'rotor.', 'As', 'a', 'result,', 'Model', 'S', 'acceleration', 'is', 'instantaneous,', 'silent', 'and', 'smooth.', 'Step', 'on', 'the', 'accelerator', 'and', 'in', 'as', 'little', 'as', '3.1', 'seconds', 'Model', 'S', 'is', 'travelling', '60', 'miles', 'per', 'hour,', 'without', 'hesitation,', 'and', 'without', 'a', 'drop', 'of', 'gasoline.', 'Model', 'S', 'is', 'an', 'evolution', 'in', 'automobile', 'engineering.', 'All-Wheel', 'Drive', 'Dual', 'Motor', 'Rear', 'Wheel', 'Drive', 'All-Wheel', 'Drive', 'Dual', 'Motor', 'Dual', 'Motor', 'Model', 'S', 'is']

## Text processing from the command line

It's often the case that we can do a huge amount of cleanup on unstructured text before using Python to process it more formally. We can delete unwanted characters, squeeze repeated characters, reformat, etc... In this section you will do a number of exercises that get you use to processing files from the command line. If you'd like to dig further, you can see [this link](http://www.tldp.org/LDP/abs/html/textproc.html).

The operating system launches all commands in a pipeline sequence as separate processes, which means they can run on multiple processors simultaneously. This gives us parallel processing without having to write complicated code. As data is completed by one stage, it passes it to the next stage of the pipeline, and continues to work on its input. The next stage consumes that input in parallel. Consequently, processing text from the command line can be extremely efficient, much more so than doing it in Python._____no_output_____### Exercise

Using the `tr` (translate) command from the terminal, strip all of the new lines from the text file you created above (`/tmp/tsla.txt`).  Look at the manual page with this command:

```python
$ man tr
```

You can pipe the output of `tr` to `head -c 150` to only print out the first 150 characters of the output._____no_output_____

! tr -s '\n' ' ' < /tmp/tsla.txt | head -c 200Model S Premium Electric Sedan An evolution in automobile engineering Tesla’s advanced electric powertrain delivers exhilarating performance. Unlike a gasoline internal combustion engine with hundr

### Exercise

Reformat the text using `tr` and `fold`. The `fold` command wraps lines at 80 characters; use its `-s` option to making break lines at spaces between words._____no_output_____

! tr -s '\n' ' ' < /tmp/tsla.txt | fold -s | head -10Model S Premium Electric Sedan An evolution in automobile engineering Tesla’s
advanced electric powertrain delivers exhilarating performance. Unlike a
gasoline internal combustion engine with hundreds of moving parts, Tesla
electric motors have only one moving piece: the rotor. As a result, Model S
acceleration is instantaneous, silent and smooth. Step on the accelerator and
in as little as 3.1 seconds Model S is travelling 60 miles per hour, without
hesitation, and without a drop of gasoline. Model S is an evolution in
automobile engineering. All-Wheel Drive Dual Motor Rear Wheel Drive All-Wheel
Drive Dual Motor Dual Motor Model S is a categorical improvement on
conventional all-wheel drive systems. With two motors, one in the front and one

### Exercise

It is sometimes useful to put a line number at the left edge of all lines. For example, you might want to create a unique ID number for each row of a CSV file. Pipe the output of the previous command to `nl` so that you get the line number on the left edge._____no_output_____

! tr -s '\n' ' ' < /tmp/tsla.txt | fold -s | nl | head -10     1	Model S Premium Electric Sedan An evolution in automobile engineering Tesla’s
     2	advanced electric powertrain delivers exhilarating performance. Unlike a
     3	gasoline internal combustion engine with hundreds of moving parts, Tesla
     4	electric motors have only one moving piece: the rotor. As a result, Model S
     5	acceleration is instantaneous, silent and smooth. Step on the accelerator and
     6	in as little as 3.1 seconds Model S is travelling 60 miles per hour, without
     7	hesitation, and without a drop of gasoline. Model S is an evolution in
     8	automobile engineering. All-Wheel Drive Dual Motor Rear Wheel Drive All-Wheel
     9	Drive Dual Motor Dual Motor Model S is a categorical improvement on
    10	conventional all-wheel drive systems. With two motors, one in the front and one

### Exercise

Convert the text to all lowercase using `tr`. Hint: `a-z` and `A-Z` are [regular expressions](http://www.rexegg.com/regex-quickstart.html) that describe English characters and uppercase English characters._____no_output_____

! tr 'A-Z' 'a-z' < /tmp/tsla.txt | head -c 150model s
premium electric sedan

an evolution
in automobile
engineering
tesla’s advanced electric powertrain
delivers exhilarating performance.
unli

### Exercise

Do the same thing but on the text that has the new lines removed._____no_output_____

! tr -s '\n' ' ' < /tmp/tsla.txt | tr 'A-Z' 'a-z' | head -c 150model s premium electric sedan an evolution in automobile engineering tesla’s advanced electric powertrain delivers exhilarating performance. unlik


# Notebook from r-e-x-a-g-o-n/scalable-data-science
Path: _in/2019/jp/02.ipynb

# [Introduction to Data Science: A Comp-Math-Stat Approach](https://lamastex.github.io/scalable-data-science/as/2019/)
## YOIYUI001, Summer 2019
©2019 Raazesh Sainudiin. [Attribution 4.0 International (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/)_____no_output_____# 02. Numbers, Strings, Booleans and Sets

_____no_output_____# Numbers and Arithmetic Operations

We will start by showing you some of the basic numeric capabilities of SageMath.

A worksheet cell is the area enclosed by a gray rectangle.
You may type any expression you want to evaluate into a worksheet cell.   We have already put some expressions into this worksheet.

When you are in a cell you can evaluate the expression in it by pressing <SHIFT><ENTER> or just by clicking the evaluate button below the cell.

To start with, we are going to be using SAGE like a hand-held calculator. Let's perform the basic arithmetic operations of addition, subtraction, multiplication, division, exponentiation, and remainder over the three standard number systems: Integers denoted by $\mathbb{Z}$, Rational Numbers denoted by $\mathbb{Q}$ and Real Numbers denoted by $\mathbb{R}$.  Let us recall the real number line and the basics of number systems next.

_____no_output_____

def showURL(url, ht=500):
    """Return an IFrame of the url to show in notebook with height ht """
    from IPython.display import IFrame
    return IFrame(url, width='95%', height=ht)
showURL('https://en.wikipedia.org/wiki/Number',400)_____no_output_____

The most basic numbers are called natural numbers and they are denoted by $\mathbb{N} :=\{0, 1,2,3,\ldots\}$.
See [https://en.wikipedia.org/wiki/Natural_number](https://en.wikipedia.org/wiki/Natural_number).

> The natural numbers are the basis from which many other number sets may be built by extension: the integers, by including (if not yet in) the neutral element 0 and an additive inverse (−n) for each nonzero natural number n; the rational numbers, by including a multiplicative inverse (1/n) for each nonzero integer n (and also the product of these inverses by integers); the real numbers by including with the rationals the limits of (converging) Cauchy sequences of rationals; the complex numbers, by including with the real numbers the unresolved square root of minus one (and also the sums and products thereof); and so on. These chains of extensions make the natural numbers canonically embedded (identified) in the other number systems._____no_output_____

showURL("https://en.wikipedia.org/wiki/Natural_number#Notation",300)_____no_output_____

Let us get our fingers dirty with some numerical operations in SageMath.

Note that anything after a '#' symbol is a comment - comments are ignored by SAGE but help programmers
to know what's going on.

### Example 1: Integer Arithmetic

Try evaluating the cell containing 1+2 below by placing the cursor in the cell and pressing  <SHIFT><ENTER>.  _____no_output_____

1+2 # one is being added to 2_____no_output_____

Now, modify the above expression and evaluate it again.  Try 3+4, for instance._____no_output_____

3-4 # subtracting 4 from 3_____no_output_____

The multiplication operator is `*`, the division operator is `/`._____no_output_____

2*6 # multiplying 2 by 6_____no_output_____15/5 # dividing 15 by 5_____no_output_____type(1)_____no_output_____

The exponentiation operator is `^`._____no_output_____

2^3 # exponentiating 2 by 3, i.e., raising 2 to the third power_____no_output_____

However, Python's exponentiation operator `**` also works._____no_output_____

2**3_____no_output_____

Being able to finding the remainder after a division is surprisingly useful in computer programming._____no_output_____

11%3 # remainder after 11 is divided by 3; i.e., 11=3*3+2_____no_output_____

Another way of referring to this is 11 modulus 3, which evaluates to 2.   Here `%` is the modulus operator.

### You try

Try typing in and evaluating some expressions of your own.  You can get new cells above or below an existing cell by clicking 'Insert' in the menu above and 'Insert Cell Above' or 'Insert Cell below'. You can also place the cursor at an existing cell and click `+` icon above to get a new cell below. _____no_output_____What happens if you put space between the characters in your expression, like:
`1   +   2`
instead of `1+2`?. _____no_output_____### Example 2: Operator Precedence for Evaluating Arithmetic Expressions

Sometimes we want to perform more than one arithmetic operation with some given integers.
Suppose, we want to

- "divide 12 by 4 then add the product of 2 and 3 and finally subtract 1."

Perhaps this can be achieved by evaluating the expression "12/4+2*3-1"?

But could that also be interpreted as

- "divide 12 by the sum of 4 and 2 and multiply the result by the difference of 3 and 1"?

In programming, there are rules for the order in which arithmetic operations are carried out.  This is called the order of precedence.

The basic arithmetic operations are: +, -, *, %, /, ^.

The order in which operations are evaluated are as follows:

- ^ Exponents are evaluated right to left
- *, %, / Then multiplication, remainder and division operations are evaluated left to right
- +, - Finally, addition and subtraction are evaluated left to right

When operators are at the same level in the list above, what matters is the evaluation order (right to left, or left to right).

Operator precedence can be forced using parenthesis._____no_output_____

showURL("https://en.wikipedia.org/wiki/Order_of_operations", 300)_____no_output_____(12/4) + (2*3) - 1 # divide 12 by 4 then add the product of 2 and 3 and finally subtract 1_____no_output_____12/4+2*3-1 # due to operator precedence this expression evaluates identically to the parenthesized expression above_____no_output_____

Operator precedence can be forced using nested parentheses. When our expression has nested parenthesis, i.e., one pair of parentheses inside another pair, the expression inside the inner-most pair of parentheses is evaluated first.

The following cell evaluates the mathematical expression:

$$\frac{12}{4+2} (3-1)$$_____no_output_____

(12/(4+2)) * (3-1)  # divide 12 by the sum of 4 and 2 and multiply the result by the difference of 3 and 1_____no_output_____

## You try

Try writing an expression which will subtract 3 from 5 and then raise the result to the power of 3._____no_output_____Find out for yourself what we mean by the precedence for exponentiation (^) being from right to left:

- What do you think the expression `3^3^2` would evaluate to?
  - Is it the same as  `(3^3)^2`, i.e., `27` squared, or
  - `3^(3^2)`, i.e., `3` raised to the power `9`?

Try typing in the different expressions to find out:_____no_output_____Find an expression which will add the squares of four numbers together and then divide that sum of squares by 4._____no_output_____Find what the precedence is for the modulus operator `%` that we discussed above:  try looking at the difference between the results for `10%2^2` and `10%2*2` (or `10^2+2`).  Can you see how SageMath is interpreting your expressions?

Note that when you have two operators at the same precedence level (like `%` and `*`), then what matters is the order - left to right or right to left.  You will see this when you evaluate `10%2*2`._____no_output_____Does putting spaces in your expression make any difference?_____no_output_____Using parenthesis or white spaces can improve readability a lot! So be generous with them to evaluate the following expression:

$$10^2 + 2^8 -4$$_____no_output_____

10^2+2^8-4_____no_output_____10^2 + 2^8 -4_____no_output_____(((10^2) + (2^8)) - 4)  # this may be overkill!_____no_output_____

The lesson to learn is that it is always good to use the parentheses:  you will make it clear to someone reading your code what you mean to happen as well as making sure that the computer actually does what you mean it to!

Try this 10 minutes-long videos to get some practice if you are really rusty with order of operations:

* [Khan Academy Order of operations - https://www.youtube.com/watch?v=ClYdw4d4OmA](https://www.youtube.com/watch?v=ClYdw4d4OmA)
_____no_output_____### Example 3: Rational Arithmetic

So far we have been dealing with integers. Integers are a type in SAGE.  Algebraically speaking, integers, rational numbers and real numbers form a *ring*.  This is something you will learn in detail in a maths course in Group Theory or Abstract Algebra, but let's take a quick peek at the definition of a ring._____no_output_____

showURL("https://en.wikipedia.org/wiki/Ring_(mathematics)#Definition_and_illustration",400)_____no_output_____type(1) # find the data type of 1_____no_output_____

The output above tells us that `1` is of type `sage.rings.integer.Integer`._____no_output_____

showURL("https://en.wikipedia.org/wiki/Integer",400)_____no_output_____

However, life with only integers denoted by $\mathbb{Z} := \{\ldots,-3,-2,-1,0,1,2,3,\ldots\}$ is a bit limited. What about values like $1/2$ or $\frac{1}{2}$?

This brings us to the rational numbers denoted by $\mathbb{Q}$._____no_output_____

showURL("https://en.wikipedia.org/wiki/Rational_number",400)_____no_output_____type(1/2) # data type of 1/2 is a sage.rings.rational.Rational_____no_output_____

Try evaluating the cell containing `1/2 + 2` below.  _____no_output_____

1/2 + 2 # add one half to 2 or four halves to obtain the rational number 5/2 or five halves_____no_output_____

SageMath seems to have done rational arithmetic for us when evaluating the above expression.

Next, modify the expression in the cell below and evaluate it again.  Try `1/3+2/4`, for instance._____no_output_____

1/2 + 1/3_____no_output_____

You can do arithmetic with rationals just as we did with integers._____no_output_____

3/4 - 1/4 # subtracting 3/4 from 1/4_____no_output_____1/2 * 1/2 # multiplying 1/2 by 1/2_____no_output_____(2/5) / (1/5) # dividing 2/5 by 1/5_____no_output_____(1/2)^3 # exponentiating 1/2 by 3, i.e., raising 1/2 to the third power_____no_output_____

### You try

Write an expression which evaluates to `1` using the rationals `1/3` and `1/12`, some integers, and some of the arithmetical operators - there are lots of different expressions you can choose, just try a few._____no_output_____What does SageMath do with something like `1/1/5`?  Can you see how this is being interpreted?  What should we do if we really want to evaluate `1` divided by `1/5`?_____no_output_____Try adding some rationals and some integers together - what type is the result?_____no_output_____## Example 4: Real Arithmetic (multi-precision floating-point arithmetic)

Recall that real numbers denoted by $\mathbb{R}$ include natural numbers ($\mathbb{N}$), integers ($\mathbb{Z}$), rational numbers ($\mathbb{Q}$) and various types of irrational numbers like:

- the square root of 2 or $\sqrt{2}$
- [Pi](https://en.wikipedia.org/wiki/Pi) or $\pi$ and
- Euler's number $e$ and
- [Euler–Mascheroni constant](https://en.wikipedia.org/wiki/Euler%E2%80%93Mascheroni_constant) $\gamma$.

Real numbers can be thought of as all the numbers in the real line between negative infinity and positive infinity.  Real numbers are represented in decimal format, for e.g. 234.4677878.

_____no_output_____

showURL("https://en.wikipedia.org/wiki/Real_number#Definition",400)_____no_output_____

We cannot do exact real arithmetic (but do it aproximately) in a computer with [http://www.mpfr.org/](http://www.mpfr.org/)'s multiprecision [floating-point numbers](http://en.wikipedia.org/wiki/Floating_point), and can combine them with integer and rational types in SageMath.

*Technical note:* Computers can be made to exactly compute in integer and rational arithmetic. But, because computers with finite memory (all computers today!) cannot represent the [uncountably infinitely many real numbers](http://en.wikipedia.org/wiki/Cantor%27s_diagonal_argument), they can only mimic or approximate arithmetic over real numbers using finitely many computer-representable floating-point numbers.

Let's take a peak at *von Neumann architecture* of any typical computer today.

The punch-line is that the *Memory unit* or *Random Access Memory (RAM)* inside the Central Processing Unit as well as *Input* and *Output Devices* are physical with finite memory. Therefore we cannot exactly represent all the uncontably infinitely many real numbers in $\mathbb{R}$ and need to resort to their approximation using *floating point* numbers._____no_output_____

showURL("https://en.wikipedia.org/wiki/Von_Neumann_architecture",400)_____no_output_____

See [SageMath Quick Start on Numerical Analysis](http://doc.sagemath.org/html/en/prep/Quickstarts/NumAnalysis.html) to understand SageMath's multiprecision real arithmetic.

For now, let's compare the results of evaluating the expressions below to the equivalent expressions using rational numbers above._____no_output_____

type(0.5) # data type of 0.5 is a sage.rings.real_mpfr.RealLiteral_____no_output_____RR # Real Field with the default 53 bits of precision_____no_output_____RR(0.5)  # RR(0.5) is the same as 0.5 in SageMath_____no_output_____type(RR(0.5)) # RR(0.5) is the same type as 0.5 from before_____no_output_____??RR_____no_output_____0.5 + 2 # one half as 0.5 is being added to 2 to obtain the real number 2.500..0 in SageMath_____no_output_____0.75 - 0.25 # subtracting 0.75 from 0.25 is the same as subtracting 0.75 from 1/4_____no_output_____0.5 * 0.5 # multiplying 0.5 by 0.5 is the same as 1/2 * 1/2_____no_output_____(2 / 5.0) / 0.2 # dividing 2/5. by 0.2 is the same as (2/5) / (1/5)_____no_output_____0.5^3.0 # exponentiating 0.5 by 3.0 is the same as (1/2)^3_____no_output_____

### You try

Find the type of `1/2`._____no_output_____Try a few different ways of getting the same result as typing  `((((1/5) / (1/10)) * (0.1 * 2/5) + 4/100))*5/(3/5)` - this exact expression has already been put in for you in the cell below you could try something just using floating point numbers.  Then see how important the parentheses are around rationals when you have an expression like this - try taking some of the parenthesis out and just play with complex expressions like these to get familiar._____no_output_____

((((1/5) / (1/10)) * (0.1 * 2/5) + 4/100))*5/(3/5)_____no_output_____((((1/5) / (1/10)) * (1/10 * 2/5) + 4/100))*5/(3/5)_____no_output_____

## Example 5: Variables and assignments of numbers and expressions

Loosely speaking one can think of a *variable* as a way of referring to a memory location used by a computer program. A variable is a symbolic name for this physical location. This memory location contains values, like numbers, text or more complicated types and crucially *what is contained in a variable can change* based on operations we do to it.

In SageMath, the symbol `=` is the *assignment operator*.  You can assign a numerical value to a *variable* in SageMath using the assignment operator.  This is a good way to store values you want to use or modify later.

(If you have programmed before using a a language like C or C++ or Java, you'll see that SageMath is a bit different because in SageMath you don't have to say what type of value is going to be assigned to the variable.)

Feel free to take a deeper dive into the computer science concept of assignment._____no_output_____

a = 1    # assign 1 to a variable named a_____no_output_____a        # disclose a  - you need to explicitly do this!_____no_output_____

Just typing the name of a variable to get the value works in the SageMath Notebook, but if you are writing a program and you want to output the value of a variable, you'll probably want to use something like the print command._____no_output_____

print(a)1
b = 2
c = 3
print a, b, c  # print out the values of a and b and c1 2 3
x=2^(1/2)
x_____no_output_____type(x)   # x is a sage symbolic expression_____no_output_____

Many of the commands in SageMath/Python are "methods" of objects.

That is, we access them by typing:

- the name of the mathematical object,
- a dot/period,
- the name of the method, and
- parentheses (possibly with an argument).

This is a huge advantage, once you get familiar with it, because it allows you to do only the things that are possible, and all such things. See [SageMath programming guide for more details on this](http://doc.sagemath.org/html/en/prep/Programming.html#methods-and-dot-notation).

Let's try to hit the Tab button after the `.` following `x` below to view all available methods for `x` which is currently `sqrt(2)`._____no_output_____

#x.  #  hit the Tab button after the '.' following 'x' _____no_output_____help(x.n)
# we can use ? after a method to get breif helpHelp on built-in function n:

n(...)
    Element.n(self, prec=None, digits=None, algorithm=None)
    File: sage/structure/element.pyx (starting at line 852)

            Alias for :meth:`numerical_approx`.

            EXAMPLES::

                sage: (2/3).n()
                0.666666666666667

x.n(digits=10) # this gives a numerical approximation for x_____no_output_____s = 1; t = 2; u = 3;
print s + t + u6
f=(5-3)^(6/2)+3*(7-2) # assign the expression to f
f # disclose f_____no_output_____type(f)_____no_output_____

### You try

Try assigning some values to some variables - you choose what values and you choose what variable names to use.  See if you can print out the values you have assigned. _____no_output_____You can reassign different values to variable names.  Using SageMath you can also change the type of the values assigned to the variable (not all programming languages allow you to do this)._____no_output_____

a = 1
print "Right now, a =", a, "and is of type", type(a) # using , and strings in double quotes print can be more flexible

a = 1/3 # reassign 1/3 to the variable a
print "Now, a =", a, "and is of type", type(a) # note the change in typeRight now, a = 1 and is of type <type 'sage.rings.integer.Integer'>
Now, a = 1/3 and is of type <type 'sage.rings.rational.Rational'>

### You try

Assign the value `2` to a variable named `x`.

On the next line down in the same cell, assign the value `3` to a variable named `y`.

Then (on a third line)  put in an expression which will evaluate `x + y`_____no_output_____Now try reassigning a different value to x and re-evaluating x + y_____no_output_____## Example 6: Strings

Variables can be strings (an not just numbers).  Anything you put inside quote marks will be treated as a string by SageMath/Python.

Strings as `str` and `unicode` are built-in [sequence types](https://docs.python.org/2/library/stdtypes.html#sequence-types-str-unicode-list-tuple-bytearray-buffer-xrange) for storing strings of bytes and unicode-encoded characters and and operating over them._____no_output_____

myStr = "this is a string"   # assign a string to the variable myStr
myStr                        # disclose myStr_____no_output_____type(myStr)                  # check the type for myStr_____no_output_____

You can also create a string by enclosing them in single quotes or three consecutive single quotes. In SageMath/Python a character (represented by the `char` type in languages like C/C++/Scala) is just a string made up of one character._____no_output_____

myStr = 'this is a string'   # assign a string to the variable myStr using single quotes
myStr                        # disclose myStr_____no_output_____

You can assign values to more than one variable on the same line, by separating the assignment expressions with a semicolon `;`.  However, it is usually best not to do this because it will make your code easier to read (it is hard to spot the other assignments on a single line after the first one)._____no_output_____

myStr = '''this is a string'''   # assign a string to the variable myStr using three consecutive single quotes
myStr                            # disclose myStr_____no_output_____

Using triple single quotes is especially useful if your string has single or double quotes within it. Triple quotes are often used to create `DocString` to document code in Pyhton/SageMath. _____no_output_____

myStrContainingQuotes = '''this string has "a double quoted sub-string" and some escaped characters: \,', - all OK!'''
myStrContainingQuotes_____no_output_____

### Str and unicode Strings

In Python/SageMath, we need to be extremely careful with strings.
The type 'str' is actually a sequence of bytes while the unicode string of type `unicode` is a sequence of unicode characters (some of which can be more than a byte in size). See [this](http://pgbovine.net/unicode-python.htm) for an nice clarification of ASCII and unicode (utf-8) encoded strings.
So, it is a good habit to convert strings from natural languages that are meant for processing into unicode strings using the `decode(utf-8)` method right away._____no_output_____

x = 'hi猫' # this is hi (each letter is encoded by one byte) followed by the Chinese character for cat (3 bytes)
type(x)  # x is of type str = sequence of bytes in Python2 / SageMath_____no_output_____len(x) # this is a sequence of five hexadecimal numbers each requiring a byte to represent_____no_output_____

Disclosing `x` below only shows the hexa-decimal numbers `68` `69` `e7` `8c` `ab`,
but only `h` for `68` and `i` for `69` from [ASCII table](http://www.asciitable.com/), are displayed as characters here,
while `\xe7\x8c\xab` are shown as hexadecimal numbers with prefix `\x`
instead of the Chinese character for cat: 猫_____no_output_____

x    _____no_output_____print(x) # printing a string displays the desired if the display is unicode-compatible_____no_output_____

Generally it is safe  to convert strings from natural languages to unicode in Python/SageMath._____no_output_____

y = x.decode('utf-8') # this decodes or converts the sequence of bytes to a sequence of unicode characters
type(y)               # the type of y now is unicode _____no_output_____len(y)  # now we have a sequence of just 3 unicode characters as we want_____no_output_____

Disclosing `y` shows the two ASCII character `h` and `i` and the Chinese cat character 猫 is specified by the corresponding entry in [utf-8 table](https://en.wikipedia.org/wiki/UTF-8)._____no_output_____

y   # output prepended by u shows it is a unicode sequence as opposed to a str which is a byte sequence_____no_output_____print yhi猫

When programmatically processing sequences of unicode characters it is much safer to work with `repr` for the canonical string representation of the object._____no_output_____

?repr # gives the canonical string representation of the object_____no_output_____print repr(y)u'hi\u732b'
print repr(y).decode('unicode_escape')u'hi猫'

## Pride and Prejudice as unicode

We will explore frequencies of strings for the most downloaded book at [Project Gutenberg](http://www.gutenberg.org/ebooks/search/?sort_order=downloads) that publishes public domain books online.
Currently, books published before 1923 are in the *public domain* - meaning anyone has the right to copy or use the text in any way.

Pride and Prejudice by Jane Austin had the most number of downloads and it's available from

- [http://www.gutenberg.org/ebooks/1342](http://www.gutenberg.org/ebooks/1342).

A quick exploration allows us to see the utf-encoded text [here](http://www.gutenberg.org/files/1342/1342-0.txt).

For now, we will just show how to download the most popular book from the project and display it's contents for processing down the road._____no_output_____

# this downloads the unicode text of the book from the right url we found at the Gutenberg Project
# and assigns it to a variable named prideAndPrejudiceRaw
from urllib import *
prideAndPrejudiceRaw = urlopen('http://www.gutenberg.org/files/1342/1342-0.txt').read().decode('utf-8')
prideAndPrejudiceRaw[0:1000] # just showing the first 1000 raw characters of the downloaded book as unicode_____no_output_____type(prideAndPrejudiceRaw) # this is a sequence of utf-8-encoded characters_____no_output_____len(prideAndPrejudiceRaw)
# the length of the unicode string is about 700 thousand unicode characters_____no_output_____

Next we will show how trivial it is to "read" all the chapters into SageMath/Python using these steps:

- we use regular expressions via the `re` library to substitue all occurences of white-space characters like one or more consecutive end-of-line, tabs, white space characters, etc with a single white space,
- we split by 'Chapter ' into multiple chapters in a list
- print the first 100 character in each of the first 10 Chapters

(don't worry about the details now - we will revist these in detail later)_____no_output_____

myString = "strBlah"
myString_____no_output_____myString.split?_____no_output_____import re
# make a list of chapters
chapterList = re.sub('\\s+', ' ',prideAndPrejudiceRaw).split('Chapter ')[1:10]
for chapter in chapterList:
    print repr(chapter[0:100]).decode('unicode_escape'), '\n';u'1 It is a truth universally acknowledged, that a single man in possession of a good fortune, must be'

u'2 Mr. Bennet was among the earliest of those who waited on Mr. Bingley. He had always intended to vi'

u'3 Not all that Mrs. Bennet, however, with the assistance of her five daughters, could ask on the sub'

u'4 When Jane and Elizabeth were alone, the former, who had been cautious in her praise of Mr. Bingley'

u'5 Within a short walk of Longbourn lived a family with whom the Bennets were particularly intimate. '

u'6 The ladies of Longbourn soon waited on those of Netherfield. The visit was soon returned in due fo'

u"7 Mr. Bennet's property consisted almost entirely in an estate of two thousand a year, which, unfort"

u"8 At five o'clock the two ladies retired to dress, and at half-past six Elizabeth was summoned to di"

u"9 Elizabeth passed the chief of the night in her sister's room, and in the morning had the pleasure "

As we learn more we will return to this popular book's unicode which is stored in our `data` directory as `data\pride_and_prejudice.txt`.

Let us motivate the Python methods we will see soon by using them below to plot the number of occurences of `he` and `she` in each of the 61 chapters of the book._____no_output_____

import re
heList = []
sheList = []
i = 0
# make a list of chapters
chapterList = re.sub('\\s+', ' ',prideAndPrejudiceRaw).split('Chapter ')[1:]
for chapter in chapterList:
    i = i+1 # increment chanpter count
    content = (repr(chapter[0:]).decode('unicode_escape')).lower() # get content as lower-case
    heCount = content.count('he') # count number of 'he' occurrences in the chapter
    sheCount = content.count('she') # count number of 'she' occurrences in the chapter
    heList.append((i,heCount)) # append to heList
    sheList.append((i,sheCount)) # append to sheList_____no_output_____p = points(heList, color='blue',legend_label='he')
p += line(heList, color='blue')
p += points(sheList, color='red',legend_label='she')
p += line(sheList, color='red')
p.axes_labels(['Chapter No.','Frequency'])
p.axes_labels_size(1.0)
p.show(figsize=[8,3])_____no_output_____#p. # uncomment and place the cursor after . and hit Tab button to see options for plotting_____no_output_____type(p) # what we did above is superimposing primitive geometric objects: points and lines_____no_output_____

### Assignment Gotcha!

Let's examine the three assignments in the cell below.

The first assignment of `x=3` is standard: Python/SageMath chooses a memory location for `x` and saves the integer value `3` in it.

The second assignment of `y=x` is more interesting and *Pythonic*: Instead of finding another location for the variable `y` and copying the value of `3` in it, Python/SageMath differs from the ways of C/C++. Since both variables will have the same value after the assignment, Python/SageMath lets `y` point to the memory location of `x`.

Finally, after the third assignment of `y=2`, `x` will be NOT be changed to `2` as because the behavior is not that of a C-pointer. Since `x` and `y` will not share the same value anymore, `y` gets its own memory location, containing `2` and `x` sticks to the originally assigned value `3`._____no_output_____

x=3
print(x) # x is 3
y=x
print(x,y) # x is 3 and y is
y=2
print(x,y)3
(3, 3)
(3, 2)

As every instance (object or variable) has an identity or `id()`, i.e. an integer which is unique within the script or program, we can use `id()` to understand the above behavior of Python/SageMath assignments.

So, let's have a look at our previous example and see how the identities change with the assignments._____no_output_____

x = 3
print('x and its id() are:')
print(x,id(x))

y = x
print('\ny and its id() are:')
print(y,id(y))

y = 2
print('\nx, y and their id()s are:')
print(x,y,id(x),id(y))x and its id() are:
(3, 139801588819408)

y and its id() are:
(3, 139801588819408)

x, y and their id()s are:
(3, 2, 139801588819408, 139801668049392)

## Example 6: Truth statements and Boolean values

Consider statements like "Today is Friday" or "2 is greater than 1" or " 1 equals 1":   statements which are either true or not true (i.e., false).   SageMath has two values, True and False  which you'll meet in this situation.   These value are called Booleans values, or values of the type Boolean.

In SageMath, we can express statements like "2 is greater than 1" or " 1 equals 1" with relational operators, also known as value comparison operators.   Have a look at the list below.

- `<` Less than
- `>` Greater than
- `<=` Less than or equal to
- `>=` Greater than or equal to
- `==` Equal to.
- `!=` Not equal to

Lets try some really simple truth statements._____no_output_____

1 < 1          # 1 is less than 1_____no_output_____

Let us evaluate the following statement._____no_output_____

1 <= 1  # 1 is less than or equal to 1_____no_output_____

We can use these operators on variables as well as on values.   Again, try assigning different values to `x` and `y`, or try using different operators, if you want to._____no_output_____

x = 1          # assign the value 1 to x
y = 2          # assign the value 2 to y
x == y         # evaluate the truth statement "x is equal to y"_____no_output_____

Note that when we check if something equals something else, we use `==`, a double equals sign. This is because `=`, a single equals sign, is the assignment operator we talked about above.  Therefore, to test if `x` equals `y` we can't write `x = y` because this would assign `y to x`, instead we use the equality operator `==` and write `x == y`.

We can also assign a Boolean value to a variable._____no_output_____

# Using the same x and y as above
myBoolean = (x == y)   # assign the result of x == y to the variable myBoolean
myBoolean              # disclose myBoolean_____no_output_____type(myBoolean)        # check the type of myBoolean_____no_output_____

If we want to check if two things are not equal we use `!=`.   As we would expect, it gives us the opposite of testing for equality:_____no_output_____

x != y                 # evaluate the truth statement "x is not equal to y"_____no_output_____print(x,y)             # Let's print x and y to make sure the above statement makes sense (1, 2)

### You try

Try assigning some values to two variables - you choose what values and you choose what variable names to use.  Try some truth statements to check if they are equal, or one is less than the other. _____no_output_____### You try
Try some strings (we looked at strings briefly in Example 5 above). Can you check if two strings are equal? Can you check if one string is less than (`<`) another string. How do you think that Sage is ordering strings (try comparing "fred" and "freddy", for example)?_____no_output_____

'raazb' <= 'raaza'_____no_output_____x = [1]
y = x
y[0] = 5
print x
print(x,id(x),y,id(y))[5]
([5], 139801588827848, [5], 139801588827848)

## Example 7: Mathematical constants

Sage has reserved words that are defined as common mathematical constants.  For example, `pi` and `e` behave as you expect.  Numerical approximations can be obtained using the `.n()` method, as before._____no_output_____

print pi, "~", pi.n()   # print a numerical approximation of the mathematical constant pi
print e, "~", e.n()     # print a numerical approximation of the mathematical constant e
print I, "~", I.n()     # print a numerical approximation of the imaginary number sqrt(-1)pi ~ 3.14159265358979
e ~ 2.71828182845905
I ~ 1.00000000000000*I
(pi/e).n(digits=100)   # print the first 100 digits of pi/e_____no_output_____e^(I*pi)+1          # Euler's identity symbolically - see https://en.wikipedia.org/wiki/Euler%27s_identity_____no_output_____

## Example 8: SageMath number types and Python number types

We showed how you can find the type of a number value and we demonstrated that by default, SageMath makes 'real' numbers like 3.1 into Sage real literals (`sage.rings.real_mpfr.RealLiteral`).

If you were just using Python (the programming language underlying most of SageMath) then a value like 3.1 would be a floating point number or float type.  Python has some interesting extra operators that you can use with Python floating point numbers, which also work with the Sage rings integer type but not with Sage real literals._____no_output_____

X = 3.1 # convert to default Sage real literal 3.1
type(X)_____no_output_____X = float(3.1) # convert the default Sage real literal 3.1 to a float 3.1
type(X)_____no_output_____

Floor Division (`//`) - The division of operands where the result is the quotient in which the digits after the decimal point are removed - the result is floored, i.e., rounded towards negative infinity: examples: 9//2 = 4 and 9.0//2.0 = 4.0, -11//3 = -4, -11.0//3 = -4.0_____no_output_____

3 // 2 # floor division _____no_output_____3.3 // 2.0 # this will give error - floor division is undefined for Sage real literals_____no_output_____float(3.5) // float(2.0)_____no_output_____

Similarly, we have the light-weight Python integer type `int` that we may want instead of SageMath integer type for non-mathematical operations._____no_output_____

type(3) # the default Sage rings integer type_____no_output_____X = int(3) # conversion to a plain Python integer type
type(X)_____no_output_____3/2 # see the result you get when dividing one default Sage rings integer type by another_____no_output_____

One of the differences of SageMath rings integers to plain Python integers is that result of dividing one SageMath rings integer by another is a rational. This probably seems very sensible, but it is not what happens at the moment with Python integers._____no_output_____

int(7)/int(2) # division using python integers is "floor division" _____no_output_____

We showed the `.n()` method.  If X is some Sage real literal and we use `X.n(20)` we will be asking for 20 bits of precision, which is about how many bits in the computer's memory will be allocated to hold the number.  If we ask for `X.n(digits=20)` will be asking for 20 digits of precision, which is not the same thing.  Also note that 20 digits of precision does not mean showing the number to 20 decimal places, it means all the digits including those in front of the decimal point. _____no_output_____

help(n) # always ask for help when you need it - or lookup in help menu aboveHelp on function numerical_approx in module sage.misc.functional:

numerical_approx(x, prec=None, digits=None, algorithm=None)
    Return a numerical approximation of ``self`` with ``prec`` bits
    (or decimal ``digits``) of precision.

    No guarantee is made about the accuracy of the result.

    .. NOTE::

        Lower case :func:`n` is an alias for :func:`numerical_approx`
        and may be used as a method.

    INPUT:

    - ``prec`` -- precision in bits

    - ``digits`` -- precision in decimal digits (only used if
      ``prec`` is not given)

    - ``algorithm`` -- which algorithm to use to compute this
      approximation (the accepted algorithms depend on the object)

    If neither ``prec`` nor ``digits`` is given, the default
    precision is 53 bits (roughly 16 digits).

    EXAMPLES::

        sage: numerical_approx(pi, 10)
        3.1
        sage: numerical_approx(pi, digits=10)
        3.141592654
        sage: numerical_approx(pi^2 + e, digits=20)
        12.587886229548403854
        sage: n(pi^2 + e)
        12.5878862295484
        sage: N(pi^2 + e)
        12.5878862295484
        sage: n(pi^2 + e, digits=50)
        12.587886229548403854194778471228813633070946500941
        sage: a = CC(-5).n(prec=40)
        sage: b = ComplexField(40)(-5)
        sage: a == b
        True
        sage: parent(a) is parent(b)
        True
        sage: numerical_approx(9)
        9.00000000000000

    You can also usually use method notation::

        sage: (pi^2 + e).n()
        12.5878862295484
        sage: (pi^2 + e).numerical_approx()
        12.5878862295484

    Vectors and matrices may also have their entries approximated::

        sage: v = vector(RDF, [1,2,3])
        sage: v.n()
        (1.00000000000000, 2.00000000000000, 3.00000000000000)

        sage: v = vector(CDF, [1,2,3])
        sage: v.n()
        (1.00000000000000, 2.00000000000000, 3.00000000000000)
        sage: _.parent()
        Vector space of dimension 3 over Complex Field with 53 bits of precision
        sage: v.n(prec=20)
        (1.0000, 2.0000, 3.0000)

        sage: u = vector(QQ, [1/2, 1/3, 1/4])
        sage: n(u, prec=15)
        (0.5000, 0.3333, 0.2500)
        sage: n(u, digits=5)
        (0.50000, 0.33333, 0.25000)

        sage: v = vector(QQ, [1/2, 0, 0, 1/3, 0, 0, 0, 1/4], sparse=True)
        sage: u = v.numerical_approx(digits=4)
        sage: u.is_sparse()
        True
        sage: u
        (0.5000, 0.0000, 0.0000, 0.3333, 0.0000, 0.0000, 0.0000, 0.2500)

        sage: A = matrix(QQ, 2, 3, range(6))
        sage: A.n()
        [0.000000000000000  1.00000000000000  2.00000000000000]
        [ 3.00000000000000  4.00000000000000  5.00000000000000]

        sage: B = matrix(Integers(12), 3, 8, srange(24))
        sage: N(B, digits=2)
        [0.00  1.0  2.0  3.0  4.0  5.0  6.0  7.0]
        [ 8.0  9.0  10.  11. 0.00  1.0  2.0  3.0]
        [ 4.0  5.0  6.0  7.0  8.0  9.0  10.  11.]

    Internally, numerical approximations of real numbers are stored in base-2.
    Therefore, numbers which look the same in their decimal expansion might be
    different::

        sage: x=N(pi, digits=3); x
        3.14
        sage: y=N(3.14, digits=3); y
        3.14
        sage: x==y
        False
        sage: x.str(base=2)
        '11.001001000100'
        sage: y.str(base=2)
        '11.001000111101'

    Increasing the precision of a floating point number is not allowed::

        sage: CC(-5).n(prec=100)
        Traceback (most recent call last):
        ...
        TypeError: cannot approximate to a precision of 100 bits, use at most 53 bits
        sage: n(1.3r, digits=20)
        Traceback (most recent call last):
        ...
        TypeError: cannot approximate to a precision of 70 bits, use at most 53 bits
        sage: RealField(24).pi().n()
        Traceback (most recent call last):
        ...
        TypeError: cannot approximate to a precision of 53 bits, use at most 24 bits

    As an exceptional case, ``digits=1`` usually leads to 2 digits (one
    significant) in the decimal output (see :trac:`11647`)::

        sage: N(pi, digits=1)
        3.2
        sage: N(pi, digits=2)
        3.1
        sage: N(100*pi, digits=1)
        320.
        sage: N(100*pi, digits=2)
        310.

    In the following example, ``pi`` and ``3`` are both approximated to two
    bits of precision and then subtracted, which kills two bits of precision::

        sage: N(pi, prec=2)
        3.0
        sage: N(3, prec=2)
        3.0
        sage: N(pi - 3, prec=2)
        0.00

    TESTS::

        sage: numerical_approx(I)
        1.00000000000000*I
        sage: x = QQ['x'].gen()
        sage: F.<k> = NumberField(x^2+2, embedding=sqrt(CC(2))*CC.0)
        sage: numerical_approx(k)
        1.41421356237309*I

        sage: type(numerical_approx(CC(1/2)))
        <type 'sage.rings.complex_number.ComplexNumber'>

    The following tests :trac:`10761`, in which ``n()`` would break when
    called on complex-valued algebraic numbers.  ::

        sage: E = matrix(3, [3,1,6,5,2,9,7,3,13]).eigenvalues(); E
        [18.16815365088822?, -0.08407682544410650? - 0.2190261484802906?*I, -0.08407682544410650? + 0.2190261484802906?*I]
        sage: E[1].parent()
        Algebraic Field
        sage: [a.n() for a in E]
        [18.1681536508882, -0.0840768254441065 - 0.219026148480291*I, -0.0840768254441065 + 0.219026148480291*I]

    Make sure we've rounded up log(10,2) enough to guarantee
    sufficient precision (:trac:`10164`)::

        sage: ks = 4*10**5, 10**6
        sage: check_str_length = lambda k: len(str(numerical_approx(1+10**-k,digits=k+1)))-1 >= k+1
        sage: check_precision = lambda k: numerical_approx(1+10**-k,digits=k+1)-1 > 0
        sage: all(check_str_length(k) and check_precision(k) for k in ks)
        True

    Testing we have sufficient precision for the golden ratio (:trac:`12163`), note
    that the decimal point adds 1 to the string length::

        sage: len(str(n(golden_ratio, digits=5000)))
        5001
        sage: len(str(n(golden_ratio, digits=5000000)))  # long time (4s on sage.math, 2012)
        5000001

    Check that :trac:`14778` is fixed::

        sage: n(0, algorithm='foo')
        0.000000000000000

X=3.55555555
X.n(digits = 3)_____no_output_____X.n(3) # this will use 3 bits of precision_____no_output_____round(X,3)_____no_output_____#?round # uncomment and evaluate this to open a window with help information that can be closed_____no_output_____

If you want to actually round a number to a specific number of decimal places, you can also use the round(...) function.

For deeper dive see documents on [Python Numeric Types](https://docs.python.org/2/library/stdtypes.html#numeric-types-int-float-long-complex) and [SageMath Numeric Types]()_____no_output_____# Sets

Set theory is at the very foundation in modern mathematics and is necessary to understand the mathematical notions of probability and statistics. We will take a practical mathemtical tour of the essential concepts from set theory that a data scientist needs to understand and build probabilistic models from the data using statistical principles._____no_output_____

showURL("https://en.wikipedia.org/wiki/Set_(mathematics)",500)_____no_output_____

## Essentials of Set Theory for Probability and Statistics

**These are black-board lectures typeset here.**

Let us learn or recall elementary set theory.  Sets are perhaps the most fundamental concept in mathematics.

### Definitions

**Set** *is a collection of distinct elements*.

We write a set by enclosing its elements with curly brackets.  Let us see some example next.

- The collection of $\star$ and $\circ$ is $\{\star,\circ\}$.
- We can name the set $\{\star,\circ\}$ by the letter $A$ and write $$A=\{\star,\circ\}.$$
- Question: Is $\{\star,\star,\circ\}$ a set?
- A set of letters and numbers that I like is $\{b,d,6,p,q,9\}$.
- The set of first five Greek alphabets is $\{\alpha,\beta,\gamma,\delta,\epsilon\}$.

The set that contains no elements is the **empty set**.  It is denoted by
$$\boxed{\emptyset = \{\}} \ .$$

We say an element belongs to or does not belong to a set with the binary operators
$$\boxed{\in \ \text{or} \ \notin} \ .$$
For example,

- $\star \in \{\star,\circ\}$ but the element $\otimes \notin \{\star,\circ\}$
- $b \in \{b,d,6,p,q,9\}$ but $8 \notin \{b,d,6,p,q,9\}$
- Question: Is $9 \in \{3,4,1,5,2,8,6,7\}$?

We say a set $C$ is a **subset** of a set $D$ and write

$$\boxed{C \subset D}$$

if every element of $C$ is also an element of $D$.  For example,

- $\{\star\} \subset \{\star,\circ\}$
- Question: Is $\{6,9\}\subset \{b,d,6,p,q,9\}$?

### Set Operations

We can add distinct new elements to an existing set by **union** operation denoted by $\cup$ symbol.

For example

- $\{\circ, \bullet\} \cup \{\star\} = \{\circ,\bullet,\star\}$
- Question: $\{\circ, \bullet\} \cup \{\bullet\} = \quad$?

More formally, we write the union of two sets $A$ and $B$ as
$$\boxed{A \cup B = \{x: x \in A \ \text{or} \ x \in B \}} \ .$$

The symbols above are read as *$A$ union $B$ is equal to the set of all $x$ such that $x$ belongs to $A$ or $x$ belongs to $B$* and simply means that $A$ union $B$ or $A \cup B$ is the set of elements that belong to $A$ or $B$.

Similarly, the **intersection** of two sets $A$ and $B$ written as
$$\boxed{A \cap B = \{x: x \in A \ \text{and} \ x \in B \}} $$
means $A$ intersection $B$ is the set of  elements that belong to both $A$ and $B$.

For example

- $\{\circ, \bullet\} \cap \{\circ\} = \{\circ\}$
- $\{\circ, \bullet\} \cap \{\bullet\} = \{\bullet\}$
- $\{\circ\} \cap \{a,b,c,d\}=\emptyset$

The **set difference** of two sets $A$ and $B$ written as
$$\boxed{A \setminus B = \{x: x \in A \ \text{and} \ x \notin B \}} $$
means $A \setminus B$ is the set of  elements that belong to $A$ and not belong to $B$.

For example

- $\{\circ, \bullet\} \setminus \{\circ\} = \{\bullet\}$
- $\{\circ, \bullet\} \setminus \{\bullet\} = \{\circ\}$
- $\{a,b,c,d\} \setminus \{a,b,c,d\}=\emptyset$

The equality of two sets $A$ and $B$ is defined in terms of subsets as follows:
$$\boxed{A = B \quad \text{if and only if} \quad A \subset B \ \text{and} \ B \subset A} \ .$$

Two sets $A$ anb $B$ are said to be **disjoint** if
$$\boxed{ A \cap B = \emptyset} \ .$$

Given a **universal set** $\Omega$, we define the **complement** of a subset $A$ of the universal set by $$\boxed{A^c = \Omega \setminus A = \{x: x \in \Omega \ \text{and} \ x \notin A\}} \ .$$
_____no_output_____### An Interactive Venn Diagram

Let us gain more intuition by seeing the unions and intersections of sets interactively.  The following interact is from [interact/misc](https://wiki.sagemath.org/interact/misc#AnInteractiveVennDiagram) page  of Sage Wiki._____no_output_____

# ignore this code for now and focus on the interact in the output cell
def f(s, braces=True):
    t = ', '.join(sorted(list(s)))
    if braces: return '{' + t + '}'
    return t
def g(s): return set(str(s).replace(',',' ').split())

@interact
def _(X='1,2,3,a', Y='2,a,3,4,apple', Z='a,b,10,apple'):
    S = [g(X), g(Y), g(Z)]
    X,Y,Z = S
    XY = X & Y
    XZ = X & Z
    YZ = Y & Z
    XYZ = XY & Z
    pretty_print(html(''))
    pretty_print(html("$X \cap Y$ = %s"%f(XY)))
    pretty_print(html("$X \cap Z$ = %s"%f(XZ)))
    pretty_print(html("$Y \cap Z$ = %s"%f(YZ)))
    pretty_print(html("$X \cap Y \cap Z$ = %s"%f(XYZ)))
    pretty_print(html(''))
    centers = [(cos(n*2*pi/3), sin(n*2*pi/3)) for n in [0,1,2]]
    scale = 1.7
    clr = ['yellow', 'blue', 'green']
    G = Graphics()
    for i in range(len(S)):
        G += circle(centers[i], scale, rgbcolor=clr[i],
             fill=True, alpha=0.3)
    for i in range(len(S)):
        G += circle(centers[i], scale, rgbcolor='black')

    # Plot what is in one but neither other
    for i in range(len(S)):
        Z = set(S[i])
        for j in range(1,len(S)):
            Z = Z.difference(S[(i+j)%3])
        G += text(f(Z,braces=False), (1.5*centers[i][0],1.7*centers[i][1]), rgbcolor='black')

    # Plot pairs of intersections
    for i in range(len(S)):
        Z = (set(S[i]) & S[(i+1)%3]) - set(XYZ)
        C = (1.3*cos(i*2*pi/3 + pi/3), 1.3*sin(i*2*pi/3 + pi/3))
        G += text(f(Z,braces=False), C, rgbcolor='black')

    # Plot intersection of all three
    G += text(f(XYZ,braces=False), (0,0), rgbcolor='black')

    # Show it
    G.show(aspect_ratio=1, axes=False)_____no_output_____

# Create and manipulate sets in SageMath._____no_output_____## Example 0: Lists before Sets

A `list` is a sequential collection that we will revisit in detail soon. For now, we just need to know that we can create a list by using delimiter `,` between items and by wrapping with left and right square brackets: `[` and `]`. For example, the following is a list of 4 integers:_____no_output_____

[1,2,3,4]_____no_output_____myList = [1,2,3,4]    # we can assign the list to a variable myList
print(myList)         # print myList
type(myList)          # and ask for its type [1, 2, 3, 4]

List is one of the most primitive data structures and has a long history in a popular computer programming language called LISP - originally created as a practical mathematical notation for computer programs.

For now, we just use lists to create sets._____no_output_____## Example 1: Making sets

In SageMath, you do have to specifically say that you want a set when you make it.

Do the following in SageMath/Python to create the mathematical set:

$$X = \{1,2,3,4\}$$_____no_output_____

X = set([1, 2, 3, 4])  # make the set X={1,2,3,4} from the List [1,2,3,4]
X                      # disclose X_____no_output_____type(X)                # what is the type of X_____no_output_____

This is a specialized datatype in Python and more details can be found in Python docs: [https://docs.python.org/2/library/datatypes.html](https://docs.python.org/2/library/datatypes.html)

Do the following in SageMath/Python to find out if:

$$4 \in X$$

i.e., if $4$ is an element of $X$ or if $4$ is an element of $X$ or $4$ is in $X$._____no_output_____

4 in X                 # 'is 4 in X?'_____no_output_____5 in X                 # 'is 5 in X?'_____no_output_____Y = set([1, 2])        # make the set Y={1,2}
Y                      # disclose Y_____no_output_____4 not in Y             # 'is 4 not in Y?'_____no_output_____1 not in Y             # 'is 1 not in Y?'_____no_output_____

We can add new elements to a set._____no_output_____

X.add(5)     # add 5 to the set X
X_____no_output_____

But remember from the mathematical exposition above that sets contain distinct elements._____no_output_____

X.add(1)     # try adding another 1 to the set X
X_____no_output_____

### You try

Try making the set $Z=\{4,5,6,7\}$ next.   The instructions are in the two cells below._____no_output_____

# Write in the expression to make set Z ={4, 5, 6, 7}
# (press ENTER at the end of this line to get a new line)
Z = set([4,5,6,7]) #([4],[5],[6,7]))_____no_output_____# Check if 4 is in Z
4 in Z
# (press ENTER at the end of this line to get a new line)_____no_output_____

Make a set with the value 2/5 (as a rational) in it.  Try adding 0.4 (as a floating point number) to the set.

Does SageMath do what you expect?_____no_output_____## Example 2: Subsets

In lectures we talked about subsets of sets.

Recall that `Y` is a subset of `X` if each element in `Y` is also in `X`._____no_output_____

print "X is", X
print "Y is", Y
print "Is Y a subset of X?"
Y <= X                       # 'is Y a subset of X?'X is set([1, 2, 3, 4, 5])
Y is set([1, 2])
Is Y a subset of X?

If you have time: We say Y is a proper subset of X if all the elements in Y are also in X but there is at least one element in X that it is not in Y.  If X is a (proper) subset of Y, then we also say that Y is a (proper) superset of X._____no_output_____

X < X     # 'is X a proper subset of itself?'_____no_output_____X > Y     # 'is X a proper superset of Y?'_____no_output_____X > X     # 'is X a proper superset of itself?'_____no_output_____X >= Y     # 'is X a superset of Y?' is the same as 'is Y a subset of X?'_____no_output_____

## Example 3: More set operations

Now let's have a look at the other set operations we talked about above:  intersection, union, and difference.

Recall that the intersection of X and Y is the set of elements that are in both  X and Y._____no_output_____

X & Y    # '&' is the intersection operator_____no_output_____

The union of X and Y is the set of elements that are in either  X or  Y._____no_output_____

X | Y    # '|' is the union operator_____no_output_____

The set difference between X and Y is the set of elements in X that are not in  Y._____no_output_____

X - Y    # '-' is the set difference operator_____no_output_____

### You try

Try some more work with sets of strings below._____no_output_____

fruit = set(['orange', 'banana', 'apple'])
fruit_____no_output_____colours = set(['red', 'green', 'blue', 'orange'])
colours_____no_output_____

Fruit and colours are different to us as people, but to the computer, the string 'orange' is just the string 'orange' whether it is in a set called fruit or a set called colours._____no_output_____

print "fruit intersection colours is", fruit & colours
print "fruit union colours is", fruit | colours
print "fruit - colours is", fruit - colours
print "colours - fruit is", colours - fruitfruit intersection colours is set(['orange'])
fruit union colours is set(['blue', 'green', 'apple', 'orange', 'banana', 'red'])
fruit - colours is set(['banana', 'apple'])
colours - fruit is set(['blue', 'green', 'red'])

Try a few other simple subset examples - make up your own sets and try some intersections, unions, and set difference operations.  The best way to try new possible operations on a set such as X we just created is to type a period after X and hit `<TAB>` key.  THis will bring up all the possible methods you can call on the set X._____no_output_____

mySet = set([1,2,3,4,5,6,7,8,9])_____no_output_____#mySet.         # uncomment and try placing the cursor after the dot and hit <TAB> key_____no_output_____#?mySet.add     # uncomment and evaluate to get help on a method by prepending a question mark_____no_output_____

Infact, there are two ways to make sets in SageMath.  We have so far used [the python set](https://docs.python.org/2/library/sets.html) to make a set.

However we can use the SageMath `Set` to maka sets too.  SageMath `Set` is more mathematically consisitent.  If you are interested in the SageMath `Set` go to the source and work through the [SageMath reference on Sets](http://doc.sagemath.org/html/en/reference/sets/sage/sets/set.html).

But, first let us appreciate the difference between Python `set` and SageMath `Set`!_____no_output_____

X = set([1, 2, 3, 4])  # make the set X={1,2,3,4} with python set
X                      # disclose X_____no_output_____type(X)                # this is the set in python_____no_output_____anotherX = Set([1, 2, 3, 4])   # make the set anotherX={1,2,3,4} in SAGE Set
anotherX #disclose it_____no_output_____type(anotherX)                 # this is the set in SAGE and is more mathy_____no_output_____#anotherX. # see what methods are available in SageMath Set_____no_output_____

## Example 4
Python also provides something called a [frozenset](https://docs.python.org/2/library/stdtypes.html#frozenset), which you can't change like an ordinary set._____no_output_____

aFrozenSet = frozenset([2/5, 0.2, 1/7, 0.1])
aFrozenSet_____no_output_____#aFrozenSet.add(0.3) # This should give an error_____no_output_____

## Under the Set's Hood

The key is to remember that sets are unordered:  a set {1, 2, 3} is the same as the set {2, 1, 3} is the same as the set {3, 1, 2} ...  (soon we will talk about ordered collections like lists where order is important - but for the moment just remember that a set is about collections of unique values or objects). Remember also that the (lower case s) set is a python type (in contrast to the 'more mathy' SageMath Set with a capital S).  For many 'non-mathy' computing purposes, what matters about a set is the speed of being able to find things in the set without making it expensive (in computer power) to add and remove things, and the best way of doing this (invisible to the programmer actually using the set) is to base the set on a [hash table](https://en.wikipedia.org/wiki/Hash_table).  In practical terms what this means is that the ordering that Sage uses to actually display a set is related to the hash values it has given to the elements in the set which may be totally unrelated to what you or I would think of as any obvious ordering based on the magnitude of the values in the set, or the order in which we put them in, etc.   Remember, you don't need to know anything about hash values and the 'under-the-hood' construction of sets for this course - this explanation is just to give a bit more background._____no_output_____## Using sets to find the words with 15 or more characters in P&P_____no_output_____

# This time we are loading the file from our data directory
with open('data/pride_and_prejudice.txt', 'r') as myfile:
  prideAndPrejudiceRaw = myfile.read().decode('utf-8')

import re
bigWordsSet = set([]) # start an empty set
# make a list of chapters
chapterList = re.sub('\\s+', ' ',prideAndPrejudiceRaw).split('Chapter ')[1:]
for chapter in chapterList:
    content = (repr(chapter[0:]).decode('unicode_escape')).lower() # get content as lower-case
    bigWordsSet = bigWordsSet | \
                    set([x for x in content.split(' ') if (len(x)>14 and x.isalpha())])
    # union with list-comprehended set - soon you will get this!_____no_output_____bigWordsSet # set of big words with > 14 characters in any chapter of P&P_____no_output_____

---
---
**Note** that there are much more convenient methods to read text files that have been structured into a certain form. You are purposely being keept much closer to the raw data here. Starting directly with convenient high-level methods is not a good way to appreciate wrangling data closer to the source. Furthermore, it is easier to pick up the high-level methods later on._____no_output_____


# Notebook from rodrigomfguedes/business-cases-21-22
Path: BC4_crypto_forecasting/scripts_updated/MATIC_notebook.ipynb

#  Forecasting - MATIC

### Master Degree Program in Data Science and Advanced Analytics

###  Business Cases with Data Science Project:
> #### Group AA

###  Done by:
> #### - Beatriz Martins Selidónio Gomes, m20210545
> #### - Catarina Inês Lopes Garcez, m20210547
> #### - Diogo André Domingues Pires, m20201076
> #### - Rodrigo Faísca Guedes, m20210587 _____no_output_____---_____no_output_____

# Table of Content

###  Import and Data Integration
- [Import the needed Libraries](#third-bullet)

###  Data Exploration and Understanding
- [Initial Analysis (EDA - Exploratory Data Analysis)](#fifth-bullet)

- [Variables Distribution](#seventh-bullet)

###  Data Preparation
- [Data Transformation](#eighth-bullet)

###  Modelling
- [Building LSTM Model](#twentysecond-bullet)

- [Get Best Parameters for LSTM](#twentythird-bullet)

- [Run the LSTM Model and Get Predictions](#twentyfourth-bullet)

- [Recursive Predictions](#twentysixth-bullet)

_____no_output_____---_____no_output_____# Import and Data Integration

## Import the needed Libraries
  [Back to TOC](#toc)_____no_output_____

import warnings
warnings.filterwarnings('ignore')

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt_____no_output_____

# Data Exploration and Understanding

## Initial Analysis (EDA - Exploratory Data Analysis)
  [Back to TOC](#toc)
_____no_output_____

df = pd.read_csv('../data/data_aux/df_MATIC.csv')
df_____no_output_____

### Data Types_____no_output_____

# Get to know the number of instances and Features, the DataTypes and if there are missing values in each Feature
df.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1826 entries, 0 to 1825
Data columns (total 7 columns):
 #   Column              Non-Null Count  Dtype
---  ------              --------------  -----
 0   Date                1826 non-null   object
 1   MATIC-USD_ADJCLOSE  1094 non-null   float64
 2   MATIC-USD_CLOSE     1094 non-null   float64
 3   MATIC-USD_HIGH      1094 non-null   float64
 4   MATIC-USD_LOW       1094 non-null   float64
 5   MATIC-USD_OPEN      1094 non-null   float64
 6   MATIC-USD_VOLUME    1094 non-null   float64
dtypes: float64(6), object(1)
memory usage: 100.0+ KB

### Missing Values_____no_output_____

# Count the number of missing values for each Feature
df.isna().sum().to_frame().rename(columns={0: 'Count Missing Values'})_____no_output_____

### Descriptive Statistics_____no_output_____

# Descriptive Statistics Table
df.describe().T_____no_output_____# settings to display all columns
pd.set_option("display.max_columns", None)

# display the dataframe head
df.sample(n=10)_____no_output_____#CHECK ROWS THAT HAVE ANY MISSING VALUE IN ONE OF THE COLUMNS
is_NaN = df.isnull()
row_has_NaN = is_NaN.any(axis=1)
rows_with_NaN = df[row_has_NaN]
rows_with_NaN_____no_output_____#FILTER OUT ROWS THAT ARE MISSING INFORMATION
df = df[~row_has_NaN]
df.reset_index(inplace=True, drop=True)
df_____no_output_____

# Data Preparation

## Data Transformation
  [Back to TOC](#toc)_____no_output_____ __`Duplicates`_______no_output_____

# Checking if exist duplicated observations
print(f'\033[1m' + "Number of duplicates: " + '\033[0m', df.duplicated().sum())  Number of duplicates:  0

 __`Convert Date to correct format`_______no_output_____

df['Date'] = pd.to_datetime(df['Date'], format='%Y-%m-%d')
df_____no_output_____

 __`Get percentual difference between open and close values and low and high values`_______no_output_____

df['pctDiff_CloseOpen'] = abs((df[df.columns[2]]-df[df.columns[5]])/df[df.columns[2]])*100
df['pctDiff_HighLow'] = abs((df[df.columns[3]]-df[df.columns[4]])/df[df.columns[4]])*100
df.head()_____no_output_____def plot_coinValue(df):

    #Get coin name
    coin_name = df.columns[2].split('-')[0]

    #Get date and coin value
    x = df['Date']
    y = df[df.columns[2]] # ADA-USD_CLOSE

    #Get the volume of trades
    v = df[df.columns[-3]]/1e9

    #Get percentual diferences
    y2 = df[df.columns[-1]] # pctDiff_HighLow
    y1= df[df.columns[-2]]  # pctDiff_CloseOpen

    fig, axs = plt.subplots(3, 1, figsize=(12,14))

    axs[0].plot(x, y)
    axs[2].plot(x, v)

    # plotting the line 1 points
    axs[1].plot(x, y1, label = "Close/Open")

    # plotting the line 2 points
    axs[1].plot(x, y2, label = "High/Low")
    axs[1].legend()

    axs[0].title.set_text('Time Evolution of '+ coin_name)
    axs[0].set(xlabel="", ylabel="Close Value in USD$")

    axs[2].title.set_text('Volume of trades of '+ coin_name)
    axs[2].set(xlabel="", ylabel="Total number of trades in billions")

    axs[1].title.set_text('Daily Market percentual differences of '+ coin_name)
    axs[1].set(xlabel="", ylabel="Percentage (%)")

    plt.savefig('../analysis/'+coin_name +'_stats'+'.png')

    return coin_name_____no_output_____coin_name = plot_coinValue(df)_____no_output_____#FILTER DATASET
df = df.loc[df['Date']>= '2021-07-01']
df_____no_output_____

# Modelling

## Building LSTM Model
  [Back to TOC](#toc)_____no_output_____## Strategy

Create a DF (windowed_df) where the middle columns will correspond to the close values of X days before the target date and the final column will correspond to the close value of the target date. Use these values for prediction and play with the value of X_____no_output_____

def get_windowed_df(X, df):
    start_Date = df['Date'] + pd.Timedelta(days=X)

    perm = np.zeros((1,X+1))

    #Get labels for DataFrame
    j=1
    labels=[]

    while j <= X:
        label = 'closeValue_' + str(j) + 'daysBefore'
        labels.append(label)

        j+=1

    labels.append('closeValue')

    for i in range(X,df.shape[0]):
        temp = np.zeros((1,X+1))

        #Date for i-th day
        #temp[0,0] = df.iloc[i]['Date']

        #Close values for k days before
        for k in range(X):
            temp[0,k] = df.iloc[i-k-1,2]

        #Close value for i-th date
        temp[0,-1] = df.iloc[i,2]

        #Add values to the permanent frame
        perm = np.vstack((perm,temp))

        #Get the array in dataframe form
        windowed_df = pd.DataFrame(perm[1:,:], columns = labels)

    return windowed_df_____no_output_____#Get the dataframe and append the dates
windowed_df = get_windowed_df(7, df)
windowed_df['Date'] = df.iloc[7:]['Date'].reset_index(drop=True)
windowed_df_____no_output_____#Get the X,y and dates into a numpy array to apply on a model

def windowed_df_to_date_X_y(windowed_dataframe):
    df_as_np = windowed_dataframe.to_numpy()

    dates = df_as_np[:, -1]

    middle_matrix = df_as_np[:, 0:-2]
    X = middle_matrix.reshape((len(dates), middle_matrix.shape[1], 1))

    Y = df_as_np[:, -2]

    return dates, X.astype(np.float32), Y.astype(np.float32)

dates, X, y = windowed_df_to_date_X_y(windowed_df)

dates.shape, X.shape, y.shape_____no_output_____#Partition for train, validation and test

q_80 = int(len(dates) * .85)
q_90 = int(len(dates) * .95)

dates_train, X_train, y_train = dates[:q_80], X[:q_80], y[:q_80]

dates_val, X_val, y_val = dates[q_80:q_90], X[q_80:q_90], y[q_80:q_90]
dates_test, X_test, y_test = dates[q_90:], X[q_90:], y[q_90:]

fig,axs = plt.subplots(1, 1, figsize=(12,5))

#Plot the partitions
axs.plot(dates_train, y_train)
axs.plot(dates_val, y_val)
axs.plot(dates_test, y_test)

axs.legend(['Train', 'Validation', 'Test'])

fig.savefig('../analysis/'+coin_name +'_partition'+'.png')_____no_output_____

## Get Best Parameters for LSTM
  [Back to TOC](#toc)_____no_output_____

#!pip install tensorflow_____no_output_____#import os
#os.environ['PYTHONHASHSEED']= '0'
#import numpy as np
#np.random.seed(1)
#import random as rn
#rn.seed(1)
#import tensorflow as tf
#tf.random.set_seed(1)
#
#from tensorflow.keras.models import Sequential
#from tensorflow.keras.optimizers import Adam
#from tensorflow.keras import layers
#from sklearn.metrics import mean_squared_error
#
## Function to create LSTM model and compute the MSE value for the given parameters
#def check_model(X_train, y_train, X_val, y_val, X_test, y_test, learning_rate,epoch,batch):
#
#    # create model
#    model = Sequential([layers.Input((7, 1)),
#                        layers.LSTM(64),
#                        layers.Dense(32, activation='relu'),
#                        layers.Dense(32, activation='relu'),
#                        layers.Dense(1)])
#    # Compile model
#    model.compile(loss='mse', optimizer=Adam(learning_rate=learning_rate), metrics=['mean_absolute_error'])
#
#    model.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=epoch, shuffle=False, batch_size=batch, verbose=2)
#
#    test_predictions = model.predict(X_test).flatten()
#
#    LSTM_mse = mean_squared_error(y_test, test_predictions)
#
#    return LSTM_mse
#
##Function that iterates the different parameters and gets the ones corresponding to the lowest MSE score.
#def search_parameters(batch_size, epochs, learn_rate, X_train, y_train, X_val, y_val, X_test, y_test):
#
#    best_score = float('inf')
#
#    for b in batch_size:
#        for e in epochs:
#            for l in learn_rate:
#                print('Batch Size: ' + str(b))
#                print('Number of Epochs: ' + str(e))
#                print('Value of Learning Rate: ' + str(l))
#                try:
#                    mse = check_model(X_train, y_train, X_val, y_val, X_test, y_test,l,e,b)
#                    print('MSE=%.3f' % (mse))
#                    if mse < best_score:
#                        best_score = mse
#                        top_params = [b, e, l]
#                except:
#                    continue
#
#    print('Best MSE=%.3f' % (best_score))
#    print('Optimal Batch Size: ' + str(top_params[0]))
#    print('Optimal Number of Epochs: ' + str(top_params[1]))
#    print('Optimal Value of Learning Rate: ' + str(top_params[2]))
#
#
## define parameters
#batch_size = [10, 100, 1000]
#epochs = [50, 100]
#learn_rate = np.linspace(0.001,0.1, num=10)
#
#warnings.filterwarnings("ignore")
#search_parameters(batch_size, epochs, learn_rate, X_train, y_train, X_val, y_val, X_test, y_test)_____no_output_____

## Run the LSTM Model and Get Predictions
  [Back to TOC](#toc)_____no_output_____

#BEST SOLUTION OF THE MODEL

# MSE=0.003
# Batch Size: 100
# Number of Epochs: 100
# Value of Learning Rate: 0.1
from tensorflow.keras.models import Sequential
from tensorflow.keras.models import Sequential
from tensorflow.keras.optimizers import Adam
from tensorflow.keras import layers
from sklearn.metrics import mean_squared_error

model = Sequential([layers.Input((7, 1)),
                    layers.LSTM(64),
                    layers.Dense(32, activation='relu'),
                    layers.Dense(32, activation='relu'),
                    layers.Dense(1)])

model.compile(loss='mse',
              optimizer=Adam(learning_rate=0.1),
              metrics=['mean_absolute_error'])

model.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=100, shuffle=False, batch_size=100, verbose=2)Epoch 1/100
3/3 - 2s - loss: 217.1404 - mean_absolute_error: 10.1857 - val_loss: 2.0833 - val_mean_absolute_error: 1.4410 - 2s/epoch - 510ms/step
Epoch 2/100
3/3 - 0s - loss: 6.9966 - mean_absolute_error: 2.4377 - val_loss: 0.8128 - val_mean_absolute_error: 0.8953 - 27ms/epoch - 9ms/step
Epoch 3/100
3/3 - 0s - loss: 1.1136 - mean_absolute_error: 0.8054 - val_loss: 0.2198 - val_mean_absolute_error: 0.4575 - 26ms/epoch - 9ms/step
Epoch 4/100
3/3 - 0s - loss: 0.7151 - mean_absolute_error: 0.7937 - val_loss: 0.1237 - val_mean_absolute_error: 0.3323 - 26ms/epoch - 9ms/step
Epoch 5/100
3/3 - 0s - loss: 0.2926 - mean_absolute_error: 0.4262 - val_loss: 0.4557 - val_mean_absolute_error: 0.6658 - 27ms/epoch - 9ms/step
Epoch 6/100
3/3 - 0s - loss: 0.6362 - mean_absolute_error: 0.6819 - val_loss: 0.1242 - val_mean_absolute_error: 0.3382 - 24ms/epoch - 8ms/step
Epoch 7/100
3/3 - 0s - loss: 0.3740 - mean_absolute_error: 0.4631 - val_loss: 0.0211 - val_mean_absolute_error: 0.1176 - 27ms/epoch - 9ms/step
Epoch 8/100
3/3 - 0s - loss: 0.1485 - mean_absolute_error: 0.3327 - val_loss: 0.1612 - val_mean_absolute_error: 0.3912 - 27ms/epoch - 9ms/step
Epoch 9/100
3/3 - 0s - loss: 0.2003 - mean_absolute_error: 0.3768 - val_loss: 0.0924 - val_mean_absolute_error: 0.2852 - 26ms/epoch - 9ms/step
Epoch 10/100
3/3 - 0s - loss: 0.1468 - mean_absolute_error: 0.2933 - val_loss: 0.0775 - val_mean_absolute_error: 0.2562 - 28ms/epoch - 9ms/step
Epoch 11/100
3/3 - 0s - loss: 0.1094 - mean_absolute_error: 0.2867 - val_loss: 0.0154 - val_mean_absolute_error: 0.1029 - 28ms/epoch - 9ms/step
Epoch 12/100
3/3 - 0s - loss: 0.1091 - mean_absolute_error: 0.2690 - val_loss: 0.0207 - val_mean_absolute_error: 0.1240 - 26ms/epoch - 9ms/step
Epoch 13/100
3/3 - 0s - loss: 0.0708 - mean_absolute_error: 0.2144 - val_loss: 0.0650 - val_mean_absolute_error: 0.2351 - 25ms/epoch - 8ms/step
Epoch 14/100
3/3 - 0s - loss: 0.0743 - mean_absolute_error: 0.2127 - val_loss: 0.0212 - val_mean_absolute_error: 0.1291 - 27ms/epoch - 9ms/step
Epoch 15/100
3/3 - 0s - loss: 0.0888 - mean_absolute_error: 0.2232 - val_loss: 0.0239 - val_mean_absolute_error: 0.1229 - 27ms/epoch - 9ms/step
Epoch 16/100
3/3 - 0s - loss: 0.0613 - mean_absolute_error: 0.1995 - val_loss: 0.0227 - val_mean_absolute_error: 0.1176 - 26ms/epoch - 9ms/step
Epoch 17/100
3/3 - 0s - loss: 0.0531 - mean_absolute_error: 0.1756 - val_loss: 0.0221 - val_mean_absolute_error: 0.1295 - 27ms/epoch - 9ms/step
Epoch 18/100
3/3 - 0s - loss: 0.0657 - mean_absolute_error: 0.2080 - val_loss: 0.0261 - val_mean_absolute_error: 0.1281 - 25ms/epoch - 8ms/step
Epoch 19/100
3/3 - 0s - loss: 0.0452 - mean_absolute_error: 0.1650 - val_loss: 0.0108 - val_mean_absolute_error: 0.0859 - 27ms/epoch - 9ms/step
Epoch 20/100
3/3 - 0s - loss: 0.0512 - mean_absolute_error: 0.1663 - val_loss: 0.0098 - val_mean_absolute_error: 0.0813 - 28ms/epoch - 9ms/step
Epoch 21/100
3/3 - 0s - loss: 0.0540 - mean_absolute_error: 0.1669 - val_loss: 0.0091 - val_mean_absolute_error: 0.0832 - 25ms/epoch - 8ms/step
Epoch 22/100
3/3 - 0s - loss: 0.0672 - mean_absolute_error: 0.1857 - val_loss: 0.0133 - val_mean_absolute_error: 0.0925 - 26ms/epoch - 9ms/step
Epoch 23/100
3/3 - 0s - loss: 0.0506 - mean_absolute_error: 0.1777 - val_loss: 0.0119 - val_mean_absolute_error: 0.0859 - 27ms/epoch - 9ms/step
Epoch 24/100
3/3 - 0s - loss: 0.0435 - mean_absolute_error: 0.1570 - val_loss: 0.0135 - val_mean_absolute_error: 0.1006 - 27ms/epoch - 9ms/step
Epoch 25/100
3/3 - 0s - loss: 0.0392 - mean_absolute_error: 0.1534 - val_loss: 0.0080 - val_mean_absolute_error: 0.0731 - 27ms/epoch - 9ms/step
Epoch 26/100
3/3 - 0s - loss: 0.0348 - mean_absolute_error: 0.1417 - val_loss: 0.0130 - val_mean_absolute_error: 0.0977 - 26ms/epoch - 9ms/step
Epoch 27/100
3/3 - 0s - loss: 0.0357 - mean_absolute_error: 0.1432 - val_loss: 0.0060 - val_mean_absolute_error: 0.0607 - 29ms/epoch - 10ms/step
Epoch 28/100
3/3 - 0s - loss: 0.0301 - mean_absolute_error: 0.1299 - val_loss: 0.0054 - val_mean_absolute_error: 0.0594 - 26ms/epoch - 9ms/step
Epoch 29/100
3/3 - 0s - loss: 0.0357 - mean_absolute_error: 0.1418 - val_loss: 0.0052 - val_mean_absolute_error: 0.0582 - 25ms/epoch - 8ms/step
Epoch 30/100
3/3 - 0s - loss: 0.0292 - mean_absolute_error: 0.1281 - val_loss: 0.0062 - val_mean_absolute_error: 0.0642 - 27ms/epoch - 9ms/step
Epoch 31/100
3/3 - 0s - loss: 0.0286 - mean_absolute_error: 0.1250 - val_loss: 0.0079 - val_mean_absolute_error: 0.0732 - 27ms/epoch - 9ms/step
Epoch 32/100
3/3 - 0s - loss: 0.0286 - mean_absolute_error: 0.1238 - val_loss: 0.0064 - val_mean_absolute_error: 0.0654 - 25ms/epoch - 8ms/step
Epoch 33/100
3/3 - 0s - loss: 0.0285 - mean_absolute_error: 0.1235 - val_loss: 0.0053 - val_mean_absolute_error: 0.0573 - 24ms/epoch - 8ms/step
Epoch 34/100
3/3 - 0s - loss: 0.0271 - mean_absolute_error: 0.1219 - val_loss: 0.0059 - val_mean_absolute_error: 0.0610 - 25ms/epoch - 8ms/step
Epoch 35/100
3/3 - 0s - loss: 0.0256 - mean_absolute_error: 0.1182 - val_loss: 0.0057 - val_mean_absolute_error: 0.0602 - 28ms/epoch - 9ms/step
Epoch 36/100
3/3 - 0s - loss: 0.0247 - mean_absolute_error: 0.1161 - val_loss: 0.0064 - val_mean_absolute_error: 0.0640 - 26ms/epoch - 9ms/step
Epoch 37/100
3/3 - 0s - loss: 0.0255 - mean_absolute_error: 0.1178 - val_loss: 0.0058 - val_mean_absolute_error: 0.0596 - 25ms/epoch - 8ms/step
Epoch 38/100
3/3 - 0s - loss: 0.0255 - mean_absolute_error: 0.1190 - val_loss: 0.0059 - val_mean_absolute_error: 0.0614 - 25ms/epoch - 8ms/step
Epoch 39/100
3/3 - 0s - loss: 0.0237 - mean_absolute_error: 0.1131 - val_loss: 0.0058 - val_mean_absolute_error: 0.0611 - 25ms/epoch - 8ms/step
Epoch 40/100
3/3 - 0s - loss: 0.0236 - mean_absolute_error: 0.1137 - val_loss: 0.0052 - val_mean_absolute_error: 0.0569 - 26ms/epoch - 9ms/step
Epoch 41/100
3/3 - 0s - loss: 0.0236 - mean_absolute_error: 0.1142 - val_loss: 0.0053 - val_mean_absolute_error: 0.0585 - 25ms/epoch - 8ms/step
Epoch 42/100
3/3 - 0s - loss: 0.0222 - mean_absolute_error: 0.1097 - val_loss: 0.0056 - val_mean_absolute_error: 0.0593 - 24ms/epoch - 8ms/step
Epoch 43/100
3/3 - 0s - loss: 0.0228 - mean_absolute_error: 0.1116 - val_loss: 0.0052 - val_mean_absolute_error: 0.0571 - 25ms/epoch - 8ms/step
Epoch 44/100
3/3 - 0s - loss: 0.0231 - mean_absolute_error: 0.1126 - val_loss: 0.0050 - val_mean_absolute_error: 0.0563 - 27ms/epoch - 9ms/step
Epoch 45/100
3/3 - 0s - loss: 0.0222 - mean_absolute_error: 0.1090 - val_loss: 0.0074 - val_mean_absolute_error: 0.0710 - 25ms/epoch - 8ms/step
Epoch 46/100
3/3 - 0s - loss: 0.0243 - mean_absolute_error: 0.1175 - val_loss: 0.0081 - val_mean_absolute_error: 0.0705 - 25ms/epoch - 8ms/step
Epoch 47/100
3/3 - 0s - loss: 0.0333 - mean_absolute_error: 0.1431 - val_loss: 0.0051 - val_mean_absolute_error: 0.0572 - 26ms/epoch - 9ms/step
Epoch 48/100
3/3 - 0s - loss: 0.0387 - mean_absolute_error: 0.1434 - val_loss: 0.0305 - val_mean_absolute_error: 0.1608 - 26ms/epoch - 9ms/step
Epoch 49/100
3/3 - 0s - loss: 0.0547 - mean_absolute_error: 0.1916 - val_loss: 0.0519 - val_mean_absolute_error: 0.2184 - 26ms/epoch - 9ms/step
Epoch 50/100
3/3 - 0s - loss: 0.0696 - mean_absolute_error: 0.2140 - val_loss: 0.0106 - val_mean_absolute_error: 0.0896 - 24ms/epoch - 8ms/step
Epoch 51/100
3/3 - 0s - loss: 0.0305 - mean_absolute_error: 0.1317 - val_loss: 0.0081 - val_mean_absolute_error: 0.0720 - 26ms/epoch - 9ms/step
Epoch 52/100
3/3 - 0s - loss: 0.0261 - mean_absolute_error: 0.1219 - val_loss: 0.0237 - val_mean_absolute_error: 0.1377 - 27ms/epoch - 9ms/step
Epoch 53/100
3/3 - 0s - loss: 0.0318 - mean_absolute_error: 0.1394 - val_loss: 0.0059 - val_mean_absolute_error: 0.0618 - 26ms/epoch - 9ms/step
Epoch 54/100
3/3 - 0s - loss: 0.0231 - mean_absolute_error: 0.1130 - val_loss: 0.0053 - val_mean_absolute_error: 0.0596 - 24ms/epoch - 8ms/step
Epoch 55/100
3/3 - 0s - loss: 0.0245 - mean_absolute_error: 0.1161 - val_loss: 0.0049 - val_mean_absolute_error: 0.0557 - 26ms/epoch - 9ms/step
Epoch 56/100
3/3 - 0s - loss: 0.0238 - mean_absolute_error: 0.1174 - val_loss: 0.0053 - val_mean_absolute_error: 0.0580 - 25ms/epoch - 8ms/step
Epoch 57/100
3/3 - 0s - loss: 0.0207 - mean_absolute_error: 0.1050 - val_loss: 0.0064 - val_mean_absolute_error: 0.0648 - 26ms/epoch - 9ms/step
#PREDICT THE VALUES USING THE MODEL
train_predictions = model.predict(X_train).flatten()
val_predictions = model.predict(X_val).flatten()
test_predictions = model.predict(X_test).flatten()

fig,axs = plt.subplots(3, 1, figsize=(14,14))

axs[0].plot(dates_train, train_predictions)
axs[0].plot(dates_train, y_train)
axs[0].legend(['Training Predictions', 'Training Observations'])

axs[1].plot(dates_val, val_predictions)
axs[1].plot(dates_val, y_val)
axs[1].legend(['Validation Predictions', 'Validation Observations'])

axs[2].plot(dates_test, test_predictions)
axs[2].plot(dates_test, y_test)
axs[2].legend(['Testing Predictions', 'Testing Observations'])

plt.savefig('../analysis/LTSM_recursive/'+coin_name +'_modelPredictions'+'.png')_____no_output_____

## Recursive Predictions
  [Back to TOC](#toc)_____no_output_____

from copy import deepcopy

#Get prediction for future dates recursively based on the previous existing information. Then update the window of days upon
#which the predictions are made

recursive_predictions = []
recursive_dates = np.concatenate([dates_test])

extra_dates = np.array(['2022-05-09', '2022-05-10', '2022-05-11'])
recursive_dates = np.append(recursive_dates,extra_dates)

last_window = deepcopy(X_train[-1])

for target_date in recursive_dates:
    next_prediction = model.predict(np.array([last_window])).flatten()
    recursive_predictions.append(next_prediction)
    last_window = np.insert(last_window,0,next_prediction)[:-1]
    _____no_output_____fig,axs = plt.subplots(2, 1, figsize=(14,10))

axs[0].plot(dates_train, train_predictions)
axs[0].plot(dates_train, y_train)
axs[0].plot(dates_val, val_predictions)
axs[0].plot(dates_val, y_val)
axs[0].plot(dates_test, test_predictions)
axs[0].plot(dates_test, y_test)
axs[0].plot(recursive_dates, recursive_predictions)
axs[0].legend(['Training Predictions',
            'Training Observations',
            'Validation Predictions',
            'Validation Observations',
            'Testing Predictions',
            'Testing Observations',
            'Recursive Predictions'])

axs[1].plot(dates_test, y_test)
axs[1].plot(recursive_dates, recursive_predictions)
axs[1].legend(['Testing Observations',
              'Recursive Predictions'])

plt.savefig('../analysis/LTSM_recursive/'+coin_name +'_recursivePredictions'+'.png')_____no_output_____may_10_prediction = coin_name +'-USD',recursive_predictions[-2][0]
may_10_prediction_____no_output_____


# Notebook from dprada/molsysmt
Path: docs/contents/demo/villin_hp35.ipynb

%load_ext autoreload
%autoreload 2_____no_output_____

# Villin subdomain HP-35_____no_output_____Chicken villin subdomain headpiece 35 (pdbid:1vii)_____no_output_____

import molsysmt as msmWarning: importing 'simtk.openmm' is deprecated.  Import 'openmm' instead.
msm.demo.villin_HP35.keys()_____no_output_____molsys = msm.convert(msm.demo.villin_HP35['solvated.msmpk'], 'molsysmt.MolSys')_____no_output_____msm.info(molsys)_____no_output_____

## References

[Piana, Stefano, Kresten Lindorff-Larsen, and David E. Shaw. “Protein Folding Kinetics and Thermodynamics from Atomistic Simulation.” Proceedings of the National Academy of Sciences 109, no. 44 (2012): 17845–50.](https://doi.org/10.1073/pnas.1201811109)

[Freddolino, Peter L., Christopher B. Harrison, Yanxin Liu, and Klaus Schulten. “Challenges in Protein-Folding Simulations.” Nature Physics 6, no. 10 (2010): 751–58.](https://doi.org/10.1038/nphys1713)

[Ensign, Daniel L., Peter M. Kasson, and Vijay S. Pande. “Heterogeneity Even at the Speed Limit of Folding: Large-Scale Molecular Dynamics Study of a Fast-Folding Variant of the Villin Headpiece.” Journal of Molecular Biology 374, no. 3 (2007): 806–16.](https://doi.org/10.1016/j.jmb.2007.09.069)

[Mori, Toshifumi, and Shinji Saito. “Molecular Mechanism Behind the Fast Folding/Unfolding Transitions of Villin Headpiece Subdomain: Hierarchy and Heterogeneity.” The Journal of Physical Chemistry B 120, no. 45 (2016): 11683–91.](https://doi.org/10.1021/acs.jpcb.6b08066)

[Bunagan, Michelle R., Jianmin Gao, Jeffery W. Kelly, and Feng Gai. “Probing the Folding Transition State Structure of the Villin Headpiece Subdomain via Side Chain and Backbone Mutagenesis.” Journal of the American Chemical Society 131, no. 21 (2009): 7470–76.](https://doi.org/10.1021/ja901860f)

[Meng, Wenli, Bing Shan, Yuefeng Tang, and Daniel P. Raleigh. “Native like Structure in the Unfolded State of the Villin Headpiece Helical Subdomain, an Ultrafast Folding Protein.” Protein Science 18, no. 8 (2009): 1692–1701.](https://doi.org/10.1002/pro.152)

[Wang, Ercheng, Peng Tao, Jun Wang, and Yi Xiao. “A Novel Folding Pathway of the Villin Headpiece Subdomain HP35.” Physical Chemistry Chemical Physics 21, no. 33 (2019): 18219–26.](https://doi.org/10.1039/c9cp01703h)

[Huang, Jing, Sarah Rauscher, Grzegorz Nawrocki, Ting Ran, Michael Feig, Bert L de Groot, Helmut Grubmüller, and Alexander D MacKerell. “CHARMM36m: An Improved Force Field for Folded and Intrinsically Disordered Proteins.” Nature Methods 14, no. 1 (2017): 71–73.](https://doi.org/10.1038/nmeth.4067)

[Eastman, Peter, Mark S. Friedrichs, John D. Chodera, Randall J. Radmer, Christopher M. Bruns, Joy P. Ku, Kyle A. Beauchamp, et al. “OpenMM 4: A Reusable, Extensible, Hardware Independent Library for High Performance Molecular Simulation.” Journal of Chemical Theory and Computation 9, no. 1 (2013): 461–69.](https://doi.org/10.1021/ct300857j)

[Harvey, M J, G Giupponi, and G De Fabritiis. “ACEMD: Accelerating Biomolecular Dynamics in the Microsecond Time Scale.” Journal of Chemical Theory and Computation 5, no. 6 (2009): 1632–39.](https://doi.org/10.1021/ct9000685)_____no_output_____

protein folding
molecular dynamics
biology