# Notebook from mohsen-gis/test
Path: beginner-lessons/cyberinfrastructure/cyberinfrastructure-2.ipynb

# Thank you for helping our study

Continue with the lesson

Throughout this lesson you will see reminders, like the one below, to ensure that all participants understand that they are in a voluntary research study.

### Reminder

By continuing with this lesson you are granting your permission to take part in this research study for the Hour of Cyberinfrastructure: Developing Cyber Literacy for GIScience project. In this study, you will be learning about cyberinfrastructure and related concepts using a web-based platform that will take approximately one hour per lesson. Participation in this study is voluntary.

Participants in this research must be 18 years or older. If you are under the age of 18 then please exit this webpage or navigate to another website such as the Hour of Code at https://hourofcode.com, which is designed for K-12 students.

If you are not interested in participating please exit the browser or navigate to this website: http://www.umn.edu. Your participation is voluntary and you are free to stop the lesson at any time.

For the full description please navigate to this website: Gateway Lesson Research Study Permission.

_____no_output_____

# This code cell starts the necessary setup for Hour of CI lesson notebooks.
# First, it enables users to hide and unhide code by producing a 'Toggle raw code' button below.
# Second, it imports the hourofci package, which is necessary for lessons and interactive Jupyter Widgets.
# Third, it helps hide/control other aspects of Jupyter Notebooks to improve the user experience
# This is an initialization cell
# It is not displayed because the Slide Type is 'Skip'

from IPython.display import HTML, IFrame, Javascript, display
from ipywidgets import interactive
import ipywidgets as widgets
from ipywidgets import Layout

import getpass # This library allows us to get the username (User agent string)

# import package for hourofci project
import sys
sys.path.append('../../supplementary') # relative path (may change depending on the location of the lesson notebook)
import hourofci

# load javascript to initialize/hide cells, get user agent string, and hide output indicator
# hide code by introducing a toggle button "Toggle raw code"
HTML('''

    <input id="toggle_code" type="button" value="Toggle raw code">
''')_____no_output_____

# Introduction and History

In this section we will cover the history of computation and computing and why the need for the word cyberinfrastructure arose.

- For a long time (thousands of years) people talked about computation and computers.
- Then in 2003, the National Science Foundation decided that a new word was needed to talk about the infrastructure that was used to support the creation of knowledge: **cyberinfrastructure**.

_____no_output_____## Why?

To understand what happened, let’s look a bit at the history of computation and computers._____no_output_____## Early History of Computation

During the time of the Roman empire (about 2,000 years ago), calculating where stones thrown by a catapult would land was an important mathematical problem.

Here is an example of a catapult, used by Rome and by countries that invaded Rome.

        Image credit: Wikimedia

_____no_output_____## Run some simulations of your own!

- To get a sense for the number of parameters you need to consider to estimate the distance a boulder (payload ) is thrown by a catapult, you can run a simulation of a catapult.
- Try it a few times!
_____no_output_____

IFrame("supplementary/catapult.html", width=984, height=750)_____no_output_____

### An early “computation device”

Archimedes was an important mathematician who made many discoveries.

He lived from 287 to 212 BC.

He was valuable to the army of Carthage (which was at war with Rome during Archimedes lifetime) because he was very good at calculating where stones thrown by a catapult would land. This was a calculation that had essentially one number as output: how far the stone would go.

        Image credit: (Wikimedia)

_____no_output_____## Let's figure out how quickly Archimedes could calculate the distance a boulder is going to fly
Once a catapult is built, there are really just two parameters you can adjust that impact the distance:
  1. Mass of the projectile
  2. Amount of tension

These two parameters are called **inputs** that will determine the **output**, which is distance a boulder will fly.
_____no_output_____## Let's figure out how quickly Archimedes could calculate the distance a boulder is going to fly

Inputs and outputs are simply **data** that Archimedes can use and produce as a human calculator to calculate distance. Let's get a little more precise in how we represent this data. Computers are based on the binary number system, which means they use 0's and 1's. One **bit** is either a 0 or a 1. One **byte** is 8 bits.

| Binary Number | Decimal Number |
|------|--------|
|00000000 |   0 |
|00000001 |   1 |
|00000010 |   2 |
|00001010 |  10 |
|00010000 |  16 |
|11111111 | 255 |

Two bytes (or 16 bits) can store numbers up to 65,536. Four bytes (or 32 bits) can store numbers up to 4,294,967,295! See more about binary numbers [here](https://en.wikipedia.org/wiki/Binary_number). If we can store more numbers, then we can be more precise.

_____no_output_____## How do bits relate to precision?

Let's take a look at the following image. If Archimedes is trying to hit the target that is in the firing range, then he needs to have small enough units to be able to communicate the location. If he has only 1 bit, then he can either fire the maximum distance (1) or half the maximum distance (0). If he has 4 bits, then he has 16 different locations that can be calculated. This is similar to the difference between measuring distance using kilometers versus meters. If the target is 1,400 meters away, then 1km is too short, but 2 km is too far.

_____no_output_____## Let's figure out how quickly Archimedes could calculate the distance a boulder is going to fly

Let’s figure that maybe people were really precise with their measurements and the two input values might take four bytes each, and the one output value might take four bytes as well...

So that means that we have:
  * 2 * 4 bytes of **input**
  * 1 * 4 bytes of **output**

We can use these to figure out the **Input/Output Rate** or **I/O Rate**, which is how quickly we can accept input, run the calculations, and produce output. What do you think the "I/O rate" of Archimedes might have been?

_____no_output_____### Well, we don't really know ....

because we don’t know how quickly Archimedes could run his calculations. Perhaps he approximated his results. He was, after all, trying to help soldiers crush other soldiers with rocks; he wasn’t doing brain surgery.

But let’s figure maybe 1 calculation in 5 minutes and 4 Bytes of I/O per minute, tops, was the I/O rate for Archimedes as a human calculator. If that is the case it would take him two minutes to get the input location from a soldier (2 * 4 bytes), five minutes to run his calculations, and another minute to communicate the distance to the catapult launcher (1 * 4 bytes). So his I/O rate would be approximately 8 minutes to input, calculate, and output 12 bytes._____no_output_____Now let's compare Archimedes to others types of calculating machines..._____no_output_____## Later there were “calculating machines”

- These are mechanical devices that performed calculations
- The Chinese Abacus was very practical and skilled people using it were very fast
- The Arithmometer, manufactured and sold in 1851, was the first commercially successful calculating machine for office use

        Image credit: [Wikimedia](https://commons.wikimedia.org/wiki/File:Abacus_6.png)

        Image credit: [Wikipedia](https://en.wikipedia.org/wiki/Arithmometer)

_____no_output_____### Wait a moment...!

You're about to see a virtual Chinese abacus with a hexadecimal numeral system. Hexadecimal numeral system is simply a numeral system that has 16 base digits (instead of 10 for decimal).
The hexadecimal base digits are 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, a, b, c, d, e, f which an "f" is equivalent to 15 in decimal system (see here for more details).

The abacus has an upper deck with two beads, each is worth 5, and the bottom has five, each worth 1.
_____no_output_____## Try a virtual Abacus simulator and see how fast you can make it go!_____no_output_____

# Auto-run
IFrame("supplementary/abacus.html", width=600, height=475)_____no_output_____

> In 2012, Naofumi Ogasawara, a 22-year-old abacus instructor from Japan completed the task of calculating 10 sums of 10 10-digit numbers each in three minutes and 11 seconds. ( [Recordholders.org](http://www.recordholders.org/en/events/worldcup/2012/results.html); [theguardian.com](https://www.theguardian.com/science/alexs-adventures-in-numberland/2012/oct/10/mental-calculation-world-cup) )

### What is the calculation rate and I/O rate that the winner achieved?

- The input was 100 integers each of which could be represented by 2 bytes (200 bytes total)
- The output was 10 integers each of which could be represented by 2 bytes (20 bytes total)
- The number of seconds was 191
- Each sum of 10 integers took 9 additions, so there were a total of 900 mathematical **operations**
- The total calculation rate was approximately (900 operations / 191 seconds) or ~4.7 operations per second
- So the total I/O rate was something like 220 bytes/191 seconds or between 1 and 2 bytes per second

_____no_output_____## The first fully electronic computer

The Z3, invented by Konrad Zuse, in Berlin, Germany in 1941.

It could accept any program and took about 1 second per addition and 3 seconds per multiplication of a 22 bit number. Faster than an abacus but not much!

We’re not quite sure what the I/O rates were - but input was with a keyboard and output with lights, so … pretty slow.

Image credit: [Wikimedia](https://upload.wikimedia.org/wikipedia/commons/4/4c/Z3_Deutsches_Museum.JPG)

_____no_output_____## Supercomputers!

In the 1970s and 1980s there were lots of different labels for computers. Mainframe computers, minicomputers, workstations. The label “supercomputers” was invented for several reasons:
  - This word distinguished the most powerful computers on earth from "ordinary" computers
  - It sounds cool

###  What makes a supercomputer "super"?
_____no_output_____##  What makes a supercomputer "super"?
  - There is no fixed and agreed on definition. The general idea is that a supercomputer is one of the most powerful computers around
  - One of the general characteristics of supercomputers is that they break computational problems up into many parts and work on those problems in **parallel** – many different processors each analyzing a part of a problem
  - If you have a computer and it costs more than \$1,000,000 and you want to call it “super” then go ahead!

*Side note:* To use supercomputers effectively to solve problems, it is important to learn about **parallel computing** so make sure to check out the beginner lesson on parallel computing if you are interested!
_____no_output_____## The First Supercomputer

The first supercomputer which was both “super” and called a supercomputer was the Control Data 6600, designed by Seymour Cray. The first system was delivered to a commercial customer in 1964.

This supercomputer could do 3,000 calculations per second.

I/O Rate. Because of the way the system was designed, it’s tricky to calculate an input rate. But output was with a teletype, so output was no more than about 10 characters per second.

Image credit: [Wikipedia](https://upload.wikimedia.org/wikipedia/commons/c/c4/CDC_6600.jc.jpg)

_____no_output_____## Today’s fastest supercomputer

The fastest (unclassified) supercomputer in the world as of Summer 2020 is called Summit, at Oak Ridge National Labs in the US.

It has achieved a calculation speed of 148,600,000,000,000 calculations per second. (Calculations are measured now in “FLOPS” - floating point operations per second - and that’s 148 PetaFLOPS. Or, in round numbers, really fast.

I/O rate is 100 GigaBytes per second from an external network (that’s 1,000,000,000 Bytes per second). Local output - to a local file system - is 2.5 TByte / second, or 2,500,000,000,000 Bytes per second.

Image credit: [Wikipedia](https://en.wikipedia.org/wiki/Summit_(supercomputer)#/media/File:Summit_(supercomputer).jpg)

_____no_output_____## Supercomputers help people do cool stuff

        Many discoveries were made with supercomputers, including:

Calculate the mass of subatomic particles

Simulate how suns form

Simulate how tornadoes form

Solving the four color problem. This is a mathematical problem that is hundreds of years old. The problem is this: prove that any map can be colored with four colors and no two adjoining countries on the map will be colored the same. Supercomputers were used to solve this problem in the mid 1970s. See the illustration

Image credit: https://commons.wikimedia.org/wiki/File:Four_Colour_Map_Example.svg

_____no_output_____## Can you solve the four color problem?_____no_output_____

IFrame("supplementary/fourcolor.html", width="970", height="730")_____no_output_____

## Let's look back: what was the evolution of I/O rates of computing devices?
In early computing devices, up to early supercomputers, the ratio of Calculation / IO was very high. This is called the Compute / Bandwidth ratio now.

| Device | Year | Calculation Rate (measured in Floating Point Operations per Second) | I/O Rate |
| :-: | :-: | :-: | :-: |
| Archimedes | 0 | <1 | 1 Byte / Minute
| Abacus | 1000 | 5 | 1 Byte / Second
| Z3 | 1941 | ⅓ to 1 operation per second | Not sure - but slow
| Control Data 6600 | 1943 | 3,000 | 10 Bytes / second
| Summit - the fastest supercomputer in the world | 2018 | 148,600,000,000,000  | In, across a network: 1,000,000,000 Bytes / second
Out, locally to a file system: 2,500,000,000,000 Bytes / second
_____no_output_____

## You now understand three very important concepts
1. The early history of computation
2. I/O Rate, the speed at which data is input and output
3. Calculation Rate (now measured in Floating Point Operations per Second or FLOP)_____no_output_____## Next

Let's learn about cyberinfrastructure and how even supercomputers are enough for science.

Click here to move to the next section to learn more!_____no_output_____


# Notebook from csavur/biosignalsnotebooks
Path: notebookToHtml/biosignalsnotebooks_html_publish/Categories/Connect/pairing_device.ipynb

_____no_output_____<link rel="stylesheet" href="../../styles/theme_style.css">

<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">

         Pairing a Device at Windows 10 [biosignalsplux]

_____no_output_____

        Difficulty Level:

                    Tags
                    connect☁pairing☁biosignalsplux☁bluetooth☁windows10

_____no_output_____For acquiring physiological data there are three systems in interaction: human body (system under exploration); signal acquisition system/sensors (system responsible for collecting data from the human body) and computer/mobile device (system that receives and processes the data collected by the acquisition system).

PLUX's acquisition systems (biosignalsplux  and BITalino ) valorize wireless communication between the acquisition system and computer (using Bluetooth  technology).

The current Jupyter Notebook is intended to explain how PLUX's acquisition systems (biosignalsplux in our example) can be quickly connected to a computer in order to ensure future real-time acquisitions through OpenSignals .

_____no_output_____
0 - Before starting an acquisition it is mandatory that your PLUX acquisition system (in our case biosignalsplux) is paired with the computer. It will be described, in the following steps, relevant instructions to fulfill this prerequisite !

The example is focused on Microsoft Windows 10 Users. For Mac users there are some relevant pages explaining how a Bluetooth device, like biosignalsplux can be paired.

 https://support.apple.com/en-us/HT201171

 https://support.apple.com/guide/mac-help/connect-a-bluetooth-device-blth1004/mac

*It is possible to use the internal Bluetooth of your computer, however to improve the connectivity and have more freedom of movement and avoid communication losses, we strongly recommend to use a Bluetooth Dongle like the one available at PLUX Store ._____no_output_____
1 - Turn on your biosignalsplux device
_____no_output__________no_output_____
2 - Navigate until the Bluetooth communication system window

This action is achievable through the "Start Menu" icon [bottom-left corner of the screen ] >> Settings  >> Devices  >> Bluetooth & other devices _____no_output__________no_output_____
3 - Enable Bluetooth communication system in your computer
_____no_output__________no_output_____
4 - Click on "Add Bluetooth or other device"

We are reaching the pairing stage between our computer and biosignalsplux device_____no_output__________no_output_____
5 - Click on "Bluetooth" option and wait for the list of available devices
_____no_output__________no_output_____
6 - Find and click on the entry labeled with "biosignalsplux"
_____no_output__________no_output_____
7 - Type the predefined password >> "123"
_____no_output__________no_output_____Mission almost accomplished ! Now you should have your biosignalsplux device ready to use in OpenSignals  software._____no_output__________no_output_____
8 - Execute OpenSignals application in order to get access to the main page
_____no_output__________no_output_____
9 - Press the "Device Configuration" icon for enabling your device
_____no_output__________no_output_____
10 - Check which is your device mac-address (at the back of your device)
_____no_output__________no_output_____
11 - Find your device on the "Device Configuration" page

You should press the "Search" icon in order to the previously added device (through Windows interface) can appear_____no_output__________no_output_____For finishing the pairing between biosignalsplux and the computer, it is only necessary to click on the device box. When a set of configurable options appear the pairing process is officially completed !_____no_output__________no_output_____The previous instructions represents a reasonable interactive guide for establishing a connection between PLUX's devices and your computer.

If you follow these steps, now you are connected with biosignalsplux, which can be the start of an amazing journey !_____no_output_____We hope that you have enjoyed this guide. biosignalsnotebooks is an environment in continuous expansion, so don't stop your journey and learn more with the remaining Notebooks  !_____no_output_____

            ☌ Project Presentation

            ☌ GitHub Repository

            ☌ How to install biosignalsnotebooks Python package ?

            ☌ Signal Library

            ☌ Notebook Categories

            ☌ Notebooks by Difficulty

            ☌ Notebooks by Signal Type

            ☌ Notebooks by Tag

_____no_output_____

from biosignalsnotebooks.__notebook_support__ import css_style_apply
css_style_apply()_____no_output_____%%html
_____no_output_____


# Notebook from salman394/AI-ml--course
Path: jupyter_english/tutorials/learn_regex_the_easy_way_aditya_soni.ipynb

## Open Machine Learning Course

The Author of the material: Aditya Soni, the nickname in the ODS @ecdrid. This notebook serves as a very short glimpse from this [website](https://www.regular-expressions.info) primarily._____no_output_____#  A Tutorial On Understanding ([Rr]ege)(x|xp|xes|xps|xen)_____no_output_____

A common workflow with regular expressions is that you write a pattern for the thing you are looking for..._____no_output_____# Let's see our very first expression

- **<\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b>**  It's a complex pattern as it includes lots of things like
    - Character Class
    - Alphabets
    - Percentage
    - Numbers
    - Underscores
    - $\{\}$, word Boundaries etc...

> In short  - **this pattern describes an email address; With the above regex pattern, we can search through a text file to find email addresses, or verify if a given string looks like an email address..**

The most basic regex pattern in a token like just an $$ i.e a single literal character. In the string **" Zebra is an animal."**, this will match the very first $b$ in the **Ze$b$** Note that it doesn't matter whether it's present in the middle of the word as of now..

Now let me introduce few very basics things used in $<regex>$ to define itself (remeber the e-mail address pattern above, now we will break into piece by piece..)

In the regex discussed in this tutorial, there are **11** characters with special meanings:
the opening square bracket $<[>$, the backslash, the caret <^>, the dollar sign <$>, the period or dot <.>, the
vertical bar or pipe symbol <|>, the question mark <?>, the asterisk or star <*>, the plus sign <+>, the opening
round bracket <(> and the closing round bracket <)>. These special characters are often called **“metacharacters”**.

|Meta character|Description|
|:----:|----|
|**.**|Period matches any single character except a line break.|
|**[ ]**|Character class. Matches any character contained between the square brackets.|
|**[^ ]**|Negated character class. Matches any character that is not contained between the square brackets .|
|*****|Matches 0 or more repetitions of the preceding symbol.|
|**+**|Matches 1 or more repetitions of the preceding symbol.|
|**?**|Makes the preceding symbol optional.|
|**{n,m}**|Braces. Matches at least "n" but not more than "m" repetitions of the preceding symbol.|
|**(xyz)**|Character group. Matches the characters xyz in that exact order.|
|**|**|Alternation. Matches either the characters before or the characters after the symbol.|
|**\**|Escapes the next character. This allows you to match reserved characters `[ ] ( ) { } . * + ? ^ $ \`.|
|**^**|Matches the beginning of the input.|
|**$**|Matches the end of the input.|

Read More [here](http://www.rexegg.com/regex-quickstart.html#chars) and [here](http://www.greenend.org.uk/rjk/tech/regexp.html). Both are Very Very Good...

- Example  - If you want to use any of these characters as a literal in a regex, you need to escape them with a backslash. If
you want to match **<1+1=2>**, the correct regex is $1\+1=2$. Otherwise, the plus sign will have a special meaning. **Note** that **<1+1=2>**, with the *backslash omitted*, is a **valid** regex. So you will **not** get an error message. But it
will not match **<1+1=2>**. _____no_output_____# The Regex-Directed Engine Always Returns the Left-most Match
This is a very important point to understand: a regex-directed engine will always return the leftmost match,
even if a **better** match could be found later. When applying a regex to a string, the engine will start at the
first character of the string. It will try all possible permutations of the regular expression at the first character.
Only if all possibilities have been tried and found to fail, will the engine continue with the second character in
the text. Again, it will try all possible permutations of the regex, in exactly the same order. The result is that
the regex-directed engine will return the leftmost match.
- When applying **<cat\>** to **He captured a catfish for his cat.**, the engine will try to match the first
token in the regex **<c\>** to the first character in the match **H**. This fails. There are no other possible
permutations of this regex, because it merely consists of a sequence of literal characters. So the regex engine
tries to match the **<c\>** with the **e**. This fails too, as does matching the **c** with the space. Arriving at the 4th
character in the match, **<c\>** matches **c**. The engine will then try to match the second token **** to the 5th
character, **a**. This succeeds too. But then, **<t\>** fails to match **p**. At that point, the engine knows the regex
cannot be matched starting at the 4th character in the match. So it will continue with the 5th: **a**. Again, **<c\>**
fails to match here and the engine carries on. At the 15th character in the match, **<c\>** again matches **c**. The
engine then proceeds to attempt to match the remainder of the regex at character 15 and finds that ****
matches **a** and **<t\>** matches **t**.

- The entire regular expression could be matched starting at character 15. The engine is **"eager"** to report a
match. **It will therefore report the first three letters of catfish as a valid match**. The engine **never** proceeds
beyond this point to see if there are any **better** matches. The *first match* is considered good enough. _____no_output_____# Regex's Fundamentals_____no_output_____## Character Sets/Classes_____no_output_____Character sets are also called character class. **Square brackets** are used to specify character sets. Use a **hyphen** inside a character set to specify the characters' range. The order of the character range inside square brackets doesn't matter. For example, the regular expression `[Tt]he` means: `an uppercase T or lowercase t, followed by the letter h, followed by the letter e.`

- **<[Tt]he>** => The car parked in the garage.

A period inside a character set, however, means a literal period. The regular expression **<ar[.]>** means: a lowercase character a, followed by letter r, followed by a period **.** character.

- **<ar[.]>** => A garage is a good place to park a car.

- **<[0-9]>** => Matches a **single digit between 0 and 9**. You can use more than one range.
- **<[0-9a-fA-F]>** => Matches a **single hexadecimal digit**, case insensitively.
- You can combine ranges and single characters. **<[0-9a-fxA-FX]>** matches a hexadecimal digit or the letter X.* Again, the order of the characters and the ranges does not matter.*
- Find a word, even if it is misspelled, such as **<sep[ae]r[ae]te>** or **
**. _____no_output_____### Negated Character Sets/Classes_____no_output_____Typing a **caret(^)** after the opening square bracket will negate the character class. **The result is that the character
class will match any character that is not  in the character class.**
- It is important to remember that a negated character class **still must match a character**. **<q[^u]>** does not
mean: ** a q not followed by a u **. It means: **<font color= red a q followed by a character that is not a u **. It will **not** match the
$q$ in the string $Iraq$. It will match the $q$ and $the space$ after the $q$ in **Iraq is a country**._____no_output_____###  Shorthand Character Sets

Regular expression provides **shorthands** for the commonly used character sets,
which offer **convenient shorthands** for commonly used regular expressions. The
shorthand character sets are as follows:

|Shorthand|Description|
|:----:|----|
|.|Any character except new line. It's the most commonly misused metacharacter.|
|\w|Matches alphanumeric characters: `[a-zA-Z0-9_]`|
|\W|Matches non-alphanumeric characters: `[^\w]`|
|\d|Matches digit: `[0-9]`|
|\D|Matches non-digit: `[^\d]`|
|\s|Matches whitespace character: `[\t\n\f\r\p{Z}]`|
|\S|Matches non-whitespace character: `[^\s]`|_____no_output_____## Repetitions

Following meta characters `+`, `*` or `?` are used to specify how many times a
subpattern can occur. These meta characters act differently in different
situations._____no_output_____### The Star *

The symbol `*` matches zero or more repetitions of the preceding matcher. The
regular expression `a*` means: zero or more repetitions of preceding lowercase
character `a`. But if it appears after a character set or class then it finds
the repetitions of the whole character set.
For example, the regular expression
- `[a-z]*` means: any number of lowercase letters in a row.

The `*` symbol can be used with the meta character `.` to match any string of
characters `.*`. The `*` symbol can be used with the whitespace character `\s`
to match a string of whitespace characters. For example, the expression
`\s*cat\s*` means: zero or more spaces, followed by lowercase character `c`,
followed by lowercase character `a`, followed by lowercase character `t`,
followed by zero or more spaces._____no_output_____### The Plus +

The symbol `+` matches one or more repetitions of the preceding character. For
example, the regular expression `c.+t` means: lowercase letter `c`, followed by
at least one character, followed by the lowercase character `t`. It needs to be
clarified that `t` is the last `t` in the sentence.

- **<c.+t>** => The fat  cat sat on the mat._____no_output_____### The Question Mark ?
In regular expression the meta character `?` makes the preceding character optional. This symbol matches zero or one instance of the preceding character. For example, the regular expression `[T]?he` means: `Optional the uppercase letter T, followed by the lowercase character h, followed by the lowercase character e.`

- **<[Tt]he>** => The car parked in the garage._____no_output_____### The Lazy Star *?

Repeats the previous item zero or more times. Lazy, so the engine first attempts to skip the
previous item, before trying permutations with ever increasing matches of the preceding
item. _____no_output_____|Regex|Means|
|:----:|----|
|abc+|        matches a string that has ab followed by one or more c|
|abc?|       matches a string that has ab followed by zero or one c|
|abc{2}|      matches a string that has ab followed by 2 c|
|abc{2,}|    matches a string that has ab followed by 2 or more c|
|abc{2,5}|    matches a string that has ab followed by 2 up to 5 c|
|a(bc)\*|     matches a string that has a followed by zero or more copies of the sequence bc|
|a(bc){2,5}|  matches a string that has a followed by 2 up to 5 copies of the sequence bc|
|**<.+>**| matches `<div>simple div</div>`|_____no_output_____## Full stop or Period or dot **.**_____no_output_____In regular expressions, the dot or period is one of the most commonly used metacharacters. Unfortunately, it
is also the most commonly misused metacharacter. The dot is short for the negated character class **<[^\n]>** (UNIX regex flavors) or
**<[^\r\n]>** (Windows regex flavors).

  Use The Dot Sparingly
- The dot is a **very powerful** regex metacharacter. It allows you to be **lazy**. `Put in a dot, and everything will
match just fine when you test the regex on valid data. The problem is that the regex will also match in cases
where it should not match..`

Example - Let’s say we want to match a date in `mm/dd/yy` format, but we
want to leave the user the choice of date separators. The quick solution is **<\d\d.\d\d.\d\d>**. Seems fine at
first sight.. It will match a date like `02/12/03` just what we intended, So fine...
-   Trouble is: 02512703 is also considered a **valid date** by this regular expression. In this match, the first dot matched $5$, and the second matched $7$. Obviously $not$ what we intended. _____no_output_____## Start of String and End of String Anchors ( $ and ^)_____no_output_____Anchors are a different breed. They do not match any character at all. Instead, they match a position before,
after or between characters. They can be used to `anchor` the regex match at a certain position.
- The caret **<^>**
matches the position before the first character in the string. Applying **<^a>** to `abc` matches `a`. **<^b>** will
not match `abc` at all, because the **** cannot be matched right after the start of the string, matched by **<^>**.
- Similarly, **<\$>** matches right after the last character in the string. **<c\$>** matches `c` in `abc`, while **** `does not` match `abc` at all...._____no_output_____ **So Now we are good to go!! Armed with regex, let's see what they can do..** _____no_output_____## Word Boundaries_____no_output_____The word boundary \b matches positions where one side is a word character (usually a letter, digit or underscore—but see below for variations across engines) and the other side is not a word character (for instance, it may be the beginning of the string or a space character).

The `\bcat\b` would therefore match `cat` in a `black cat`, but it wouldn't match it in `catatonic`, `tomcat` or `certificate`. Removing one of the boundaries, `\bcat` would match `cat` in `catfish`, and `cat\b` would match `cat` in `tomcat`, but not vice-versa. Both, of course, would match `cat` on its own.

Word boundaries are useful when you want to match a sequence of letters (or digits) on their own, or to ensure that they occur at the beginning or the end of a sequence of characters.

Be aware, though, that `\bcat\b` will not match `cat` in `_cat` or in `cat25` because there is no boundary between an underscore and a letter, nor between a letter and a digit: these all belong to what regex defines as word characters._____no_output_____## Grouping ()_____no_output_____By placing part of a regular expression inside round brackets or parentheses, you can group that part of the
regular expression together. This allows you to apply a regex operator, e.g. a repetition operator, to the entire
group.
Only round brackets can be used for grouping. Square brackets define a character class, and curly
braces are used by a special repetition operator.

- The regex `Set(Value)?` matches "**Set** or **SetValue**".
    - In the first case, the first backreference will be
empty, because it did not match anything.
    - In the second case, the `first backreference` will contain **Value**._____no_output_____### So How can we use it?_____no_output_____`Backreferences allow you to re-use part of the regex match. You can reuse it inside the regular expression before or afterwards depending on the Regex Flavour you are using...`
Some regex flavours use `\`, some flavours use `$`, etc..
In Perl, you can use the magic variables $1, $2, etc. to access the part of the string matched by the backreference_____no_output_____**Regex** : `(\w+)\1` on String `seek` will match `ee`

PS I am myself studying this section properly, hence couldn't add more details :))_____no_output_____# Examples_____no_output_____- Regex are now written in Quotes (\`)
- The String to be matched is in ("bold")

Suppose you want to use a regex to match a list of function names in a programming language: "**Get, GetValue, Set or SetValue.**"
- The obvious solution is `Get|GetValue|Set|SetValue`

*Now take a look closer carefully at the regex and the string, both.
Here are some other ways to do the same task*
-  `Get(Value)?|Set(Value)?`
-  `\b(Get|GetValue|Set|SetValue)\b`
-  `\b(Get(Value)?|Set(Value)?)\b`
- Even this one is correct `\b(Get|Set)(Value)?\b`

**Regex**:	`<[^>]+>`
- **What it does**:	This finds any HTML, such as `<\a>, <\b>, <\img />, <\br />, etc`. You can use this to find segments that have HTML tags you need to deal with, or to remove all HTML tags from a text.

**Regex**:	`https?:\/\/[\w\.\/\-?=&%,]+`
- What it does:	This will find a URL. It will capture most URLs that begin with http:// or https://.

**Regex**:	`'\w+?'`
- **What it does**:	This finds single words that are surrounded by apostrophes.

**Regex**:	`([-A-Za-z0-9_]*?([-A-Za-z_][0-9]|[0-9][-A-Za-z_])[-A-Za-z0-9_]*)`
- **What it does**:	Alphanumeric part numbers and references like: 1111_A, AA1AAA or 1-1-1-A, 21A1 and 10UC10P-BACW, abcd-1234, 1234-pqtJK, sft-0021 or 21-1_AB and 55A or AK7_GY.
This can be very useful if you are translating documents that have a lot of alphanumeric codes or references in them, and you need to be able to find them easily.

**Regex**:	`\b(the|The)\b.*?\b(?=\W?\b(is|are|was|can|shall| must|that|which|about|by|at|if|when|should|among|above|under|$)\b)`
- **What it does**:	This finds text that begins with the or The and ends with stop words such as is, are, was, can, shall, must, that, which, about, by, at, if, when, should, among, above or under, or the end of the segment.
This is particularly useful when you need to extract terminology. Suppose you have segments like these:
`
The Web based look up is our new feature. A project manager should not proofread... Our Product Name is...`
    - The Regex shown above would find anything between The and is, or should. With most texts, there is a good chance that anything this Regex finds is a good term that you can add to your Termbase.

**Regex**:	`\b(a|an|A|An)\b.*?\b(?=\W?\b(is|are|was|can|shall|must |that|which|about|by|at|if|when|among|above|under|$)\b)`
- **What it does**:	This works much like the Regex shown above, except that it finds text that begins with a or an, rather than the. This can also be very helpful when you need to extract terminology from a project.

**Regex**:  `\b(this|these|This|These)\b.*?\b(?=\W?\b(is|are|was|can|shall|must|that|which|about|by|at|if|when|among|above|under|$)\b)`
    - **What it does**:	This works much like the Regex shown above, except that it finds text that begins with this or these. This can also be very helpful when you need to extract terminology from a project.

**Regex** :`(.*?)`
- **What it does** : Accept blah-blah-blah..._____no_output_____## Python [re module](https://docs.python.org/3/library/re.html)

- `re.sub(regex, replacement, subject)` performs a search-and-replace across subject, replacing all
matches of regex in subject with replacement. The result is returned by the sub() function. **The subject
string you pass is not modified**. The re.sub() function applies the same backslash logic to the replacement text as is applied to the regular expression. Therefore, you should use raw strings for the replacement text..._____no_output_____

%load_ext autoreload
%autoreload
import re
import time_____no_output_____s = "How do you do this"
print(
    "After applying re.sub -- ",
    re.sub(r"How do you", "How do I", s),
    "\nOriginal Text is still -- ",
    s,
)_____no_output_____

So does that mens that we have to type one regex expression everytime, run and check it and then the substituion willl happen? i.e Can't we stack `re.sub(), re.sub(), re.sub()....`

Surely not, Remeber `re.sub()` is returning a string after making the chnages that matched the pattern you asked more.._____no_output_____

s_old = "How do you do this"
print("After applying re.sub -- ", end="")
s_new = re.sub(r"How do you", "How do I", s_old)
print(f"\nOriginal Text isn't still **{s_old}** but it's now **{s_new}**")
# Obviously s_old and s_new are different, I am just trying to show that we can stack the operations...._____no_output_____tweet = "#fingerprint #Pregnancy Test https://goo.gl/h1MfQV #android +#apps +#beautiful \
         #cute #health #igers #iphoneonly #iphonesia #iphone \
             <3 ;D :( :-("

# Let's take care of emojis and the #(hash-tags)...

print(f"Original Tweet ---- \n {tweet}")

## Replacing #hashtag with only hashtag
tweet = re.sub(r"#(\S+)", r" \1 ", tweet)
# this gets a bit technical as here we are using Backreferencing and Character Sets Shorthands and replacing the captured Group.
# \S = [^\s] Matches any charachter that isn't white space
print(f"\n Tweet after replacing hashtags ----\n  {tweet}")

## Love -- <3, :*
tweet = re.sub(r"(<3|:\*)", " EMO_POS ", tweet)
print(f"\n Tweet after replacing Emojis for Love with EMP_POS ----\n  {tweet}")

# The parentheses are for Grouping, so we search (remeber the raw string (`r`))
# either for <3 or(|) :\* (as * is a meta character, so preceeded by the backslash)

## Wink -- ;-), ;), ;-D, ;D, (;,  (-;
tweet = re.sub(r"(;-?\)|;-?D|\(-?;)", " EMO_POS ", tweet)
print(f"\n Tweet after replacing Emojis for Wink with EMP_POS ----\n  {tweet}")

# The parentheses are for Grouping as usual, then we first focus on `;-), ;),`, so we can see that 1st we need to have a ;
# and then we can either have a `-` or nothing, so we can do this via using our `?` clubbed with `;` and hence we have the very
# starting with `(;-?\)` and simarly for others...

## Sad -- :-(, : (, :(, ):, )-:
tweet = re.sub(r"(:\s?\(|:-\(|\)\s?:|\)-:)", " EMO_NEG ", tweet)
print(f"\n Tweet after replacing Emojis for Sad with EMP_NEG ----\n  {tweet}")_____no_output_____##See the Output Carefully, there are Spaces inbetween un-necessary...
## Replace multiple spaces with a single space
tweet = re.sub(r"\s+", " ", tweet)
print(f"\n Tweet after replacing xtra spaces ----\n  {tweet}")

##Replace the Puctuations (+,;)
tweet = re.sub(r"[^\w\s]", "", tweet)
print(f"\n Tweet after replacing Punctuation + with PUNC ----\n  {tweet}")_____no_output_____# bags of positive/negative smiles (You can extend the above example to take care of these few too...))) A good Excercise...

positive_emojis = set(
    [
        ":‑)",
        ":)",
        ":-]",
        ":]",
        ":-3",
        ":3",
        ":->",
        ":>",
        "8-)",
        "8)",
        ":-}",
        ":}",
        ":o)",
        ":c)",
        ":^)",
        "=]",
        "=)",
        ":‑D",
        ":D",
        "8‑D",
        "8D",
        "x‑D",
        "xD",
        "X‑D",
        "XD",
        "=D",
        "=3",
        "B^D",
        ":-))",
        ";‑)",
        ";)",
        "*-)",
        "*)",
        ";‑]",
        ";]",
        ";^)",
        ":‑,",
        ";D",
        ":‑P",
        ":P",
        "X‑P",
        "XP",
        "x‑p",
        "xp",
        ":‑p",
        ":p",
        ":‑Þ",
        ":Þ",
        ":‑þ",
        ":þ",
        ":‑b",
        ":b",
        "d:",
        "=p",
        ">:P",
        ":'‑)",
        ":')",
        ":-*",
        ":*",
        ":×",
    ]
)
negative_emojis = set(
    [
        ":‑(",
        ":(",
        ":‑c",
        ":c",
        ":‑<",
        ":<",
        ":‑[",
        ":[",
        ":-||",
        ">:[",
        ":{",
        ":@",
        ">:(",
        "D‑':",
        "D:<",
        "D:",
        "D8",
        "D;",
        "D=",
        "DX",
        ":‑/",
        ":/",
        ":‑.",
        ">:\\",
        ">:/",
        ":\\",
        "=/",
        "=\\",
        ":L",
        "=L",
        ":S",
        ":‑|",
        ":|",
        "|‑O",
        "<:‑|",
    ]
)_____no_output_____## Pattern to match any IP Addresses
pattern = r"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b"_____no_output_____

the above pattern will also match `999.999.999.999` but that isn't a valid IP at all
Now this depends on the data at hand as to how far you want the regex to be accurate...
To restrict all `4` numbers in the IP address to `0..255`, you can use this
complex beast:
- `\b(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-
9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-
4][0-9]|[01]?[0-9][0-9]?)\b`_____no_output_____

updated_pattern = r"\b(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b"
updated_pattern_____no_output_____if re.search(pattern, "999.999.999.999"):
    print("Matched")
if re.search(updated_pattern, "256.999.999.999"):
    print("Matched")
else:
    print("Not Matched")_____no_output_____# Valid Dates..
pattern = r"(19|20)\d\d[- /.](0[1-9]|1[012])[- /.](0[1-9]|[12][0-9]|3[01])"_____no_output_____

- matches a date in yyyy-mm-dd format from between 1900-01-01 and 2099-12-31, with a choice of four separators(space included :))

- The year is matched by `(19|20)\d\d`
- The month is matched by `(0[1-9]|1[012])` (rounding brackets are necessary so that to include both the options)
    - By using character classes,
        - the first option matches a number between `01 and 09`, and
        - the second matches `10, 11 or 12`
- The last part of the regex consists of three options. The first matches the numbers `01
through 09`, the second `10 through 29`, and the third matches `30 or 31`... _____no_output_____# References (A lot)

- http://www.rexegg.com/ <<<< THE ULTIMATE WEBSITE>>>>
- https://github.com/aloisdg/awesome-regex
- http://linuxreviews.org/beginner/tao_of_regular_expressions/tao_of_regular_expressions.en.print.pdf
- https://developers.google.com/edu/python/regular-expressions
- https://www.youtube.com/watch?v=EkluES9Rvak_____no_output_____ Oh, yes, and forget about practice, that's completely overrated.
      Just kidding....   _____no_output__________no_output_____


# Notebook from pablocarderam/HAL_analysis
Path: HAL Compound Clustering.ipynb

# HAL Compound Rescue Analysis and Dendrogram_____no_output_____By Pablo Cárdenas, pcarden@mit.edu

For Nakashima, et al.

### Objectives:
- Establish whether there were differences between compounds with rescue and non-rescue phenotypes for protein HAL knock-down in _Plasmodium falciparum_.
- Create phylogenetic dendrogram of HAL sequences.
_____no_output_____## Contents:
* [0. Setup](#setup)
* [1. Datasets: chemometrics](#datasets)
* [2. Unsupervised clustering by chemometrics](#clustering)
* [3. Phylogenetic dendrogram](#phylogenetics)_____no_output_____## 0. Setup _____no_output_____### 0.1 Imports_____no_output_____

# General purpose data handling and graphing
import random
import copy as cp
import numpy as np
import pandas as pd
from pylab import *
from matplotlib import pyplot as plt
from matplotlib.colors import ListedColormap
import seaborn as sns

# Clustering
import scipy.cluster as clu
import scipy.spatial as spa
import sklearn.decomposition as dec
import sklearn.cluster as skc
import sklearn.ensemble as ske
import sklearn.model_selection as skms
import sklearn.metrics as skm
import sklearn.preprocessing as skp
import sklearn.tree as skt
import scipy.cluster.hierarchy as hc_____no_output_____

### 0.2 Plots_____no_output_____

matplotlib.rcParams.update({'font.size': 12}) # bigger text_____no_output_____

## 1. Datasets: chemometrics and structure _____no_output_____### 1.1 Names_____no_output_____

# Relates compound numbers to names.
# Analysis was originally performed with compound names blinded.
compound_names = {
    1: 'Atorvastatin',
    2: 'Simvastatin',
    3: 'Propranolol',
    4: 'Betaxolol',
    5: 'Alprenolol',
    6: 'Practolol',
    7: 'Salbutamol',
    8: 'Isoproteranol',
    9: 'Chloroquine',
    10: 'Mefloquine',
    11: 'Clotrimazole',
    12: 'Lumefantrine',
    13: 'Piperaquine',
    14: 'Bilirubin',
    15: 'Alpha-tocopherol',
    16: 'Probucol',
    17: 'Ascorbate',
    18: 'Dehydroascorbate',
    19: 'Chlorpromazine',
    20: 'Clomipramine',
    21: 'Chembridge SMM_1 (ID: 19635129)',
    22: 'Chembridge SMM_2 (ID: 48011879)',
    23: 'Chembridge_SMM_3 (ID: 60165744)',
    24: 'Chembridge_SMM_4 (ID: 6965313)',
    25: 'ACMP',
    26: 'NITD609',
    27: 'Quercitin',
    28: 'Norflurazon'
}
# pd.DataFrame(pd.Series(compound_names))_____no_output_____

### 1.2 Chemometrics and phenotype_____no_output_____

# Import chemometric data and rescue phenotype data

dat_cm = pd.read_csv('Supplementary table - HAL_Rescue_Compounds.csv') # read data
dat_cm.index = dat_cm['Compound'] # preserve IDs
dat_cm = dat_cm.drop(columns='Compound')
dat_cm = dat_cm.transpose()

# Data labels: rescue phenotype
dat_lab = dat_cm['Rescue']
dat_cm = dat_cm.drop(columns='Rescue')

# Log-transform these columns to make them more informative,
# given their highly skewed distribution
dat_cm['Lipid accumulation  ratio (LAR)'] \
    = np.log(dat_cm['Lipid accumulation  ratio (LAR)'])
dat_cm['Vacuolar accumulation  ratio (VAR)']\
    = np.log(dat_cm['Vacuolar accumulation  ratio (VAR)'])

dat_cm = dat_cm.drop(columns='log D7.4 (ChemAxon)')
    # turns out this is the same as log LAR, so we drop it

dat_cm_scaled = pd.DataFrame(
    skp.scale( dat_cm,axis=0 ),
    index=dat_cm.index, columns=dat_cm.columns
)
    # we rescale along time points, each protein independently_____no_output_____

## 2. Unsupervised clustering by chemometrics  _____no_output_____### 2.1 Dimensionality reduction: PCA_____no_output_____

pca = dec.PCA() # create PCA object
pca.fit(dat_cm_scaled) # fit to data, only numeric data
dat_pca = pca.transform(dat_cm_scaled) # transform it to matrix_____no_output_____print('explained variance ratio (first components): %s'
      % str(np.sum(pca.explained_variance_ratio_[0:2])))explained variance ratio (first components): 0.7622937710352167
# Explained variance plot

fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.set_xlabel('Principal component')
ax.set_ylabel('Explained variance')

sns.barplot(np.arange(1,dat_pca.shape[1]+1),pca.explained_variance_ratio_)
plt.tight_layout()
fig.savefig("PCA_Explained_variance.png", dpi=300)_____no_output_____# Loadings

ld = pca.components_
fig = plt.figure(figsize = (7,6))
ax = fig.add_subplot(1,1,1)
plt.scatter(ld[0,:],ld[1,:], alpha=0.6, label="Loadings", color='purple');
ax.set_xlabel('Principal Component 1')
ax.set_ylabel('Principal Component 2')
for i in range(ld.shape[1]): # show text for all
    if i%50==0:
        plt.text(ld[0,i]+0.01,ld[1,i]+0.003,dat_cm_scaled.columns[i],fontsize=10,alpha=0.6)
    else:
        plt.text(ld[0,i]+0.01,ld[1,i]-0.02,dat_cm_scaled.columns[i],fontsize=10,alpha=0.6)

plt.tight_layout()
fig.savefig("Loadings_12.png", dpi=300)
fig.savefig("Loadings_12.eps", dpi=300)The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
# PC plot by phenotype

fig = plt.figure(figsize = (5,5))
ax = fig.add_subplot(1,1,1)
colors = sns.color_palette('colorblind', 2)
    # get a color palette of right length
types = ['Rescue','No rescue']
ax.set_xlabel('Principal Component 1')
ax.set_ylabel('Principal Component 2')

for lab, color in zip(dat_lab.unique(),colors):
        # plots first one group of points with one color, then the other
    plt.scatter(dat_pca[dat_lab == lab, 0], dat_pca[dat_lab == lab, 1],
                color=color, alpha=.8, lw=2,label=types[int(lab==0)])

for i in range(len(dat_cm_scaled.index)): # show text for all
    plt.text(dat_pca[i,0]-0.2,dat_pca[i,1]-0.3,
             dat_cm_scaled.index[i],fontsize=10,alpha=0.6)

plt.legend(loc='best', shadow=False, scatterpoints=1)

plt.tight_layout()
fig.savefig("PCA_by_phenotype.png", dpi=300)
fig.savefig("PCA_by_phenotype.eps", dpi=300)The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.
The PostScript backend does not support transparency; partially transparent artists will be rendered opaque.

### 2.2 Hierarchical clustering_____no_output_____

'''
Plots clustermaps
'''
def heatmapClusters(
    dat,labels,method='average',metric='euclidean',
    file_name="",col_palette=""):

    # Plot
    dat = dat.transpose()
        # we put the features along rows of heatmap to
        # get compound dendrogram on top
    dat.columns = [
        str(i+1)+': '+compound_names[i+1] for i in range(len(compound_names))
    ] # change index to names
    if col_palette == "": # default behavior for colors
        col_palette = sns.color_palette('colorblind', len(labels.unique()))
        # get a color palette of right length

    lut = dict(zip(labels.unique(), col_palette))
        # associate each cluster with a color
    row_colors = pd.Series(labels).map(lut)
        # mark each clade with a color
    row_colors.index = dat.columns
        # allows correct colorbar mapping
    g = sns.clustermap(
        dat, col_colors=row_colors,method=method,metric=metric,
        cbar_kws={'label': 'Scaled value'},
        cmap=sns.cubehelix_palette(start=-1, rot=1, as_cmap=True, reverse=True),
        figsize=(12,10)
    ) # show it
    g.ax_heatmap.set_xticklabels(g.ax_heatmap.get_xticklabels(),
                                 rotation=45, horizontalalignment='right')

    if len(file_name) > 0:
        plt.tight_layout()
        g.savefig(file_name+".eps", dpi=300)

    return g_____no_output_____# Best clustermap, with rescue phenotype color bar

fig = plt.figure()
g = heatmapClusters(dat_cm_scaled,dat_lab,
                    method='complete',metric='cityblock')
g.savefig("Compounds_heatmap_phenotypes.png", dpi=300)
g.savefig("Compounds_heatmap_phenotypes.eps", dpi=300)_____no_output_____

### 2.3 Examine individual features_____no_output_____

figure = plt.figure(figsize=(10,6))

dat_melt = pd.DataFrame(
    dat_pca, columns=[i+1 for i in range(dat_pca.shape[1])],
    index=dat_lab.index
)
dat_melt['Rescue'] = dat_lab
dat_melt = dat_melt.melt(id_vars=['Rescue'])
dat_melt.columns = ['Rescue','Principal Component','Value']
dat_melt.loc[ dat_melt['Rescue']==1, 'Rescue' ] = 'Rescue'
dat_melt.loc[ dat_melt['Rescue']==0, 'Rescue' ] = 'No rescue'

ax = sns.violinplot(x='Principal Component', y='Value', hue='Rescue',
                    data=dat_melt, palette="muted", split=True)
# ax.set_xticklabels(ax.get_xticklabels(), rotation=45,
#                    horizontalalignment='right')

figure.tight_layout()
figure.savefig("PC_Comp.png", dpi=300)
# figure.savefig("PC_Comp.eps", dpi=300)_____no_output_____

## 3. Phylogenetic dendrogram _____no_output_____

dat = pd.read_csv('27 lipocalins percentage matrix.csv') # read data
dat.head()_____no_output_____dat.columns = ['a','b','organism'] + list(dat['BBP Pieris brassicae'])

dat.index = list(dat['organism'])
dat = dat.drop(columns=['a','b','organism'])
dat = 100-dat

dat.head()_____no_output_____linkage = hc.linkage(spa.distance.squareform(dat), method='weighted',optimal_ordering=True)
g = sns.clustermap(dat, row_linkage=linkage, col_linkage=linkage,
                   cmap=sns.cubehelix_palette(start=.5, rot=-.75, as_cmap=True, reverse=True),
                   cbar_kws={'label': '% Distance'}) # hierarchical clustering
g.savefig("MFP_Dendrogram.eps", dpi=300)_____no_output_____


# Notebook from chenlongzhen/eat_tensorflow2_in_30_days
Path: README.ipynb

# How to eat TensorFlow2 in 30 days ?🔥🔥

Click here for [Chinese Version（中文版）](#30天吃掉那只-tensorflow2)

🚀 github repository: https://github.com/lyhue1991/eat_tensorflow2_in_30_days

### 1. TensorFlow2 🍎 or Pytorch🔥

Conclusion first:

**For the engineers, priority goes to TensorFlow2.**

**For the students and researchers，first choice should be Pytorch.**

**The best way is to master both of them if having sufficient time.**_____no_output_____Reasons:

* 1. **Model implementation is the most important in the industry. Deployment supporting tensorflow models (not Pytorch) exclusively is the present situation in the majority of the Internet enterprises in China.** What's more, the industry prefers the models with higher availability; in most cases, they use well-validated modeling architectures with the minimized requirements of adjustment._____no_output_____* 2. **Fast iterative development and publication is the most important for the researchers since they need to test a lot of new models. Pytorch has advantages in accessing and debugging comparing with TensorFlow2.** Pytorch is most frequently used in academy since 2019 with a large amount of the cutting-edge results._____no_output_____* 3. Overall, TensorFlow2 and Pytorch are quite similar in programming nowadays, so mastering one helps learning the other. Mastering both framework provides you a lot more open-sourced models and helps you switching between them._____no_output_____### 2. Keras🍏 and tf.keras 🍎

Conclusion first:

**Keras will be discontinued in development after version 2.3.0, so use tf.keras.**_____no_output_____Keras is a high-level API for the deep learning frameworks. It help the users to define and training DL networks with a more intuitive way.

The Keras libraries installed by pip implement this high-level API for the backends in tensorflow, theano, CNTK, etc.

tf.keras is the high-level API just for Tensorflow, which is based on low-level APIs in Tensorflow.

Most but not all of the functions in tf.keras are the same for those in Keras (which is compatible to many kinds of backend). tf.keras has a tighter combination to TensorFlow comparing to Keras.

With the acquisition by Google, Keras will not update after version 2.3.0 , thus the users should use tf.keras from now on, instead of using Keras installed by pip._____no_output_____### 3. What Should You Know Before Reading This Book 📖?

**It is suggested that the readers have foundamental knowledges of machine/deep learning and experience of modeling using Keras or TensorFlow 1.0.**

**For those who have zero experience of machine/deep learning, it is strongly suggested to refer to ["Deep Learning with Python"](https://www.amazon.com/Deep-Learning-Python-Francois-Chollet/dp/1617294438/ref=sr_1_1?dchild=1&keywords=Deep+Learning+with+Python&qid=1586194568&sr=8-1) along with reading this book.**_____no_output_____["Deep Learning with Python"](https://www.amazon.com/Deep-Learning-Python-Francois-Chollet/dp/1617294438/ref=sr_1_1?dchild=1&keywords=Deep+Learning+with+Python&qid=1586194568&sr=8-1) is written by François Chollet, the inventor of Keras. This book is based on Keras and has no machine learning related prerequisites to the reader.

"Deep Learning with Python" is easy to understand as it uses various examples to demonstrate. **No mathematical equation is in this book since it focuses on cultivating the intuitive to the deep learning.**
_____no_output_____### 4. Writing Style 🍉 of This Book_____no_output_____**This is a introduction reference book which is extremely friendly to human being. The lowest goal of the authors is to avoid giving up due to the difficulties, while "Don't let the readers think" is the highest target.**

This book is mainly based on the official documents of TensorFlow together with its functions.

However, the authors made a thorough restructuring and a lot optimizations on the demonstrations.

It is different from the official documents, which is disordered and contains both tutorial and guidance with lack of systematic logic, that our book redesigns the content according to the difficulties, readers' searching habits, and the architecture of TensorFlow. We now make it progressive for TensorFlow studying with a clear path, and an easy access to the corresponding examples.

In contrast to the verbose demonstrating code, the authors of this book try to minimize the length of the examples to make it easy for reading and implementation. What's more, most of the code cells can be used in your project instantaneously.

**Given the level of difficulty as 9 for learning Tensorflow through official documents, it would be reduced to 3 if learning through this book.**

This difference in difficulties could be demonstrated as the following figure:

![](./data/30天吃掉那个TF2.0_en.jpg)
_____no_output_____### 5. How to Learn With This Book ⏰

**(1) Study Plan**

The authors wrote this book using the spare time, especially the two-month unexpected "holiday" of COVID-19. Most readers should be able to completely master all the content within 30 days.

Time required everyday would be between 30 minutes to 2 hours.

This book could also be used as library examples to consult when implementing machine learning projects with TensorFlow2.

**Click the blue captions to enter the corresponding chapter.**_____no_output_____|Date |Contents                                                       | Difficulties   | Est. Time | Update Status|
|----:|:--------------------------------------------------------------|-----------:|----------:|-----:|
| |[**Chapter 1: Modeling Procedure of TensorFlow**](./english/Chapter1.md)    |⭐   |   0hour   |✅    |
|Day 1 |  [1-1 Example: Modeling Procedure for Structured Data](./english/Chapter1-1.md)    | ⭐⭐⭐ |   1hour    |✅    |
|Day 2 |[1-2 Example: Modeling Procedure for Images](./english/Chapter1-2.md)    | ⭐⭐⭐⭐  |   2hours    |✅    |
|Day 3 |  [1-3 Example: Modeling Procedure for Texts](./english/Chapter1-3.md)   | ⭐⭐⭐⭐⭐  |   2hours    |✅    |
|Day 4 |  [1-4 Example: Modeling Procedure for Temporal Sequences](./english/Chapter1-4.md)   | ⭐⭐⭐⭐⭐  |   2hours    |✅    |
|     |[**Chapter 2: Key Concepts of TensorFlow**](./english/Chapter2.md)  | ⭐  |  0hour |✅  |
|Day 5 |  [2-1 Data Structure of Tensor](./english/Chapter2-1.md)  | ⭐⭐⭐⭐   |   1hour    |✅    |
|Day 6 |  [2-2 Three Types of Graph](./english/Chapter2-2.md)  | ⭐⭐⭐⭐⭐   |   2hours    |✅    |
|Day 7 |  [2-3 Automatic Differentiate](./english/Chapter2-3.md)  | ⭐⭐⭐   |   1hour    |✅    |
|  |[**Chapter 3: Hierarchy of TensorFlow**](./english/Chapter3.md) |   ⭐  |  0hour   |✅  |
|Day 8 |  [3-1 Low-level API: Demonstration](./english/Chapter3-1.md)   | ⭐⭐⭐⭐ |   1hour    |✅   |
|Day 9 |  [3-2 Mid-level API: Demonstration](./english/Chapter3-2.md)   | ⭐⭐⭐   |   1hour    |✅  |
|Day 10 |  [3-3 High-level API: Demonstration](./english/Chapter3-3.md)  | ⭐⭐⭐   |   1hour    |✅  |
|  |[**Chapter 4: Low-level API in TensorFlow**](./english/Chapter4.md) |⭐    | 0hour|✅  |
|Day 11|  [4-1 Structural Operations of the Tensor](./english/Chapter4-1.md)  | ⭐⭐⭐⭐⭐   |   2hours    |✅   |
|Day 12|  [4-2 Mathematical Operations of the Tensor](./english/Chapter4-2.md)   | ⭐⭐⭐⭐   |   1hour    |✅  |
|Day 13|  [4-3 Rules of Using the AutoGraph](./english/Chapter4-3.md)| ⭐⭐⭐   |   0.5hour    | ✅  |
|Day 14|  [4-4 Mechanisms of the AutoGraph](./english/Chapter4-4.md)    | ⭐⭐⭐⭐⭐   |   2hours    |✅  |
|Day 15|  [4-5 AutoGraph and tf.Module](./english/Chapter4-5.md)  | ⭐⭐⭐⭐   |   1hour    |✅  |
|  |[**Chapter 5: Mid-level API in TensorFlow**](./english/Chapter5.md) |  ⭐  | 0hour|✅ |
|Day 16|  [5-1 Dataset](./english/Chapter5-1.md)   | ⭐⭐⭐⭐⭐   |   2hours    |✅  |
|Day 17|  [5-2 feature_column](./english/Chapter5-2.md)   | ⭐⭐⭐⭐   |   1hour    |✅  |
|Day 18|  [5-3 activation](./english/Chapter5-3.md)    | ⭐⭐⭐   |   0.5hour    |✅   |
|Day 19|  [5-4 layers](./english/Chapter5-4.md)  | ⭐⭐⭐   |   1hour    |✅  |
|Day 20|  [5-5 losses](./english/Chapter5-5.md)    | ⭐⭐⭐   |   1hour    |✅  |
|Day 21|  [5-6 metrics](./english/Chapter5-6.md)    | ⭐⭐⭐   |   1hour    |✅   |
|Day 22|  [5-7 optimizers](./english/Chapter5-7.md)    | ⭐⭐⭐   |   0.5hour    |✅   |
|Day 23|  [5-8 callbacks](./english/Chapter5-8.md)   | ⭐⭐⭐⭐   |   1hour    |✅   |
|  |[**Chapter 6: High-level API in TensorFlow**](./english/Chapter6.md)|    ⭐ | 0hour|✅  |
|Day 24|  [6-1 Three Ways of Modeling](./english/Chapter6-1.md)   | ⭐⭐⭐   |   1hour    |✅ |
|Day 25|  [6-2 Three Ways of Training](./english/Chapter6-2.md)  | ⭐⭐⭐⭐   |   1hour    |✅   |
|Day 26|  [6-3 Model Training Using Single GPU](./english/Chapter6-3.md)    | ⭐⭐   |   0.5hour    |✅   |
|Day 27|  [6-4 Model Training Using Multiple GPUs](./english/Chapter6-4.md)    | ⭐⭐   |   0.5hour    |✅  |
|Day 28|  [6-5 Model Training Using TPU](./english/Chapter6-5.md)   | ⭐⭐   |   0.5hour    |✅  |
|Day 29| [6-6 Model Deploying Using tensorflow-serving](./english/Chapter6-6.md) | ⭐⭐⭐⭐| 1hour |✅   |
|Day 30| [6-7 Call Tensorflow Model Using spark-scala](./english/Chapter6-7.md) | ⭐⭐⭐⭐⭐|2hours|✅  |
| | [Epilogue: A Story Between a Foodie and Cuisine](./english/Epilogue.md) | ⭐|0hour|✅  |_____no_output_____**(2) Software environment for studying**_____no_output_____All the source codes are tested in jupyter. It is suggested to clone the repository to local machine and run them in jupyter for an interactive learning experience.

The authors would suggest to install jupytext that converts markdown files into ipynb, so the readers would be able to open markdown files in jupyter directly._____no_output_____

#For the readers in mainland China, using gitee will allow cloning with a faster speed
#!git clone https://gitee.com/Python_Ai_Road/eat_tensorflow2_in_30_days

#It is suggested to install jupytext that converts and run markdown files as ipynb.
#!pip install -i https://pypi.tuna.tsinghua.edu.cn/simple -U jupytext

#It is also suggested to install the latest version of TensorFlow to test the demonstrating code in this book
#!pip install -i https://pypi.tuna.tsinghua.edu.cn/simple  -U tensorflow_____no_output_____import tensorflow as tf

#Note: all the codes are tested under TensorFlow 2.1
tf.print("tensorflow version:",tf.__version__)

a = tf.constant("hello")
b = tf.constant("tensorflow2")
c = tf.strings.join([a,b]," ")
tf.print(c)_____no_output_____

```
tensorflow version: 2.1.0
hello tensorflow2
```_____no_output_____### 6. Contact and support the author 🎈🎈_____no_output_____**If you find this book helpful and want to support the author, please give a star ⭐ to this repository and don't forget to share it to your friends 😊**

Please leave comments in the WeChat official account "Python与算法之美" (Elegance of Python and Algorithms) if you want to communicate with the author about the content. The author will try best to reply given the limited time available.

![image.png](./data/Python与算法之美logo.jpg)_____no_output_____# 30天吃掉那只 TensorFlow2

📚 gitbook电子书地址： https://lyhue1991.github.io/eat_tensorflow2_in_30_days

🚀 github项目地址：https://github.com/lyhue1991/eat_tensorflow2_in_30_days

🐳 kesci专栏地址：https://www.kesci.com/home/column/5d8ef3c3037db3002d3aa3a0_____no_output_____### 一，TensorFlow2 🍎 or Pytorch🔥

先说结论:

**如果是工程师，应该优先选TensorFlow2.**

**如果是学生或者研究人员，应该优先选择Pytorch.**

**如果时间足够，最好TensorFlow2和Pytorch都要学习掌握。**_____no_output_____理由如下：

* 1，**在工业界最重要的是模型落地，目前国内的大部分互联网企业只支持TensorFlow模型的在线部署，不支持Pytorch。** 并且工业界更加注重的是模型的高可用性，许多时候使用的都是成熟的模型架构，调试需求并不大。_____no_output_____* 2，**研究人员最重要的是快速迭代发表文章，需要尝试一些较新的模型架构。而Pytorch在易用性上相比TensorFlow2有一些优势，更加方便调试。** 并且在2019年以来在学术界占领了大半壁江山，能够找到的相应最新研究成果更多。_____no_output_____* 3，TensorFlow2和Pytorch实际上整体风格已经非常相似了，学会了其中一个，学习另外一个将比较容易。两种框架都掌握的话，能够参考的开源模型案例更多，并且可以方便地在两种框架之间切换。_____no_output_____### 二，Keras🍏 and  tf.keras 🍎

先说结论：

**Keras库在2.3.0版本后将不再更新，用户应该使用tf.keras。**_____no_output_____Keras可以看成是一种深度学习框架的高阶接口规范，它帮助用户以更简洁的形式定义和训练深度学习网络。

使用pip安装的Keras库同时在tensorflow,theano,CNTK等后端基础上进行了这种高阶接口规范的实现。

而tf.keras是在TensorFlow中以TensorFlow低阶API为基础实现的这种高阶接口，它是Tensorflow的一个子模块。

tf.keras绝大部分功能和兼容多种后端的Keras库用法完全一样，但并非全部，它和TensorFlow之间的结合更为紧密。

随着谷歌对Keras的收购，Keras库2.3.0版本后也将不再进行更新，用户应当使用tf.keras而不是使用pip安装的Keras._____no_output_____### 三，本书📖面向读者 👼_____no_output_____**本书假定读者有一定的机器学习和深度学习基础，使用过Keras或者Tensorflow1.0或者Pytorch搭建训练过模型。**

**对于没有任何机器学习和深度学习基础的同学，建议在学习本书时同步参考学习《Python深度学习》一书。**

《Python深度学习》这本书是Keras之父Francois Chollet所著，该书假定读者无任何机器学习知识，以Keras为工具，

使用丰富的范例示范深度学习的最佳实践，该书通俗易懂，**全书没有一个数学公式，注重培养读者的深度学习直觉。**。_____no_output_____### 四，本书写作风格 🍉_____no_output_____**本书是一本对人类用户极其友善的TensorFlow2.0入门工具书，不刻意恶心读者是本书的底限要求，Don't let me think是本书的最高追求。**

本书主要是在参考TensorFlow官方文档和函数doc文档基础上整理写成的。

但本书在篇章结构和范例选取上做了大量的优化。

不同于官方文档混乱的篇章结构，既有教程又有指南，缺少整体的编排逻辑。

本书按照内容难易程度、读者检索习惯和TensorFlow自身的层次结构设计内容，循序渐进，层次清晰，方便按照功能查找相应范例。

不同于官方文档冗长的范例代码，本书在范例设计上尽可能简约化和结构化，增强范例易读性和通用性，大部分代码片段在实践中可即取即用。

**如果说通过学习TensorFlow官方文档掌握TensorFlow2.0的难度大概是9的话，那么通过学习本书掌握TensorFlow2.0的难度应该大概是3.**

谨以下图对比一下TensorFlow官方教程与本教程的差异。

![](./data/30天吃掉那个TF2.0.jpg)
_____no_output_____### 五，本书学习方案 ⏰

**1，学习计划**

本书是作者利用工作之余和疫情放假期间大概2个月写成的，大部分读者应该在30天可以完全学会。

预计每天花费的学习时间在30分钟到2个小时之间。

当然，本书也非常适合作为TensorFlow的工具手册在工程落地时作为范例库参考。

**点击学习内容蓝色标题即可进入该章节。**_____no_output_____|日期 | 学习内容                                                       | 内容难度   | 预计学习时间 | 更新状态|
|----:|:--------------------------------------------------------------|-----------:|----------:|-----:|
| |[**一、TensorFlow的建模流程**](./一、TensorFlow的建模流程.md)    |⭐   |   0hour   |✅    |
|day1 |  [1-1,结构化数据建模流程范例](./1-1,结构化数据建模流程范例.md)    | ⭐⭐⭐ |   1hour    |✅    |
|day2 |[1-2,图片数据建模流程范例](./1-2,图片数据建模流程范例.md)    | ⭐⭐⭐⭐  |   2hour    |✅    |
|day3 |  [1-3,文本数据建模流程范例](./1-3,文本数据建模流程范例.md)   | ⭐⭐⭐⭐⭐  |   2hour    |✅    |
|day4 |  [1-4,时间序列数据建模流程范例](./1-4,时间序列数据建模流程范例.md)   | ⭐⭐⭐⭐⭐  |   2hour    |✅    |
|     |[**二、TensorFlow的核心概念**](./二、TensorFlow的核心概念.md)  | ⭐  |  0hour |✅  |
|day5 |  [2-1,张量数据结构](./2-1,张量数据结构.md)  | ⭐⭐⭐⭐   |   1hour    |✅    |
|day6 |  [2-2,三种计算图](./2-2,三种计算图.md)  | ⭐⭐⭐⭐⭐   |   2hour    |✅    |
|day7 |  [2-3,自动微分机制](./2-3,自动微分机制.md)  | ⭐⭐⭐   |   1hour    |✅    |
|  |[**三、TensorFlow的层次结构**](./三、TensorFlow的层次结构.md) |   ⭐  |  0hour   |✅  |
|day8 |  [3-1,低阶API示范](./3-1,低阶API示范.md)   | ⭐⭐⭐⭐   |   1hour    |✅   |
|day9 |  [3-2,中阶API示范](./3-2,中阶API示范.md)   | ⭐⭐⭐   |  1hour    |✅  |
|day10 |  [3-3,高阶API示范](./3-3,高阶API示范.md)  | ⭐⭐⭐  |   1hour    |✅  |
|  |[**四、TensorFlow的低阶API**](./四、TensorFlow的低阶API.md) |⭐    | 0hour|✅  |
|day11|  [4-1,张量的结构操作](./4-1,张量的结构操作.md)  | ⭐⭐⭐⭐⭐   |   2hour    |✅   |
|day12|  [4-2,张量的数学运算](./4-2,张量的数学运算.md)   | ⭐⭐⭐⭐   |   1hour    |✅  |
|day13|  [4-3,AutoGraph的使用规范](./4-3,AutoGraph的使用规范.md)| ⭐⭐⭐   |   0.5hour    |✅  |
|day14|  [4-4,AutoGraph的机制原理](./4-4,AutoGraph的机制原理.md)    | ⭐⭐⭐⭐⭐   |   2hour    |✅  |
|day15|  [4-5,AutoGraph和tf.Module](./4-5,AutoGraph和tf.Module.md)  | ⭐⭐⭐⭐   |   1hour    |✅  |
|  |[**五、TensorFlow的中阶API**](./五、TensorFlow的中阶API.md) |  ⭐  | 0hour|✅ |
|day16|  [5-1,数据管道Dataset](./5-1,数据管道Dataset.md)   | ⭐⭐⭐⭐⭐   |   2hour    |✅  |
|day17|  [5-2,特征列feature_column](./5-2,特征列feature_column.md)   | ⭐⭐⭐⭐   |   1hour    |✅  |
|day18|  [5-3,激活函数activation](./5-3,激活函数activation.md)    | ⭐⭐⭐   |   0.5hour    |✅   |
|day19|  [5-4,模型层layers](./5-4,模型层layers.md)  | ⭐⭐⭐   |   1hour    |✅  |
|day20|  [5-5,损失函数losses](./5-5,损失函数losses.md)    | ⭐⭐⭐   |   1hour    |✅  |
|day21|  [5-6,评估指标metrics](./5-6,评估指标metrics.md)    | ⭐⭐⭐   |   1hour    |✅   |
|day22|  [5-7,优化器optimizers](./5-7,优化器optimizers.md)    | ⭐⭐⭐   |   0.5hour    |✅   |
|day23|  [5-8,回调函数callbacks](./5-8,回调函数callbacks.md)   | ⭐⭐⭐⭐   |   1hour    |✅   |
|  |[**六、TensorFlow的高阶API**](./六、TensorFlow的高阶API.md)|    ⭐ | 0hour|✅  |
|day24|  [6-1,构建模型的3种方法](./6-1,构建模型的3种方法.md)   | ⭐⭐⭐   |   1hour    |✅ |
|day25|  [6-2,训练模型的3种方法](./6-2,训练模型的3种方法.md)  | ⭐⭐⭐⭐   |   1hour    |✅   |
|day26|  [6-3,使用单GPU训练模型](./6-3,使用单GPU训练模型.md)    | ⭐⭐   |   0.5hour    |✅   |
|day27|  [6-4,使用多GPU训练模型](./6-4,使用多GPU训练模型.md)    | ⭐⭐   |   0.5hour    |✅  |
|day28|  [6-5,使用TPU训练模型](./6-5,使用TPU训练模型.md)   | ⭐⭐   |   0.5hour    |✅  |
|day29| [6-6,使用tensorflow-serving部署模型](./6-6,使用tensorflow-serving部署模型.md) | ⭐⭐⭐⭐| 1hour |✅   |
|day30| [6-7,使用spark-scala调用tensorflow模型](./6-7,使用spark-scala调用tensorflow模型.md) | ⭐⭐⭐⭐⭐|2hour|✅  |
| | [后记：一个吃货和一道菜的故事](./后记：一个吃货和一道菜的故事.md) | ⭐|0hour|✅  |
_____no_output_____**2，学习环境**_____no_output_____本书全部源码在jupyter中编写测试通过，建议通过git克隆到本地，并在jupyter中交互式运行学习。

为了直接能够在jupyter中打开markdown文件，建议安装jupytext，将markdown转换成ipynb文件。

**此外，本项目也与和鲸社区达成了合作，可以在和鲸专栏fork本项目，并直接在云笔记本上运行代码，避免环境配置痛苦。**

🐳和鲸专栏地址：https://www.kesci.com/home/column/5d8ef3c3037db3002d3aa3a0_____no_output_____

#克隆本书源码到本地,使用码云镜像仓库国内下载速度更快
#!git clone https://gitee.com/Python_Ai_Road/eat_tensorflow2_in_30_days

#建议在jupyter notebook 上安装jupytext，以便能够将本书各章节markdown文件视作ipynb文件运行
#!pip install -i https://pypi.tuna.tsinghua.edu.cn/simple -U jupytext

#建议在jupyter notebook 上安装最新版本tensorflow 测试本书中的代码
#!pip install -i https://pypi.tuna.tsinghua.edu.cn/simple  -U tensorflow_____no_output_____import tensorflow as tf

#注：本书全部代码在tensorflow 2.1版本测试通过
tf.print("tensorflow version:",tf.__version__)

a = tf.constant("hello")
b = tf.constant("tensorflow2")
c = tf.strings.join([a,b]," ")
tf.print(c)_____no_output_____

```
tensorflow version: 2.1.0
hello tensorflow2
```

_____no_output_____### 六，鼓励和联系作者 🎈🎈_____no_output_____**如果本书对你有所帮助，想鼓励一下作者，记得给本项目加一颗星星star⭐，并分享给你的朋友们喔😊!**

如果对本书内容理解上有需要进一步和作者交流的地方，欢迎在公众号"Python与算法之美"下留言。作者时间和精力有限，会酌情予以回复。

也可以在公众号后台回复关键字：**加群**，加入读者交流群和大家讨论。

![image.png](./data/Python与算法之美logo.jpg)_____no_output_____