# Notebook from vvchauit/Data-Science
Path: Bash-Tutorial.ipynb

___Introduction to Unix Bash commands________no_output_____While I use Jupyter notebooks for illustration purposes, it is more common to directly use [terminal](https://en.wikipedia.org/wiki/Terminal_(macOS)). You can find it in `Others -> Terminal` or by spotlight search.

If you are also interested in using Bash in notebook, please checkout [takluyver/bash_kernel](https://github.com/takluyver/bash_kernel)_____no_output_____
_____no_output_____#### Introduction_____no_output_____Bash commands are no different from many other languages such as Java or Python. We can what we code. For example, we can print out the current working directory._____no_output_____

!which python /usr/local/bin/python
!which git/usr/bin/git
!which go_____no_output_____pwd_____no_output_____

We can also print out what are in the current working directories._____no_output_____

lssample_data/
%cd sample_data
!ls /content/sample_data
anscombe.json		      mnist_test.csv
california_housing_test.csv   mnist_train_small.csv
california_housing_train.csv  README.md
!cat README.mdThis directory includes a few sample datasets to get you started.

*   `california_housing_data*.csv` is California housing data from the 1990 US
    Census; more information is available at:
    https://developers.google.com/machine-learning/crash-course/california-housing-data-description

*   `mnist_*.csv` is a small sample of the
    [MNIST database](https://en.wikipedia.org/wiki/MNIST_database), which is
    described at: http://yann.lecun.com/exdb/mnist/

*   `anscombe.json` contains a copy of
    [Anscombe's quartet](https://en.wikipedia.org/wiki/Anscombe%27s_quartet); it
    was originally described in

    Anscombe, F. J. (1973). 'Graphs in Statistical Analysis'. American
    Statistician. 27 (1): 17-21. JSTOR 2682899.

    and our copy was prepared by the
    [vega_datasets library](https://github.com/altair-viz/vega_datasets/blob/4f67bdaad10f45e3549984e17e1b3088c731503d/vega_datasets/_data/anscombe.json).

Just as graphical user interfaces (GUIs), we can speak "bash language" to interact with our computers. In fact, they are more powerful. For example, you cannot use the [HPC](https://hpc.uiowa.edu/) systems until you know something about shell programming. Note that Bash is only one of the shell programs but probably the most popular one.

---_____no_output_____#### Working directories_____no_output_____Let's get started by the concept of ___working directory___. As its name suggests, working directory is where you work in, or just which folder/directory you are at right now. As the previous example shows, there is a ___program___ called `pwd` that can help us do such thing._____no_output_____

from google.colab import drive
drive.mount('/content/gdrive')Mounted at /content/gdrive
pwd_____no_output_____cd ../content
lsgdrive/  sample_data/
%cd /content/gdrive/My\ Drive//content/gdrive/My Drive
!pwd/content/gdrive/My Drive
!ls anh-o-phap
'Colab Notebooks'
 Courses
'Data-Driven Feature Characterization Techniques for Laser Printer Attribution.pdf'
 du-lich
 hinh-cuoi
'Identifying 3D Printer Residual Data via Open Source Documentation .pdf'
 IMG_2749.JPG
 IMG_2752.JPG
 IMG_2754.JPG
 IMG_2758.JPG
 IMG_2759.JPG
 IMG_2762.JPG
 IMG_2784.JPG
 IMG_2800.JPG
 IMG_2801.JPG
'IMG_2816 (1).JPG'
 IMG_2816.JPG
 IMG_2872.JPG
 IMG_2886.JPG
 IMG_2905.JPG
'IMG_2906 (1).JPG'
 IMG_2906.JPG
 IMG_2907.JPG
 IMG_2908.JPG
 IMG_2909.JPG
 IMG_2910.JPG
 IMG_2911.JPG
 IMG_2912.JPG
 IMG_2913.JPG
'IMG_2914 (1).JPG'
 IMG_2914.JPG
'IMG_2915 (1).JPG'
 IMG_2915.JPG
 IMG_2916.JPG
 IMG_2917.JPG
 IMG_2918.JPG
 IMG_2919.JPG
 IMG_2920.JPG
'IMG_2921 (1).JPG'
 IMG_2921.JPG
'IMG_2922 (1).JPG'
'IMG_2922 (2).JPG'
 IMG_2922.JPG
 IMG_2924.JPG
 IMG_2925.JPG
 IMG_2926.JPG
 IMG_2927.JPG
 IMG_2928.JPG
 IMG_2929.JPG
 IMG_2930.JPG
 IMG_2931.JPG
 IMG_2934.JPG
 IMG_2935.JPG
'IMG_2936 (1).JPG'
 IMG_2936.JPG
 IMG_2937.JPG
 IMG_2938.JPG
 IMG_2940.JPG
 IMG_2941.JPG
 IMG_2943.JPG
 IMG_2944.JPG
 IMG_2945.JPG
 IMG_2946.JPG
 IMG_2948.JPG
 IMG_2949.JPG
 IMG_2951.JPG
 IMG_2953.JPG
 IMG_2954.JPG
 IMG_2955.JPG
 IMG_2957.JPG
 IMG_2959.JPG
 IMG_2961.JPG
 IMG_2962.JPG
 IMG_2964.JPG
 IMG_2965.JPG
 IMG_2966.JPG
 IMG_2967.JPG
 IMG_2968.JPG
 IMG_2970.JPG
 IMG_2971.JPG
 IMG_2972.JPG
 IMG_2973.JPG
 IMG_2974.JPG
 IMG_2975.JPG
 IMG_2976.JPG
 IMG_2980.JPG
 IMG_2981.JPG
 IMG_2983.JPG
'IMG_2986 (1).JPG'
 IMG_2986.JPG
'IMG_2987 (1).JPG'
 IMG_2987.JPG
'IMG_2988 (1).JPG'
 IMG_2988.JPG
 IMG_2989.JPG
 IMG_2992.JPG
'IMG_2994 (1).JPG'
 IMG_2994.JPG
 IMG_3105.JPG
 IMG_3106.JPG
 IMG_3124.JPG
 IMG_3125.JPG
'IMG_3162 (1).JPG'
 IMG_3162.JPG
'IMG_3163 (1).JPG'
 IMG_3163.JPG
'IMG_3164 (1).JPG'
 IMG_3164.JPG
 IMG_3216.JPG
 IMG_3248.JPG
 IMG_3384.JPG
 IMG_3385.JPG
'IMG_3433 (1).JPG'
 IMG_3433.JPG
'IMG_3444 (1).JPG'
 IMG_3444.JPG
'IMG_3445 (1).JPG'
 IMG_3445.JPG
'IMG_3446 (1).JPG'
 IMG_3446.JPG
 IMG_3465.JPG
 IMG_3466.JPG
 IMG_3467.JPG
 IMG_3468.JPG
 IMG_3469.JPG
 IMG_3470.JPG
 IMG_3543.JPG
 IMG_3547.JPG
 IMG_3548.JPG
 IMG_3602.JPG
'IMG_3671 (1).JPG'
 IMG_3671.JPG
 IMG_3672.JPG
 IMG_3673.JPG
 IMG_3675.JPG
'IMG_3685 (1).JPG'
 IMG_3685.JPG
'IMG_3710 (1).JPG'
 IMG_3710.JPG
'IMG_3711 (1).JPG'
 IMG_3711.JPG
'IMG_3712 (1).JPG'
 IMG_3712.JPG
'IMG_3713 (1).JPG'
 IMG_3713.JPG
 IMG_3720.JPG
 IMG_3721.JPG
'IMG_3755 (1).JPG'
 IMG_3755.JPG
 IMG_3757.JPG
 IMG_3770.JPG
'IMG_3771 (1).JPG'
 IMG_3771.JPG
 IMG_3772.JPG
 IMG_3773.JPG
 IMG_3774.JPG
'IMG_3777 (1).JPG'
 IMG_3777.JPG
 IMG_3807.JPG
 IMG_3808.JPG
'IMG_3840 (1).JPG'
 IMG_3840.JPG
'IMG_3841 (1).JPG'
 IMG_3841.JPG
 IMG_3955.JPG
 IMG_3956.JPG
 IMG_3976.JPG
'IMG_4069 (1).JPG'
 IMG_4069.JPG
 IMG_4070.JPG
'IMG_4073 (1).JPG'
 IMG_4073.JPG
 IMG_4079.JPG
 IMG_4080.JPG
 IMG_4336.JPG
 IMG_4338.JPG
'IMG_4439 (1).JPG'
 IMG_4439.JPG
 IMG_4553.JPG
'IMG_4559 (1).JPG'
 IMG_4559.JPG
'IMG_4628 (1).JPG'
 IMG_4628.JPG
'IMG_4666 (1).JPG'
 IMG_4666.JPG
'IMG_4668 (1).JPG'
 IMG_4668.JPG
'IMG_4674 (1).JPG'
 IMG_4674.JPG
'IMG_4675 (1).JPG'
 IMG_4675.JPG
 IMG_4684.JPG
'IMG_4707 (1).JPG'
 IMG_4707.JPG
 IMG_4708.JPG
 IMG_4709.JPG
 IMG_4710.JPG
 IMG_4830.JPG
 IMG_4831.JPG
 IMG_4834.JPG
 IMG_4835.JPG
 IMG_4836.JPG
'IMG_4936 (1).JPG'
 IMG_4936.JPG
 IMG_4980.JPG
 IMG_4999.JPG
 IMG_5001.JPG
 IMG_5002.JPG
 IMG_5003.JPG
 IMG_5004.JPG
 IMG_5005.JPG
 IMG_5006.JPG
 IMG_5007.JPG
 IMG_5008.JPG
 IMG_5009.JPG
 IMG_5010.JPG
 IMG_5011.JPG
 IMG_5012.JPG
 IMG_5013.JPG
 IMG_5014.JPG
 IMG_5015.JPG
 IMG_5016.JPG
 IMG_5017.JPG
 IMG_5018.JPG
 IMG_5019.JPG
 IMG_5020.JPG
 IMG_5022.JPG
 IMG_5023.JPG
 IMG_5024.JPG
 IMG_5025.JPG
 IMG_5026.JPG
 IMG_5027.JPG
 IMG_5028.JPG
 IMG_5029.JPG
 IMG_5030.JPG
 IMG_5031.JPG
 IMG_5032.JPG
 IMG_5033.JPG
 IMG_5034.JPG
 IMG_5035.JPG
 IMG_5036.JPG
 IMG_5037.JPG
 IMG_5038.JPG
 IMG_5039.JPG
 IMG_5040.JPG
 IMG_5041.JPG
 IMG_5043.JPG
 IMG_5064.JPG
 IMG_5065.JPG
 IMG_5108.JPG
 IMG_5109.JPG
 IMG_5110.JPG
 IMG_5111.JPG
'Intelligent Printing Technique Recognition and Photocopy Detection for Forensic Document Examination.pdf'
'Intro to Golang.gslides'
'Learning deep features for source color laser printer identification based on cascaded learning.pdf'
 Music
 paper-An2.tex
 paper-An.tex
'Passive Classification of Source Printer using Text-line-level Geometric Distortion Signatures from Scanned Images of Printed Documents.pdf'
 Photos
'PrinTracker-Fingerprinting 3D Printers using Commodity Scanners.pdf'
 production-data-science-master
 python-machine-learning-book-master
 Research
'Rua toi'
 Teaching-task
'The Fake vs Real Goods Problem- Microscopy and Machine Learning to the Rescue.pdf'
'Untitled spreadsheet.gsheet'
 Xquant

`!mkdir /content/gdrive/My\ Drive/Courses`_____no_output_____`!mkdir /content/gdrive/My\ Drive/Courses/python-DS`_____no_output_____

!mkdir /content/gdrive/My\ Drive/Courses2_____no_output_____!mkdir /content/gdrive/My\ Drive/Courses2/python-DS_____no_output_____!pwd
import os
os.chdir('/content/gdrive/My Drive/Courses/python-DS')
!pwd/content/gdrive/My Drive
/content/gdrive/My Drive/Courses/python-DS

And, as before, we can list what we have in our current working directory by `ls`_____no_output_____

lssample-data/

What if I do not want to stay here? Suppose I want to go to ___sample-data___ folder, I can ___change directory___ by `cd`_____no_output_____

cd sample-data/content/gdrive/My Drive/Courses/python-DS/sample-data

Now we can verify that we indeed changed our directory by `pwd` and `ls`_____no_output_____

pwd_____no_output_____lsag_news.csv       happy_test.txt  sample.html        terrorists/
another_test.txt  karate.gml      sample_tweets.csv

Note that we use escaping characters for each space for our path (although this is actually NOT a good habit)_____no_output_____Instead of typing everything, we can use ___relative path___. The name is pretty self-explanatory: we can refer to a place, relative to our current working directory. Two important notations here:
- `.` (a dot) means current directory
- `..` (two dots without spaces) means upper level directory.

For example, we can use paths after `ls` command to print files in that given path._____no_output_____

ls .ag_news.csv       happy_test.txt  sample.html        terrorists/
another_test.txt  karate.gml      sample_tweets.csv
ls ..sample-data/

Therefore, by `..` (two dots ), we can go back one level without switching working directory:_____no_output_____

!pwd
!cat sample_tweets.csv/content/gdrive/My Drive/Courses/python-DS/sample-data
We'll have a slice of that 👇 #PiDay https://t.co/Gfzhl5n9Ly
That smile you get on your face when you realize you're #AlwaysAHawkeye. https://t.co/5wiAl9sbiX
In a first of its kind study, #uiowa researchers proved that early intervention for children with hearing loss can… https://t.co/7dcmQ4k63G
Hawkeyes are #B1GTourney champions! Congratulations, @IowaWBB! It's time to go dancing. #FightForIowa 🖤💛🏀 https://t.co/cS4P336JfI
Judges have a history of finding ways to avoid big sentences in white collar economic crimes, with the exception of… https://t.co/KeGgfeiUSs
Creator and co-star of the hit Netflix series Love, Hawkeye Paul Rust said it best this weekend when he visited Iow… https://t.co/jLpGSp3XCd
On #InternationalWomensDay we're celebrating the many incredibly talented and inspirational women of #uiowa. 🖤💛 https://t.co/3y2U8EyF2U
Current Mood: 😁🏆  #MondayMotivation 🖤💛🏀 @IowaWBB https://t.co/klUrInwN7c
New: Montserrat Fuentes, dean and professor in the College of Humanities & Sciences at Virginia Commonwealth Univer… https://t.co/Oto8jMe88I
Emily Maxwell and @BlairBraverman have three things in common: they're dogsledders, Hawkeyes, and they're both comp… https://t.co/hrNhZirAwg
10 seasons, 134 children, and a lifetime of memories make up the Kid Captain program. Read about how the special bo… https://t.co/A40ALrvgWx
When you love living the Hawkeye life.🖤💛🐶 https://t.co/YHs0WLOZUj
Proud to see my article co-written w/ Dr. @tigerlilyrocks focused on online feminism, Wikipedia & digital humanitie… https://t.co/RHGGfd41bi
Calling young writers in India, USA and Pakistan!  - Are you between the ages of 18 and 22? - Do you want to spend… https://t.co/YelZzFk8M9

Finally, it is noteworthy that `~` (tilde) means ___home directory___ in unix systems._____no_output_____

!ls ~_____no_output_____!echo $HOME/root

---_____no_output_____#### Options/Input arguments _____no_output_____Shell commands can take input arguments or options. A convention is to use `-` (dash) to specify arguments. For example, we can ask `ls` to show detailed information of each file/folder:_____no_output_____

ls -ltotal 1853
-rw------- 1 root root 1883166 Mar 26  2019 ag_news.csv
-rw------- 1 root root      46 May 27  2020 another_test.txt
-rw------- 1 root root     929 Mar 17  2012 happy_test.txt
-rw------- 1 root root    5077 Mar 26  2019 karate.gml
-rw------- 1 root root     152 May 27  2020 sample.html
-rw------- 1 root root    1755 Mar 26  2019 sample_tweets.csv
drwx------ 2 root root    4096 Mar  1 02:21 terrorists/

We can aggregate different options by directly appending options one after another. The following example shows how to show size in human readable formats (`-h` option) along with a detailed view (`-l`)_____no_output_____

ls -lhtotal 1.9M
-rw------- 1 root root 1.8M Mar 26  2019 ag_news.csv
-rw------- 1 root root   46 May 27  2020 another_test.txt
-rw------- 1 root root  929 Mar 17  2012 happy_test.txt
-rw------- 1 root root 5.0K Mar 26  2019 karate.gml
-rw------- 1 root root  152 May 27  2020 sample.html
-rw------- 1 root root 1.8K Mar 26  2019 sample_tweets.csv
drwx------ 2 root root 4.0K Mar  1 02:21 terrorists/

Sometimes commands take in arguments for various purposes. Again, using `ls` as example, it can take ___path___ as an argument. Without the path, it will by default show the current listings, as shown above. Given a path, it will list items in that path:_____no_output_____

ls ../sample-data/
ls ../sample-data/terrorists/terrorist.gml  terrorist.groups  terrorist.names  terrorist.pairs

Note that all these options can hardly be memorized. Often we will refer to the manual (or documentation). To do this, we can use `man command_name`. For example:_____no_output_____

man ls | head -20_____no_output_____

Here, `man` is a ___command___ that takes one input argument (which should be a Bash command) and outputs the corresponding manual. Therefore, we can definitely pull up the manual for `man` 🤓_____no_output_____

man man | head -20_____no_output_____

Note that I use `| head -20` to limit the number of output to 20 lines/rows. `|` is ___pipe character___ and `head` is a command to show the ___head___ of some output, where `- 20` limit to the first 20 lines/rows. Detailed coverage is beyond the scope of this workshop though._____no_output_____

!cat > sample.txthi class
class
nice to meet you
you
have you had lunch?
^C
ls -ltotal 1853
-rw------- 1 root root 1883166 Mar 26  2019 ag_news.csv
-rw------- 1 root root      46 May 27  2020 another_test.txt
-rw------- 1 root root     929 Mar 17  2012 happy_test.txt
-rw------- 1 root root    5077 Mar 26  2019 karate.gml
-rw------- 1 root root     152 May 27  2020 sample.html
-rw------- 1 root root    1755 Mar 26  2019 sample_tweets.csv
-rw------- 1 root root      56 Jun 28 12:11 sample.txt
drwx------ 2 root root    4096 Mar  1 02:21 terrorists/
cat sample.txthi class
class
nice to meet you
you
have you had lunch?
!touch sample2.txt_____no_output_____ls -ltotal 1853
-rw------- 1 root root 1883166 Mar 26  2019 ag_news.csv
-rw------- 1 root root      46 May 27  2020 another_test.txt
-rw------- 1 root root     929 Mar 17  2012 happy_test.txt
-rw------- 1 root root    5077 Mar 26  2019 karate.gml
-rw------- 1 root root       0 Jun 28 12:13 sample2.txt
-rw------- 1 root root     152 May 27  2020 sample.html
-rw------- 1 root root    1755 Mar 26  2019 sample_tweets.csv
-rw------- 1 root root      56 Jun 28 12:11 sample.txt
drwx------ 2 root root    4096 Mar  1 02:21 terrorists/
!rm sample.txt sample2.txt_____no_output_____ls -ltotal 1853
-rw------- 1 root root 1883166 Mar 26  2019 ag_news.csv
-rw------- 1 root root      46 May 27  2020 another_test.txt
-rw------- 1 root root     929 Mar 17  2012 happy_test.txt
-rw------- 1 root root    5077 Mar 26  2019 karate.gml
-rw------- 1 root root     152 May 27  2020 sample.html
-rw------- 1 root root    1755 Mar 26  2019 sample_tweets.csv
drwx------ 2 root root    4096 Mar  1 02:21 terrorists/
!which python/usr/local/bin/python
!which python3/usr/bin/python3
!echo $PATH/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tools/node/bin:/tools/google-cloud-sdk/bin:/opt/bin
!echo $HOME/root
!unameLinux
!ps    PID TTY          TIME CMD
      1 ?        00:00:04 node
     16 ?        00:00:00 tail
     49 ?        00:00:05 jupyter-noteboo
     50 ?        00:00:02 dap_multiplexer
     61 ?        00:00:22 python3
     81 ?        00:00:04 python3
    294 ?        00:00:00 bash
    295 ?        00:00:00 drive
    296 ?        00:00:00 grep
    402 ?        00:00:00 drive
    416 ?        00:00:00 fusermount <defunct>
    457 ?        00:00:00 bash
    458 ?        00:00:00 tail
    459 ?        00:00:00 python3
    772 ?        00:00:00 ps
!whoamiroot
!which git/usr/bin/git
!which pip/usr/local/bin/pip
!which pip3/usr/local/bin/pip3
!pip install missingnoRequirement already satisfied: missingno in /usr/local/lib/python3.7/dist-packages (0.4.2)
Requirement already satisfied: seaborn in /usr/local/lib/python3.7/dist-packages (from missingno) (0.11.1)
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from missingno) (1.19.5)
Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from missingno) (1.4.1)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from missingno) (3.2.2)
Requirement already satisfied: pandas>=0.23 in /usr/local/lib/python3.7/dist-packages (from seaborn->missingno) (1.1.5)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->missingno) (0.10.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->missingno) (1.3.1)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->missingno) (2.8.1)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->missingno) (2.4.7)
Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.23->seaborn->missingno) (2018.9)
Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from cycler>=0.10->matplotlib->missingno) (1.15.0)

### Daily best practice of DS

`wc` stands for word count. As the name implies, it is mainly used for counting purpose. It is used to find out number of lines, word count, byte and characters count in the files specified in the file arguments_____no_output_____

!wc ag_news.csv   7601  280558 1883166 ag_news.csv

The `uniq` command in Linux is a command line utility that reports or filters out the repeated lines in a file.
In simple words, `uniq` is the tool that helps to detect the adjacent duplicate lines and also deletes the duplicate lines_____no_output_____

!uniq happy_test.txtI'm happy for him...really, I am. She's an amazing girl, and they deserve each other. He's happy & thats all that matters...right?.....
Feel so happy with no reason... Just happy... Hey my brain, am I missing something? :))
We finished our first season of @TheBEATDance & I am so happy & proud & thankful & overwhelmed & lots of other good stuff! So Amazing #2013
am i allowed to be happy about something, or do yo wanna distroy the little i have left?
I am so happy right now I can't even focus on anything else
Why am I being sneaked around her fam when I'm open about us.... But we both happy shit don't add up.
Heavens suppose to be the happiest place in the world I am happy everyday with the people I love but I feel like I live in heaven everyday:)
I am  so happy since I have get an $100,00 STARBUCKS GIFT-CARD for Free. I grab it here http://t.co/cg8M1Ubq
I am one #happy girl :)
I Am So HAPPY .

`sed` perform lot’s of function on file like, searching, find and replace, insertion or deletion. Simply speaking, it's used for searching and replacement_____no_output_____

!sed 's/amazing/interesting/' happy_test.txtI'm happy for him...really, I am. She's an interesting girl, and they deserve each other. He's happy & thats all that matters...right?.....
Feel so happy with no reason... Just happy... Hey my brain, am I missing something? :))
We finished our first season of @TheBEATDance & I am so happy & proud & thankful & overwhelmed & lots of other good stuff! So Amazing #2013
am i allowed to be happy about something, or do yo wanna distroy the little i have left?
I am so happy right now I can't even focus on anything else
Why am I being sneaked around her fam when I'm open about us.... But we both happy shit don't add up.
Heavens suppose to be the happiest place in the world I am happy everyday with the people I love but I feel like I live in heaven everyday:)
I am  so happy since I have get an $100,00 STARBUCKS GIFT-CARD for Free. I grab it here http://t.co/cg8M1Ubq
I am one #happy girl :)
I Am So HAPPY .
!echo 'I like programming so much' | awk '{ print $4 $5}'somuch

---_____no_output_____#### Conclusions_____no_output_____Due to time contraints, we can only cover these simple examples. There are really a lot more to read: `grep`, `ssh`, `scp`, `ps`, `rm`, etc... Bash command is really powerful and used extensitvely for various purposes. Below I list two tutorials for Bash that I find really helpful._____no_output_____Further readings:

- http://www.bash.academy/
- https://ryanstutorials.net/bash-scripting-tutorial/_____no_output_____## Exercises

1. Create a new folder named `Practice`
2. In `Practice`, create a `test.txt` file named with content `Hello world`
3. In the parent directory of `Practice`, create en empty `test2.txt` file.
4. Remove both `test.txt` and `test2.txt` _____no_output_____

_____no_output_____


# Notebook from nlazarevic/gi-2021-etf
Path: notebooks/07_Bio_Python.ipynb

# BIOPYTHON_____no_output_____pip install biopython

http://biopython.org/DIST/docs/tutorial/Tutorial.html_____no_output_____

from Bio import SeqIO
from Bio.Seq import Seq
import os_____no_output_____fasta_path = 'https://raw.githubusercontent.com/biopython/biopython/master/Doc/examples/ls_orchid.fasta'
if not os.path.exists('ls_orchid.fasta'):
    !wget {fasta_path}
for seq_record in SeqIO.parse("ls_orchid.fasta", "fasta"):
    print(seq_record.id)
    print(repr(seq_record.seq))
    print(len(seq_record))gi|2765658|emb|Z78533.1|CIZ78533
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGATGAGACCGTGG...CGC')
740
gi|2765657|emb|Z78532.1|CCZ78532
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGACAACAG...GGC')
753
gi|2765656|emb|Z78531.1|CFZ78531
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGACAGCAG...TAA')
748
gi|2765655|emb|Z78530.1|CMZ78530
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAAACAACAT...CAT')
744
gi|2765654|emb|Z78529.1|CLZ78529
Seq('ACGGCGAGCTGCCGAAGGACATTGTTGAGACAGCAGAATATACGATTGAGTGAA...AAA')
733
gi|2765652|emb|Z78527.1|CYZ78527
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGACAGTAG...CCC')
718
gi|2765651|emb|Z78526.1|CGZ78526
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGACAGTAG...TGT')
730
gi|2765650|emb|Z78525.1|CAZ78525
Seq('TGTTGAGATAGCAGAATATACATCGAGTGAATCCGGAGGACCTGTGGTTATTCG...GCA')
704
gi|2765649|emb|Z78524.1|CFZ78524
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATAGTAG...AGC')
740
gi|2765648|emb|Z78523.1|CHZ78523
Seq('CGTAACCAGGTTTCCGTAGGTGAACCTGCGGCAGGATCATTGTTGAGACAGCAG...AAG')
709
gi|2765647|emb|Z78522.1|CMZ78522
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGACAGCAG...GAG')
700
gi|2765646|emb|Z78521.1|CCZ78521
Seq('GTAGGTGAACCTGCGGAAGGATCATTGTTGAGACAGTAGAATATATGATCGAGT...ACC')
726
gi|2765645|emb|Z78520.1|CSZ78520
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGACAGCAG...TTT')
753
gi|2765644|emb|Z78519.1|CPZ78519
Seq('ATATGATCGAGTGAATCTGGTGGACTTGTGGTTACTCAGCTCGCCATAGGCTTT...TTA')
699
gi|2765643|emb|Z78518.1|CRZ78518
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGGAGGATCATTGTTGAGATAGTAG...TCC')
658
gi|2765642|emb|Z78517.1|CFZ78517
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGACAGTAG...AGC')
752
gi|2765641|emb|Z78516.1|CPZ78516
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGACAGTAT...TAA')
726
gi|2765640|emb|Z78515.1|MXZ78515
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGCTGAGACCGTAG...AGC')
765
gi|2765639|emb|Z78514.1|PSZ78514
Seq('CGTAACAAGGTTTCCGTAGGTGGACCTTCGGGAGGATCATTTTTGAAGCCCCCA...CTA')
755
gi|2765638|emb|Z78513.1|PBZ78513
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGACCGCCA...GAG')
742
gi|2765637|emb|Z78512.1|PWZ78512
Seq('CGTAACAAGGTTTCCGTAGGTGGACCTTCGGGAGGATCATTTTTGAAGCCCCCA...AGC')
762
gi|2765636|emb|Z78511.1|PEZ78511
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTTCGGAAGGATCATTGTTGAGACCCCCA...GGA')
745
gi|2765635|emb|Z78510.1|PCZ78510
Seq('CTAACCAGGGTTCCGAGGTGACCTTCGGGAGGATTCCTTTTTAAGCCCCCGAAA...TTA')
750
gi|2765634|emb|Z78509.1|PPZ78509
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGACCGCCA...GGA')
731
gi|2765633|emb|Z78508.1|PLZ78508
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGACCGCCA...TGA')
741
gi|2765632|emb|Z78507.1|PLZ78507
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGACCCCCA...TGA')
740
gi|2765631|emb|Z78506.1|PLZ78506
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGACCGCAA...TGA')
727
gi|2765630|emb|Z78505.1|PSZ78505
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGACCGCCA...TTT')
711
gi|2765629|emb|Z78504.1|PKZ78504
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTTCGGAAGGATCATTGTTGAGACCGCAA...TAA')
743
gi|2765628|emb|Z78503.1|PCZ78503
Seq('CGTAACCAGGTTTCCGTAGGTGAACCTCCGGAAGGATCCTTGTTGAGACCGCCA...TAA')
727
gi|2765627|emb|Z78502.1|PBZ78502
Seq('CGTAACCAGGTTTCCGTAGGTGAACCTCCGGAAGGATCATTGTTGAGACCGCCA...CGC')
757
gi|2765626|emb|Z78501.1|PCZ78501
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGACCGCAA...AGA')
770
gi|2765625|emb|Z78500.1|PWZ78500
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGCTCATTGTTGAGACCGCAA...AAG')
767
gi|2765624|emb|Z78499.1|PMZ78499
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAGGGATCATTGTTGAGATCGCAT...ACC')
759
gi|2765623|emb|Z78498.1|PMZ78498
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAAGGTCATTGTTGAGATCACAT...AGC')
750
gi|2765622|emb|Z78497.1|PDZ78497
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...AGC')
788
gi|2765621|emb|Z78496.1|PAZ78496
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCGCAT...AGC')
774
gi|2765620|emb|Z78495.1|PEZ78495
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTCCGGAAGGATCATTGTTGAGATCACAT...GTG')
789
gi|2765619|emb|Z78494.1|PNZ78494
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGGTCGCAT...AAG')
688
gi|2765618|emb|Z78493.1|PGZ78493
Seq('CGTAACAAGGATTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCGCAT...CCC')
719
gi|2765617|emb|Z78492.1|PBZ78492
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCGCAT...ATA')
743
gi|2765616|emb|Z78491.1|PCZ78491
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCGCAT...AGC')
737
gi|2765615|emb|Z78490.1|PFZ78490
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...TGA')
728
gi|2765614|emb|Z78489.1|PDZ78489
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...GGC')
740
gi|2765613|emb|Z78488.1|PTZ78488
Seq('CTGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACGCAATAATTGATCGA...GCT')
696
gi|2765612|emb|Z78487.1|PHZ78487
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...TAA')
732
gi|2765611|emb|Z78486.1|PBZ78486
Seq('CGTCACGAGGTTTCCGTAGGTGAATCTGCGGGAGGATCATTGTTGAGATCACAT...TGA')
731
gi|2765610|emb|Z78485.1|PHZ78485
Seq('CTGAACCTGGTGTCCGAAGGTGAATCTGCGGATGGATCATTGTTGAGATATCAT...GTA')
735
gi|2765609|emb|Z78484.1|PCZ78484
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGGGGAAGGATCATTGTTGAGATCACAT...TTT')
720
gi|2765608|emb|Z78483.1|PVZ78483
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...GCA')
740
gi|2765607|emb|Z78482.1|PEZ78482
Seq('TCTACTGCAGTGACCGAGATTTGCCATCGAGCCTCCTGGGAGCTTTCTTGCTGG...GCA')
629
gi|2765606|emb|Z78481.1|PIZ78481
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...TGA')
572
gi|2765605|emb|Z78480.1|PGZ78480
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...TGA')
587
gi|2765604|emb|Z78479.1|PPZ78479
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...AGT')
700
gi|2765603|emb|Z78478.1|PVZ78478
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTCCGGAAGGATCAGTGTTGAGATCACAT...GGC')
636
gi|2765602|emb|Z78477.1|PVZ78477
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...TGC')
716
gi|2765601|emb|Z78476.1|PGZ78476
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...CCC')
592
gi|2765600|emb|Z78475.1|PSZ78475
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...GGT')
716
gi|2765599|emb|Z78474.1|PKZ78474
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACGT...CTT')
733
gi|2765598|emb|Z78473.1|PSZ78473
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...AGG')
626
gi|2765597|emb|Z78472.1|PLZ78472
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...AGC')
737
gi|2765596|emb|Z78471.1|PDZ78471
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...AGC')
740
gi|2765595|emb|Z78470.1|PPZ78470
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...GTT')
574
gi|2765594|emb|Z78469.1|PHZ78469
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...GTT')
594
gi|2765593|emb|Z78468.1|PAZ78468
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCGCAT...GTT')
610
gi|2765592|emb|Z78467.1|PSZ78467
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...TGA')
730
gi|2765591|emb|Z78466.1|PPZ78466
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...CCC')
641
gi|2765590|emb|Z78465.1|PRZ78465
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...TGC')
702
gi|2765589|emb|Z78464.1|PGZ78464
Seq('CGTAACAAGGTTTCCGTAGGTGAGCGGAAGGGTCATTGTTGAGATCACATAATA...AGC')
733
gi|2765588|emb|Z78463.1|PGZ78463
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGTTCATTGTTGAGATCACAT...AGC')
738
gi|2765587|emb|Z78462.1|PSZ78462
Seq('CGTCACGAGGTCTCCGGATGTGACCCTGCGGAAGGATCATTGTTGAGATCACAT...CAT')
736
gi|2765586|emb|Z78461.1|PWZ78461
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTCCGGAAGGATCATTGTTGAGATCACAT...TAA')
732
gi|2765585|emb|Z78460.1|PCZ78460
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTCCGGAAGGATCATTGTTGAGATCACAT...TTA')
745
gi|2765584|emb|Z78459.1|PDZ78459
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...TTT')
744
gi|2765583|emb|Z78458.1|PHZ78458
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...TTG')
738
gi|2765582|emb|Z78457.1|PCZ78457
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTCCGGAAGGATCATTGTTGAGATCACAT...GAG')
739
gi|2765581|emb|Z78456.1|PTZ78456
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...AGC')
740
gi|2765580|emb|Z78455.1|PJZ78455
Seq('CGTAACCAGGTTTCCGTAGGTGGACCTTCGGGAGGATCATTTTTGAGATCACAT...GCA')
745
gi|2765579|emb|Z78454.1|PFZ78454
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...AAC')
695
gi|2765578|emb|Z78453.1|PSZ78453
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...GCA')
745
gi|2765577|emb|Z78452.1|PBZ78452
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...GCA')
743
gi|2765576|emb|Z78451.1|PHZ78451
Seq('CGTAACAAGGTTTCCGTAGGTGTACCTCCGGAAGGATCATTGTTGAGATCACAT...AGC')
730
gi|2765575|emb|Z78450.1|PPZ78450
Seq('GGAAGGATCATTGCTGATATCACATAATAATTGATCGAGTTAAGCTGGAGGATC...GAG')
706
gi|2765574|emb|Z78449.1|PMZ78449
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...TGC')
744
gi|2765573|emb|Z78448.1|PAZ78448
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...AGG')
742
gi|2765572|emb|Z78447.1|PVZ78447
Seq('CGTAACAAGGATTCCGTAGGTGAACCTGCGGGAGGATCATTGTTGAGATCACAT...AGC')
694
gi|2765571|emb|Z78446.1|PAZ78446
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTCCGGAAGGATCATTGTTGAGATCACAT...CCC')
712
gi|2765570|emb|Z78445.1|PUZ78445
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...TGT')
715
gi|2765569|emb|Z78444.1|PAZ78444
Seq('CGTAACAAGGTTTCCGTAGGGTGAACTGCGGAAGGATCATTGTTGAGATCACAT...ATT')
688
gi|2765568|emb|Z78443.1|PLZ78443
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...AGG')
784
gi|2765567|emb|Z78442.1|PBZ78442
Seq('GTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACATAATAATTGATCGAGT...AGT')
721
gi|2765566|emb|Z78441.1|PSZ78441
Seq('GGAAGGTCATTGCCGATATCACATAATAATTGATCGAGTTAATCTGGAGGATCT...GAG')
703
gi|2765565|emb|Z78440.1|PPZ78440
Seq('CGTAACAAGGTTTCCGTAGGTGGACCTCCGGGAGGATCATTGTTGAGATCACAT...GCA')
744
gi|2765564|emb|Z78439.1|PBZ78439
Seq('CATTGTTGAGATCACATAATAATTGATCGAGTTAATCTGGAGGATCTGTTTACT...GCC')
592

# BLAST_____no_output_____The Basic Local Alignment Search Tool ([BLAST](https://blast.ncbi.nlm.nih.gov/Blast.cgi)) finds regions of local similarity between sequences.

The program compares nucleotide or protein sequences to sequence databases and calculates the statistical significance of matches. BLAST can be used to infer functional and evolutionary relationships between sequences as well as help identify members of gene families._____no_output_____

from Bio.Blast import NCBIWWW
help(NCBIWWW.qblast)Help on function qblast in module Bio.Blast.NCBIWWW:

qblast(program, database, sequence, url_base='https://blast.ncbi.nlm.nih.gov/Blast.cgi', auto_format=None, composition_based_statistics=None, db_genetic_code=None, endpoints=None, entrez_query='(none)', expect=10.0, filter=None, gapcosts=None, genetic_code=None, hitlist_size=50, i_thresh=None, layout=None, lcase_mask=None, matrix_name=None, nucl_penalty=None, nucl_reward=None, other_advanced=None, perc_ident=None, phi_pattern=None, query_file=None, query_believe_defline=None, query_from=None, query_to=None, searchsp_eff=None, service=None, threshold=None, ungapped_alignment=None, word_size=None, short_query=None, alignments=500, alignment_view=None, descriptions=500, entrez_links_new_window=None, expect_low=None, expect_high=None, format_entrez_query=None, format_object=None, format_type='XML', ncbi_gi=None, results_file=None, show_overview=None, megablast=None, template_type=None, template_length=None)
    BLAST search using NCBI's QBLAST server or a cloud service provider.

    Supports all parameters of the old qblast API for Put and Get.

    Please note that NCBI uses the new Common URL API for BLAST searches
    on the internet (http://ncbi.github.io/blast-cloud/dev/api.html). Thus,
    some of the parameters used by this function are not (or are no longer)
    officially supported by NCBI. Although they are still functioning, this
    may change in the future.

    The Common URL API (http://ncbi.github.io/blast-cloud/dev/api.html) allows
    doing BLAST searches on cloud servers. To use this feature, please set
    ``url_base='http://host.my.cloud.service.provider.com/cgi-bin/blast.cgi'``
    and ``format_object='Alignment'``. For more details, please see
    https://blast.ncbi.nlm.nih.gov/Blast.cgi?PAGE_TYPE=BlastDocs&DOC_TYPE=CloudBlast

    Some useful parameters:

     - program        blastn, blastp, blastx, tblastn, or tblastx (lower case)
     - database       Which database to search against (e.g. "nr").
     - sequence       The sequence to search.
     - ncbi_gi        TRUE/FALSE whether to give 'gi' identifier.
     - descriptions   Number of descriptions to show.  Def 500.
     - alignments     Number of alignments to show.  Def 500.
     - expect         An expect value cutoff.  Def 10.0.
     - matrix_name    Specify an alt. matrix (PAM30, PAM70, BLOSUM80, BLOSUM45).
     - filter         "none" turns off filtering.  Default no filtering
     - format_type    "HTML", "Text", "ASN.1", or "XML".  Def. "XML".
     - entrez_query   Entrez query to limit Blast search
     - hitlist_size   Number of hits to return. Default 50
     - megablast      TRUE/FALSE whether to use MEga BLAST algorithm (blastn only)
     - short_query    TRUE/FALSE whether to adjust the search parameters for a
                      short query sequence. Note that this will override
                      manually set parameters like word size and e value. Turns
                      off when sequence length is > 30 residues. Default: None.
     - service        plain, psi, phi, rpsblast, megablast (lower case)

    This function does no checking of the validity of the parameters
    and passes the values to the server as is.  More help is available at:
    https://ncbi.github.io/blast-cloud/dev/api.html

from Bio.Blast import NCBIWWW
from Bio.Blast import NCBIXML
from Bio import SeqIO
record = SeqIO.read("opuntia.fasta", format="fasta")
result_handle = NCBIWWW.qblast("blastn", "nt", record.seq)
blast_record = NCBIXML.read(result_handle)_____no_output_____E_VALUE_THRESH = 0.04
for alignment in blast_record.alignments:
    # A High-scoring Segment Pair (HSP) is a local alignment with no gaps that
    # achieves one of the highest alignment scores in a given search.
    for hsp in alignment.hsps:
        if hsp.expect < E_VALUE_THRESH:
            print("****Alignment****")
            print("sequence:", alignment.title)
            print("length:", alignment.length)
            print("e value:", hsp.expect)
            print(hsp.query[0:75] + "...")
            print(hsp.match[0:75] + "...")
            print(hsp.sbjct[0:75] + "...")
****Alignment****
sequence: gi|6273291|gb|AF191665.1|AF191665 Opuntia marenae rpl16 gene; chloroplast gene for chloroplast product, partial intron sequence
length: 902
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||...
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
****Alignment****
sequence: gi|6273290|gb|AF191664.1|AF191664 Opuntia clavata rpl16 gene; chloroplast gene for chloroplast product, partial intron sequence
length: 899
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||...
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
****Alignment****
sequence: gi|6273289|gb|AF191663.1|AF191663 Opuntia bradtiana rpl16 gene; chloroplast gene for chloroplast product, partial intron sequence
length: 899
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||...
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
****Alignment****
sequence: gi|330887234|gb|HM041475.1| Cylindropuntia imbricata ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1197
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||...
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
****Alignment****
sequence: gi|57240067|gb|AY851607.1| Pereskiopsis porteri rpl16 gene, intron; chloroplast
length: 902
e value: 0.0
CATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGATTCC...
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||||||...
CATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGATTCC...
****Alignment****
sequence: gi|330887194|gb|HM041435.1| Grusonia bradtiana ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1181
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
||||| |||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||...
TATACTTTAAGGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
****Alignment****
sequence: gi|6273284|gb|AF191658.1|AF191658 Opuntia subulata rpl16 gene; chloroplast gene for chloroplast product, partial intron sequence
length: 896
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||||||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACATTAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|57240072|gb|AY851612.1| Opuntia subulata rpl16 gene, intron; chloroplast
length: 892
e value: 0.0
CATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGATTCC...
|||||||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||||||...
CATTAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGATTCC...
****Alignment****
sequence: gi|6273286|gb|AF191660.1|AF191660 Opuntia echinacea rpl16 gene; chloroplast gene for chloroplast product, partial intron sequence
length: 893
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
||||||| |||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACATAAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|330887254|gb|HM041495.1| Pereskiopsis sp. THH-2011 ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1168
e value: 0.0
CATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGATTCC...
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||  |||||...
CATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAC-ATTCC...
****Alignment****
sequence: gi|6273285|gb|AF191659.1|AF191659 Opuntia pachypus rpl16 gene; chloroplast gene for chloroplast product, partial intron sequence
length: 894
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||||||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACATTAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|330887236|gb|HM041477.1| Cylindropuntia leptocaulis ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1197
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
||||| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||...
TATACTTTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
****Alignment****
sequence: gi|6273287|gb|AF191661.1|AF191661 Opuntia kuehnrichiana rpl16 gene; chloroplast gene for chloroplast product, partial intron sequence
length: 895
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||||||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACATTAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|330887282|gb|HM041523.1| Tephrocactus articulatus ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1171
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
||||| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACTTTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|330887257|gb|HM041498.1| Pterocactus gonjianii ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1180
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
||||| || ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACNTTCAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|330887241|gb|HM041482.1| Cylindropuntia tunicata ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1197
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
||||| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACNTTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|330887261|gb|HM041502.1| Quiabentia verticillata ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1194
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|1972904678|gb|MT359369.1| Opuntia monacantha voucher M.Kohler 380 (ICN) ribosomal protein L16 (rpl16) gene, complete cds; chloroplast
length: 1537
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|1972904629|gb|MT359341.1| Opuntia arechavaletae voucher M.Kohler 320 (ICN) ribosomal protein L16 (rpl16) gene, complete cds; chloroplast
length: 1539
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|1972904615|gb|MT359333.1| Brasiliopuntia brasiliensis voucher M.Kohler 472 (ICN) ribosomal protein L16 (rpl16) gene, complete cds; chloroplast
length: 1539
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|330887216|gb|HM041457.1| Miqueliopuntia miquelii ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1197
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
||||| |||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACNTTAANGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|330887280|gb|HM041521.1| Tacinga funalis ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1188
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACATTAAGGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|1841709044|gb|MN114084.1| Opuntia quimilo chloroplast, complete genome
length: 150374
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|1841709044|gb|MN114084.1| Opuntia quimilo chloroplast, complete genome
length: 150374
e value: 0.00767111
CGAGGAGCCGGATGAGAAGAAACTCTCATGTCCGGTTCTGTAG...
|||||||||| |||||  |||| |||||||| |||||||||||...
CGAGGAGCCGTATGAGGTGAAAATCTCATGTACGGTTCTGTAG...
****Alignment****
sequence: gi|1972904692|gb|MT359377.1| Opuntia rioplatense voucher M.Kohler 287 (ICN) ribosomal protein L16 (rpl16) gene, complete cds; chloroplast
length: 1541
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|1972904650|gb|MT359353.1| Opuntia bonaerensis voucher M.Kohler 426 (ICN) ribosomal protein L16 (rpl16) gene, complete cds; chloroplast
length: 1540
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|1972904643|gb|MT359349.1| Opuntia bonaerensis voucher F.Font 423 (BAF) ribosomal protein L16 (rpl16) gene, complete cds; chloroplast
length: 1540
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|1972904622|gb|MT359337.1| Opuntia bonaerensis voucher M.Kohler 295 (ICN) ribosomal protein L16 (rpl16) gene, complete cds; chloroplast
length: 1540
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|330887231|gb|HM041472.1| Opuntia echios ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1182
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|330887223|gb|HM041464.1| Nopalea cochenillifera ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1184
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
||||| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACTTTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|6273288|gb|AF191662.1|AF191662 Pterocactus kuntzei rpl16 gene; chloroplast gene for chloroplast product, partial intron sequence
length: 892
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACATTCAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|330887240|gb|HM041481.1| Opuntia palmadora ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1200
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
||||| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACNTTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|330887260|gb|HM041501.1| Quiabentia chacoensis ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1199
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
||||| |||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACTTTAAGGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|330887233|gb|HM041474.1| Opuntia helleri ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1163
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|1972904671|gb|MT359365.1| Opuntia megapotamica voucher F.Font 856 (BAF) ribosomal protein L16 (rpl16) gene, complete cds; chloroplast
length: 1542
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|1972904657|gb|MT359357.1| Opuntia elata voucher F.Font 580 (BAF) ribosomal protein L16 (rpl16) gene, complete cds; chloroplast
length: 1542
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|330887238|gb|HM041479.1| Opuntia megasperma ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1197
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
||||| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACTTTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|57240073|gb|AY851613.1| Pterocactus kuntzei rpl16 gene, intron; chloroplast
length: 888
e value: 0.0
CATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGATTCC...
|||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||||||...
CATTCAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGATTCC...
****Alignment****
sequence: gi|330887228|gb|HM041469.1| Nopalea sp. THH-2011 ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1189
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
||||| |||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACTTTAAGGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|57240071|gb|AY851611.1| Opuntia polyacantha rpl16 gene, intron; chloroplast
length: 881
e value: 0.0
CATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGATTCC...
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||||||...
CATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGATTCC...
****Alignment****
sequence: gi|330887225|gb|HM041466.1| Nopalea gaumeri ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1205
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|330887227|gb|HM041468.1| Nopalea lutea ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1202
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
||||| |||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACTTTAAGGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|330887224|gb|HM041465.1| Nopalea dejecta ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1190
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
||||| |||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACTTTAANGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|330887237|gb|HM041478.1| Opuntia macbridei ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1187
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|330887232|gb|HM041473.1| Opuntia excelsa ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1203
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|330887281|gb|HM041522.1| Tephrocactus alexanderi ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1194
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATG-AATCTAAATGATATAGG...
||||| |||| ||||||||||||||| |||||||||||||||| ||||||||||||| |||||||| |||||| |...
TATACTTTAANGGAGGGGGATGCGGA-AAATGGAAAGGCGAAANAAAGAAAAAAATGGAATCTAAANGATATACG...
****Alignment****
sequence: gi|1972904636|gb|MT359345.1| Opuntia austrina voucher L.C.Majure 4216 (FLAS) ribosomal protein L16 (rpl16) gene, complete cds; chloroplast
length: 1537
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAA---TGAATCTAAATGATATA...
|||||||||||||||||||||||||||||||||||||||||||||||||||||||   |||||||||||||||||...
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAAAAATGAATCTAAATGATATA...
****Alignment****
sequence: gi|1972904664|gb|MT359361.1| Opuntia macrorhiza voucher B.Snow 2089 (FLAS) ribosomal protein L16 (rpl16) gene, complete cds; chloroplast
length: 1536
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAA-TGAATCTAAATGATATAGG...
||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||||||||||||||||| |...
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAAATGAATCTAAATGATATACG...
****Alignment****
sequence: gi|1972904685|gb|MT359373.1| Opuntia penicilligera voucher M.Kohler 541 (ICN) ribosomal protein L16 (rpl16) gene, complete cds; chloroplast
length: 1537
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAA--TGAATCTAAATGATATAG...
|||||||||||||||||||||||||||||||||||||||||||||||||||||||  ||||||||||||||||| ...
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAAAATGAATCTAAATGATATAC...
****Alignment****
sequence: gi|330887235|gb|HM041476.1| Opuntia lasiacantha ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1205
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
||| | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||...
TATCCTTTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
****Alignment****
sequence: gi|330887286|gb|HM041527.1| Tunilla corrugata ribosomal protein L16-like (rpl16) gene, partial sequence; chloroplast
length: 1210
e value: 0.0
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAATCTAAATGATATAGGA...
||||| |||||||||||||||||  |||||||||| |   |||||||||||||||||||||||||||||||| ||...
TATACTTTAAAGGAGGGGGATGCAAATAAATGGAAGGNGAAAAGAAAGAAAAAAATGAATCTAAATGATATACGA...
# Slice before start codon
sliced_dna = record.seq[record.seq.find("ATG"):]
protein = sliced_dna.translate(to_stop=True)
protein/usr/local/Caskroom/miniconda/base/lib/python3.8/site-packages/Bio/Seq.py:2334: BiopythonWarning: Partial codon, len(sequence) not a multiple of three. Explicitly trim the sequence or add trailing N before translation. This may become an error in future.
  warnings.warn(


# Notebook from UIUC-iSchool-DataViz/spring2019online
Path: week09/prep_notebook_week09.ipynb

# Activity #1: MarketMap
* another way to visualize mappable data_____no_output_____## 1.a : explore the dataset_____no_output_____

# our usual stuff
%matplotlib inline
import pandas as pd
import numpy as np_____no_output_____#!pip install xlrd # JPN, might have to run this

# note: this is quering from the web!  How neat is that??
df = pd.read_excel('https://query.data.world/s/ivl45pdpubos6jpsii3djsjwm2pcjv', skiprows=5)
# the above might take a while to load all the data_____no_output_____# what is in this dataframe? lets take a look at the top
df.head()
# this dataset is called: "Surgery Charges Across the U.S."
#  and its just showing us how much different procedures
#  cost from different hospitals_____no_output_____# what kinds of data are we working with?
df.dtypes_____no_output_____# lets look at some summary data
# recall: this is like R's "summary" function
df.describe()
# so, things like the mean zipcode aren't
#  meaningful, same thing with provider ID
#  But certainly looking at the average
#  total payments, discharges, might
#  be useful_____no_output_____# lets look at how many seperate types of surgery are
#  represented in this dataset:
df["DRG Definition"].unique().size_____no_output_____# what about how many provider (hospital) names?
df["Provider Name"].unique().size_____no_output_____# how many states are represented
df["Provider State"].unique().size_____no_output_____# what are the state codes?
df["Provider State"].unique()_____no_output_____# lets figure out  what the most common surgeries are via how
#  many many folks are discharged after each type of surgery
# (1)
most_common = df.groupby("DRG Definition")["Total Discharges"].sum()
most_common

# (2) but lets sort by the largest on top
most_common = df.groupby("DRG Definition")["Total Discharges"].sum().sort_values(ascending=False)
most_common

# (3) lets look at only the top 5, for fun
most_common[:5]

# (4) or we can only look at the names of the top 5:
most_common[:5].index.values_____no_output_____

## 1.b: formatting data for MarketMap
* here we are going to practice doing some fancy things to clean this data
* this will be good practice for when you run into other datasets "in the wild"_____no_output_____

# (1) lets create a little table of total discharges for
#  each type of surgery & state
total_discharges = df.groupby(["DRG Definition", "Provider State"])["Total Discharges"].sum()
total_discharges

# (2) the above is not intuative, lets prettify it
total_discharges = df.groupby(["DRG Definition", "Provider State"])["Total Discharges"].sum().unstack()
total_discharges_____no_output_____

### Aside: lets quick check out what are the most frequent surgeries_____no_output_____

# for our map, we are going to want to
# normalize the discharges or each surgery
# for each
# state by the total discharges across all
# states for a particular type of surger
#  lets add this to our total_discharges DF
total_discharges["Total"] = total_discharges.sum(axis = 1)
total_discharges["Total"].head() # just look at the first few_____no_output_____# finally, lets check out the most often
#  performed surgery across all states

# we can do this by sorting our DF by this total we just
# calculated:
total_discharges.sort_values(by = "Total",
                             ascending=False,
                             inplace = True)

# now lets just look at the first few of our
#  sorted array
total_discharges.head()

# so, from this we see that joint replacement
#  or reattachment of a lower extremeity is
#  the most likely surgery (in number of discharges)
# followed by surgeries for sepsis and then heart failure_____no_output_____# neat.  We won't need these for plotting, so we can remove our
# total column we just calculated
del total_discharges["Total"]
total_discharges.head()
# now we see that we are back to just states & surgeries
#  *but* our sorting is still by the total that we
# previously calculated.
# spiffy!_____no_output_____

## 1.c: plot data with bqplot_____no_output_____

import bqplot
# by default bqplot does not import
#  all packages, we have to
#  explicitely import market_map
import bqplot.market_map # for access to market_map_____no_output_____# lets do our usual thing, but with a market map
#  instead of a heat map

# scales:
x_sc, y_sc = bqplot.OrdinalScale(), bqplot.OrdinalScale() # note, just a different way to call things
c_sc = bqplot.ColorScale(scheme="Blues")

# just a color axes for now:
c_ax = bqplot.ColorAxis(scale = c_sc, orientation = 'vertical')

# lets make the market map:

# (1) what should we plot for our color? lets take a look:
total_discharges.iloc[0].values, total_discharges.columns.values
# this is the total discharges for the most
# popular surgical procedure
# the columns will be states

# (2) lets put this into a map
mmap = bqplot.market_map.MarketMap(color = total_discharges.iloc[0].values,
                                  names = total_discharges.columns.values,
                                  scales={'color':c_sc},
                                   axes=[c_ax])

# (3) ok, but just clicking on things doesn't tell us too much
# lets add a little label to print out the total of the selected
import ipywidgets
label = ipywidgets.Label()
# link to market map
def get_data(change):
    # (3.1)
    #print(change['owner'].selected)
    # (3.2) loop
    v = 0.0 # to store total value
    for s in change['owner'].selected:
        v += total_discharges.iloc[0][total_discharges.iloc[0].index == s].values
    if v > 0: # in case nothing is selected
        # what are we printing?
        l = 'Total discharges of ' + \
        total_discharges.iloc[0].name + \
        ' = ' + str(v[0]) # note: v is by default an array
        label.value = l

mmap.observe(get_data,'selected')

#mmap

# (3)
ipywidgets.VBox([label,mmap])_____no_output_____

## Discussion:
* think back to the map we had last week: we can certainly plot this information with a more geo-realistic map
* what are the pros & cons of each style of map?  What do each highlight?  How are each biased?_____no_output_____## IF we have time: Re-do with other mapping system:_____no_output_____

from us_state_abbrev import us_state_abbrev

sc_geo = bqplot.AlbersUSA()
state_data = bqplot.topo_load('map_data/USStatesMap.json')

#(1)
#states_map = bqplot.Map(map_data=state_data, scales={'projection':sc_geo})

#(2)
# library from last time
from states_utils import get_ids_and_names
ids, state_names = get_ids_and_names(states_map)

# color maps
import matplotlib.cm as cm
cmap = cm.Blues

# most popular surgery
popSurg = total_discharges.iloc[0]

# here, we will go through the process of getting colors to plot
#  each state with its similar color to the marketmap above:

#!pip install webcolors
from webcolors import rgb_to_hex
d = {} # empty dict to store colors
for s in states_map.map_data['objects']['subunits']['geometries']:
    if s['properties'] is not None:
        #print(s['properties']['name'], s['id'])
        # match states to abbreviations
        state_abbrev = us_state_abbrev[s['properties']['name']]
        #print(state_abbrev)
        v = popSurg[popSurg.index == state_abbrev].values[0]
        # renorm v to colors and then number of states
        v = (v - popSurg.values.min())/(popSurg.values.max()-popSurg.values.min())
        #print(v, int(cmap(v)[0]), int(cmap(v)[1]), int(cmap(v)[2]))
        # convert to from 0-1 to 0-255 rgbs
        c = [int(cmap(v)[i]*255) for i in range(3)]
        #d[s['id']] = rgb_to_hex([int(cmap(v)[0]*255), int(cmap(v)[1]*255), int(cmap(v)[2]*255)])
        d[s['id']] = rgb_to_hex(c)

def_tt = bqplot.Tooltip(fields=['name'])

states_map = bqplot.Map(map_data=state_data, scales={'projection':sc_geo}, colors = d, tooltip=def_tt)
# add interactions
states_map.interactions = {'click': 'select', 'hover': 'tooltip'}

# (3)
label = ipywidgets.Label()
# link to heat map
def get_data(change):
    v = 0.0 # to store total value
    if change['owner'].selected is not None:
        for s in change['owner'].selected:
            #print(s)
            sn = state_names[s == ids][0]
            state_abbrev = us_state_abbrev[sn]
            v += popSurg[popSurg.index == state_abbrev].values[0]
        if v > 0: # in case nothing is selected
            # what are we printing?
            l = 'Total discharges of ' + \
            popSurg.name + \
            ' = ' + str(v) # note: v is by default an array
            label.value = l

states_map.observe(get_data,'selected')

fig=bqplot.Figure(marks=[states_map],
                  title='US States Map Example',
                  fig_margin={'top': 0, 'bottom': 0, 'left': 0, 'right': 0}) # try w/o first and see
#fig
# (3)
ipywidgets.VBox([label,fig])_____no_output_____

# Activity #2: Real quick ipyleaflets
* since cartopy wasn't working for folks, we'll quickly look at another option: ipyleaflets_____no_output_____

#!pip install ipyleaflet
from ipyleaflet import *
# note: you might have to close and reopen you notebook
# to see the map

m = Map(center=(52, 10), zoom=8, basemap=basemaps.Hydda.Full)

#(2) street maps
strata_all = basemap_to_tiles(basemaps.Strava.All)
m.add_layer(strata_all)
m_____no_output_____

### Note: more examples available here - https://github.com/jupyter-widgets/ipyleaflet/tree/master/examples_____no_output_____# Activity #3: Networked data - Simple example
_____no_output_____

# lets start with some very basic node data
# **copy paste into chat **
node_data = [
    {"label": "Luke Skywalker", "media": "Star Wars", "shape": "rect"},
    {"label": "Jean-Luc Picard", "media": "Star Trek", "shape": "rect"},
    {"label": "Doctor Who", "media": "Doctor Who", "shape": "rect"},
    {"label": "Pikachu", "media": "Detective Pikachu", "shape": "circle"},
]

# we'll use bqplot.Graph to plot these
graph = bqplot.Graph(node_data=node_data,
                    colors = ["red", "red", "red", "red"])

fig = bqplot.Figure(marks = [graph])
fig

# you note I can pick them up and move them around, but they aren't connected in any way
# lets make some connections_____no_output_____node_data = [
    {"label": "Luke Skywalker", "media": "Star Wars", "shape": "rect"},
    {"label": "Jean-Luc Picard", "media": "Star Trek", "shape": "rect"},
    {"label": "Doctor Who", "media": "Doctor Who", "shape": "rect"},
    {"label": "Pikachu", "media": "Detective Pikachu", "shape": "circle"},
]

# lets link the 0th entry (luke skywalker) to both
#  jean-luc picard (1th entry) and pikachu (3rd entry)
link_data = [{'source': 0, 'target': 1}, {'source': 0, 'target':  3}]

graph = bqplot.Graph(node_data=node_data, link_data=link_data,
                    colors = ["red", "red", "red", "red"])

#(2) we can also play with the springiness of our links:
graph.charge = -300 # setting it to positive makes them want to overlap and is, ingeneral, a lot of fun
# -300 is default

# (3) we can also change the link type:
graph.link_type = 'line' # arc = default, line, slant_line

# (4) highlight link direction, or not
graph.directed = False

fig = bqplot.Figure(marks = [graph])
fig_____no_output_____# we can do all the same things we've done with
# our previous map plots:
# for example, we can add a tooltip:
#(1)
tooltip = bqplot.Tooltip(fields=["media"])
graph = bqplot.Graph(node_data=node_data, link_data=link_data,
                    colors = ["red", "red", "red", "red"],
                    tooltip=tooltip)

# we can also do interactive things with labels
label = ipywidgets.Label()

# note here that the calling sequence
# is a little different - instead
# of "change" we have "obj" and
# "element"
def printstuff(obj, element):
    # (1.1)
    #print(obj)
    #print(element)
    label.value = 'Media = ' + element['data']['media']

graph.on_element_click(printstuff)

fig = bqplot.Figure(marks = [graph])
ipywidgets.VBox([label,fig])_____no_output_____

# Activity #4: Network data - subset of facebook friends dataset
* from: https://snap.stanford.edu/data/egonets-Facebook.html
* dataset of friends lists

#### Info about this dataset:
* the original file you can read in has about 80,000 different connections
* it is ordered by the most connected person (person 0) at the top
* because this network would be computationally slow and just a hairball - we're going to be working with downsampled data
* for example, a file tagged "000090_000010" starts with the 10th most connected person, and only included connections up to the 90th most connected person
* Its worth noting that this dataset (linked here and on the webpage) also includes feature data like gender, last name, school, etc - however it is too sparse to be of visualization use to us

Check out the other social network links at the SNAP data webpage!_____no_output_____

# from 10 to 150 connections, a few large nodes
#filename = 'facebook_combined_sm000150_000010.txt'

# this might be too large: one large node, up to 100 connections
#filename='facebook_combined_sm000100.txt'

# start here
filename = 'facebook_combined_sm000090_000010.txt'

# then this one
#filename = 'facebook_combined_sm000030_000000.txt'
# note how different the topologies are

network = pd.read_csv('/Users/jillnaiman1/Downloads/'+filename,
                sep=' ', names=['ind1', 'ind2'])
network_____no_output_____# build the network
node_data = []
link_data = []
color_data = [] # all same color

# add nodes
maxNet = max([network['ind1'].max(),network['ind2'].max()])
for i in range(maxNet+1):
    node_data.append({"label": str(i), 'shape_attrs': {'r': 8} }) # small circles

# now, make links
for i in range(len(network)):
    # we are linking the ith object to another jth object, but we
    #  gotta figure out with jth object it is
    source_id = network.iloc[i]['ind1']
    target_id = network.iloc[i]['ind2']
    link_data.append({'source': source_id, 'target': target_id})
    color_data.append('blue')

#link_data,node_data
#color_data_____no_output_____# plot

graph = bqplot.Graph(node_data=node_data,
                     link_data = link_data,
                    colors=color_data)

# play with these for different graphs
graph.charge = -100
graph.link_type = 'line'
graph.link_distance=50
# there is no direction to links
graph.directed = False

fig = bqplot.Figure(marks = [graph])
fig.layout.min_width='1000px'
fig.layout.min_height='900px'
# note: I think this has to be the layout for this to look right
fig

# in theory, we could color this network by what school folks are in, or some such
#  but while the dataset does contain some of these features, the
#  answer rate is too sparse for our subset here_____no_output_____

# Note: the below is just prep if you want to make your own subset datasets_____no_output_____

# prep fb data by downsampling
minCon = 0
maxCon = 30
G = pd.read_csv('/Users/jillnaiman1/Downloads/facebook_combined.txt',sep=' ', names=['ind1', 'ind2'])
Gnew = np.zeros([2],dtype='int')
# loop and append
Gnew = G.loc[G['ind1']==minCon].values[0]
for i in xrange(G.loc[G['ind1']==minCon].index[0],len(G)):
    gl = G.loc[i].values
    if (gl[0] <= maxCon) and (gl[1] <= maxCon) and (gl[0] >= minCon) and (gl[1] >= minCon):
        Gnew = np.vstack((Gnew,gl))

np.savetxt('/Users/jillnaiman1/spring2019online/week09/data/facebook_combined_sm' + \
           str(maxCon).zfill(6) + '_' + str(minCon).zfill(6) + '.txt', Gnew,fmt='%i')_____no_output_____graph.link_distance_____no_output_____