# Notebook from nalomran/Bioinformatics-with-Python-Cookbook-Second-Edition
Path: Chapter02/Working_with_BAM.ipynb

# Getting the necessary data_____no_output_____You just need to do this only once_____no_output_____

!rm -f NA18489.chrom20.ILLUMINA.bwa.YRI.exome.20121211.bam 2>/dev/null
!rm -f NA18489.chrom20.ILLUMINA.bwa.YRI.exome.20121211.bam.bai 2>/dev/null
!wget ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/phase3/data/NA18489/exome_alignment/NA18489.chrom20.ILLUMINA.bwa.YRI.exome.20121211.bam
!wget ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/phase3/data/NA18489/exome_alignment/NA18489.chrom20.ILLUMINA.bwa.YRI.exome.20121211.bam.bai--2018-08-26 16:51:35--  ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/phase3/data/NA18489/exome_alignment/NA18489.chrom20.ILLUMINA.bwa.YRI.exome.20121211.bam
           => ‘NA18489.chrom20.ILLUMINA.bwa.YRI.exome.20121211.bam’
Resolving ftp.1000genomes.ebi.ac.uk (ftp.1000genomes.ebi.ac.uk)... 193.62.192.8
Connecting to ftp.1000genomes.ebi.ac.uk (ftp.1000genomes.ebi.ac.uk)|193.62.192.8|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /vol1/ftp/phase3/data/NA18489/exome_alignment ... done.
==> SIZE NA18489.chrom20.ILLUMINA.bwa.YRI.exome.20121211.bam ... 327067172
==> PASV ... done.    ==> RETR NA18489.chrom20.ILLUMINA.bwa.YRI.exome.20121211.bam ... done.
Length: 327067172 (312M) (unauthoritative)

NA18489.chrom20.ILL 100%[===================>] 311.92M  9.92MB/s    in 34s

2018-08-26 16:52:11 (9.18 MB/s) - ‘NA18489.chrom20.ILLUMINA.bwa.YRI.exome.20121211.bam’ saved [327067172]

--2018-08-26 16:52:11--  ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/phase3/data/NA18489/exome_alignment/NA18489.chrom20.ILLUMINA.bwa.YRI.exome.20121211.bam.bai
           => ‘NA18489.chrom20.ILLUMINA.bwa.YRI.exome.20121211.bam.bai’
Resolving ftp.1000genomes.ebi.ac.uk (ftp.1000genomes.ebi.ac.uk)... 193.62.192.8
Connecting to ftp.1000genomes.ebi.ac.uk (ftp.1000genomes.ebi.ac.uk)|193.62.192.8|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /vol1/ftp/phase3/data/NA18489/exome_alignment ... done.
==> SIZE NA18489.chrom20.ILLUMINA.bwa.YRI.exome.20121211.bam.bai ... 170688
==> PASV ... done.    ==> RETR NA18489.chrom20.ILLUMINA.bwa.YRI.exome.20121211.bam.bai ... done.
Length: 170688 (167K) (unauthoritative)

NA18489.chrom20.ILL 100%[===================>] 166.69K   403KB/s    in 0.4s

2018-08-26 16:52:13 (403 KB/s) - ‘NA18489.chrom20.ILLUMINA.bwa.YRI.exome.20121211.bam.bai’ saved [170688]

# The recipe_____no_output_____

#pip install pysam
from collections import defaultdict

import numpy as np

%matplotlib inline
import seaborn as sns
import matplotlib.pyplot as plt

import pysam/home/tiago/anaconda3/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/home/tiago/anaconda3/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
bam = pysam.AlignmentFile('NA18489.chrom20.ILLUMINA.bwa.YRI.exome.20121211.bam', 'rb')_____no_output_____headers = bam.header
for record_type, records in headers.items():
    print (record_type)
    for i, record in enumerate(records):
        if type(record) == dict:
            print('\t%d' % (i + 1))
            for field, value in record.items():
                print('\t\t%s\t%s' % (field, value))
        else:
            print('\t\t%s' % record)HD
		SO
		VN
SQ
	1
		SN	1
		M5	1b22b98cdeb4a9304cb5d48026a85128
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	249250621
	2
		SN	2
		M5	a0d9851da00400dec1098a9255ac712e
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	243199373
	3
		SN	3
		M5	fdfd811849cc2fadebc929bb925902e5
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	198022430
	4
		SN	4
		M5	23dccd106897542ad87d2765d28a19a1
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	191154276
	5
		SN	5
		M5	0740173db9ffd264d728f32784845cd7
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	180915260
	6
		SN	6
		M5	1d3a93a248d92a729ee764823acbbc6b
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	171115067
	7
		SN	7
		M5	618366e953d6aaad97dbe4777c29375e
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	159138663
	8
		SN	8
		M5	96f514a9929e410c6651697bded59aec
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	146364022
	9
		SN	9
		M5	3e273117f15e0a400f01055d9f393768
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	141213431
	10
		SN	10
		M5	988c28e000e84c26d552359af1ea2e1d
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	135534747
	11
		SN	11
		M5	98c59049a2df285c76ffb1c6db8f8b96
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	135006516
	12
		SN	12
		M5	51851ac0e1a115847ad36449b0015864
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	133851895
	13
		SN	13
		M5	283f8d7892baa81b510a015719ca7b0b
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	115169878
	14
		SN	14
		M5	98f3cae32b2a2e9524bc19813927542e
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	107349540
	15
		SN	15
		M5	e5645a794a8238215b2cd77acb95a078
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	102531392
	16
		SN	16
		M5	fc9b1a7b42b97a864f56b348b06095e6
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	90354753
	17
		SN	17
		M5	351f64d4f4f9ddd45b35336ad97aa6de
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	81195210
	18
		SN	18
		M5	b15d4b2d29dde9d3e4f93d1d0f2cbc9c
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	78077248
	19
		SN	19
		M5	1aacd71f30db8e561810913e0b72636d
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	59128983
	20
		SN	20
		M5	0dec9660ec1efaaf33281c0d5ea2560f
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	63025520
	21
		SN	21
		M5	2979a6085bfe28e3ad6f552f361ed74d
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	48129895
	22
		SN	22
		M5	a718acaa6135fdca8357d5bfe94211dd
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	51304566
	23
		SN	X
		M5	7e0e2e580297b7764e31dbc80c2540dd
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	155270560
	24
		SN	Y
		M5	1fa3474750af0948bdf97d5a0ee52e51
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	59373566
	25
		SN	MT
		M5	c68f52674c9fb33aef52dcf399755519
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	16569
	26
		SN	GL000207.1
		M5	f3814841f1939d3ca19072d9e89f3fd7
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	4262
	27
		SN	GL000226.1
		M5	1c1b2cd1fccbc0a99b6a447fa24d1504
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	15008
	28
		SN	GL000229.1
		M5	d0f40ec87de311d8e715b52e4c7062e1
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	19913
	29
		SN	GL000231.1
		M5	ba8882ce3a1efa2080e5d29b956568a4
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	27386
	30
		SN	GL000210.1
		M5	851106a74238044126131ce2a8e5847c
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	27682
	31
		SN	GL000239.1
		M5	99795f15702caec4fa1c4e15f8a29c07
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	33824
	32
		SN	GL000235.1
		M5	118a25ca210cfbcdfb6c2ebb249f9680
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	34474
	33
		SN	GL000201.1
		M5	dfb7e7ec60ffdcb85cb359ea28454ee9
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	36148
	34
		SN	GL000247.1
		M5	7de00226bb7df1c57276ca6baabafd15
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	36422
	35
		SN	GL000245.1
		M5	89bc61960f37d94abf0df2d481ada0ec
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	36651
	36
		SN	GL000197.1
		M5	6f5efdd36643a9b8c8ccad6f2f1edc7b
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	37175
	37
		SN	GL000203.1
		M5	96358c325fe0e70bee73436e8bb14dbd
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	37498
	38
		SN	GL000246.1
		M5	e4afcd31912af9d9c2546acf1cb23af2
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	38154
	39
		SN	GL000249.1
		M5	1d78abec37c15fe29a275eb08d5af236
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	38502
	40
		SN	GL000196.1
		M5	d92206d1bb4c3b4019c43c0875c06dc0
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	38914
	41
		SN	GL000248.1
		M5	5a8e43bec9be36c7b49c84d585107776
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	39786
	42
		SN	GL000244.1
		M5	0996b4475f353ca98bacb756ac479140
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	39929
	43
		SN	GL000238.1
		M5	131b1efc3270cc838686b54e7c34b17b
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	39939
	44
		SN	GL000202.1
		M5	06cbf126247d89664a4faebad130fe9c
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	40103
	45
		SN	GL000234.1
		M5	93f998536b61a56fd0ff47322a911d4b
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	40531
	46
		SN	GL000232.1
		M5	3e06b6741061ad93a8587531307057d8
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	40652
	47
		SN	GL000206.1
		M5	43f69e423533e948bfae5ce1d45bd3f1
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	41001
	48
		SN	GL000240.1
		M5	445a86173da9f237d7bcf41c6cb8cc62
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	41933
	49
		SN	GL000236.1
		M5	fdcd739913efa1fdc64b6c0cd7016779
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	41934
	50
		SN	GL000241.1
		M5	ef4258cdc5a45c206cea8fc3e1d858cf
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	42152
	51
		SN	GL000243.1
		M5	cc34279a7e353136741c9fce79bc4396
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	43341
	52
		SN	GL000242.1
		M5	2f8694fc47576bc81b5fe9e7de0ba49e
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	43523
	53
		SN	GL000230.1
		M5	b4eb71ee878d3706246b7c1dbef69299
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	43691
	54
		SN	GL000237.1
		M5	e0c82e7751df73f4f6d0ed30cdc853c0
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	45867
	55
		SN	GL000233.1
		M5	7fed60298a8d62ff808b74b6ce820001
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	45941
	56
		SN	GL000204.1
		M5	efc49c871536fa8d79cb0a06fa739722
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	81310
	57
		SN	GL000198.1
		M5	868e7784040da90d900d2d1b667a1383
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	90085
	58
		SN	GL000208.1
		M5	aa81be49bf3fe63a79bdc6a6f279abf6
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	92689
	59
		SN	GL000191.1
		M5	d75b436f50a8214ee9c2a51d30b2c2cc
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	106433
	60
		SN	GL000227.1
		M5	a4aead23f8053f2655e468bcc6ecdceb
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	128374
	61
		SN	GL000228.1
		M5	c5a17c97e2c1a0b6a9cc5a6b064b714f
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	129120
	62
		SN	GL000214.1
		M5	46c2032c37f2ed899eb41c0473319a69
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	137718
	63
		SN	GL000221.1
		M5	3238fb74ea87ae857f9c7508d315babb
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	155397
	64
		SN	GL000209.1
		M5	f40598e2a5a6b26e84a3775e0d1e2c81
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	159169
	65
		SN	GL000218.1
		M5	1d708b54644c26c7e01c2dad5426d38c
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	161147
	66
		SN	GL000220.1
		M5	fc35de963c57bf7648429e6454f1c9db
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	161802
	67
		SN	GL000213.1
		M5	9d424fdcc98866650b58f004080a992a
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	164239
	68
		SN	GL000211.1
		M5	7daaa45c66b288847b9b32b964e623d3
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	166566
	69
		SN	GL000199.1
		M5	569af3b73522fab4b40995ae4944e78e
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	169874
	70
		SN	GL000217.1
		M5	6d243e18dea1945fb7f2517615b8f52e
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	172149
	71
		SN	GL000216.1
		M5	642a232d91c486ac339263820aef7fe0
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	172294
	72
		SN	GL000215.1
		M5	5eb3b418480ae67a997957c909375a73
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	172545
	73
		SN	GL000205.1
		M5	d22441398d99caf673e9afb9a1908ec5
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	174588
	74
		SN	GL000219.1
		M5	f977edd13bac459cb2ed4a5457dba1b3
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	179198
	75
		SN	GL000224.1
		M5	d5b2fc04f6b41b212a4198a07f450e20
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	179693
	76
		SN	GL000223.1
		M5	399dfa03bf32022ab52a846f7ca35b30
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	180455
	77
		SN	GL000195.1
		M5	5d9ec007868d517e73543b005ba48535
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	182896
	78
		SN	GL000212.1
		M5	563531689f3dbd691331fd6c5730a88b
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	186858
	79
		SN	GL000222.1
		M5	6fe9abac455169f50470f5a6b01d0f59
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	186861
	80
		SN	GL000200.1
		M5	75e4c8d17cd4addf3917d1703cacaf25
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	187035
	81
		SN	GL000193.1
		M5	dbb6e8ece0b5de29da56601613007c2a
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	189789
	82
		SN	GL000194.1
		M5	6ac8f815bf8e845bb3031b73f812c012
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	191469
	83
		SN	GL000225.1
		M5	63945c3e6962f28ffd469719a747e73c
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	211173
	84
		SN	GL000192.1
		M5	325ba9e808f669dfeee210fdd7b470ac
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	547496
	85
		SN	NC_007605
		M5	6743bd63b3ff2b5b8985d8933c53290a
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	171823
	86
		SN	hs37d5
		M5	5b6a4b3a81a2d3c134b7d14bf6ad39f1
		UR	ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz        AS:NCBI37       SP:Human
		LN	35477943
RG
	1
		SM	NA18489
		LB	Solexa-51039
		DS	SRP004074
		ID	SRR100025
		PI	220
		CN	BI
		PL	ILLUMINA
PG
	1
		VN	0.5.9-r16
		ID	bwa_index
		PN	bwa
		CL	bwa index -a bwtsw $reference_fasta
	2
		PP	bwa_index
		VN	0.5.9-r16
		ID	bwa_aln_fastq
		PN	bwa
		CL	bwa aln -q 15 -f $sai_file $reference_fasta $fastq_file
	3
		PP	bwa_aln_fastq
		VN	0.5.9-r16
		ID	bwa_sam
		PN	bwa
		CL	bwa sampe -a 660 -r $rg_line -f $sam_file $reference_fasta $sai_file(s) $fastq_file(s)
	4
		PP	bwa_sam
		VN	0.1.17 (r973:277)
		ID	sam_to_fixed_bam
		PN	samtools
		CL	samtools view -bSu $sam_file | samtools sort -n -o - samtools_nsort_tmp | samtools fixmate /dev/stdin /dev/stdout | samtools sort -o - samtools_csort_tmp | samtools fillmd -u - $reference_fasta > $fixed_bam_file
	5
		PP	sam_to_fixed_bam
		VN	1.2-29-g0acaf2d
		ID	gatk_target_interval_creator
		PN	GenomeAnalysisTK
		CL	java $jvm_args -jar GenomeAnalysisTK.jar -T RealignerTargetCreator -R $reference_fasta -o $intervals_file -known $known_indels_file(s)
	6
		PP	gatk_target_interval_creator
		VN	1.2-29-g0acaf2d
		ID	bam_realignment_around_known_indels
		PN	GenomeAnalysisTK
		CL	java $jvm_args -jar GenomeAnalysisTK.jar -T IndelRealigner -R $reference_fasta -I $bam_file -o $realigned_bam_file -targetIntervals $intervals_file -known $known_indels_file(s) -LOD 0.4 -model KNOWNS_ONLY -compress 0 --disable_bam_indexing
	7
		PP	bam_realignment_around_known_indels
		VN	1.2-29-g0acaf2d
		ID	bam_count_covariates
		PN	GenomeAnalysisTK
		CL	java $jvm_args -jar GenomeAnalysisTK.jar -T CountCovariates -R $reference_fasta -I $bam_file -recalFile $bam_file.recal_data.csv -knownSites $known_sites_file(s) -l INFO -L '1;2;3;4;5;6;7;8;9;10;11;12;13;14;15;16;17;18;19;20;21;22;X;Y;MT' -cov ReadGroupCovariate -cov QualityScoreCovariate -cov CycleCovariate -cov DinucCovariate
	8
		PP	bam_count_covariates
		VN	1.2-29-g0acaf2d
		ID	bam_recalibrate_quality_scores
		PN	GenomeAnalysisTK
		CL	java $jvm_args -jar GenomeAnalysisTK.jar -T TableRecalibration -R $reference_fasta -recalFile $bam_file.recal_data.csv -I $bam_file -o $recalibrated_bam_file -l INFO -compress 0 --disable_bam_indexing
	9
		PP	bam_recalibrate_quality_scores
		VN	0.1.17 (r973:277)
		ID	bam_calculate_bq
		PN	samtools
		CL	samtools calmd -Erb $bam_file $reference_fasta > $bq_bam_file
	10
		PP	bam_calculate_bq
		VN	1.53
		ID	bam_merge
		PN	picard
		CL	java $jvm_args -jar MergeSamFiles.jar INPUT=$bam_file(s) OUTPUT=$merged_bam VALIDATION_STRINGENCY=SILENT
	11
		PP	bam_merge
		VN	1.53
		ID	bam_mark_duplicates
		PN	picard
		CL	java $jvm_args -jar MarkDuplicates.jar INPUT=$bam_file OUTPUT=$markdup_bam_file ASSUME_SORTED=TRUE METRICS_FILE=/dev/null VALIDATION_STRINGENCY=SILENT
	12
		PP	bam_mark_duplicates
		VN	1.53
		ID	bam_merge.1
		PN	picard
		CL	java $jvm_args -jar MergeSamFiles.jar INPUT=$bam_file(s) OUTPUT=$merged_bam VALIDATION_STRINGENCY=SILENT
CO
		$known_indels_file(s) = ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_mapping_resources/ALL.wgs.indels_mills_devine_hg19_leftAligned_collapsed_double_hit.indels.sites.vcf.gz
		$known_indels_file(s) .= ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_mapping_resources/ALL.wgs.low_coverage_vqsr.20101123.indels.sites.vcf.gz
		$known_sites_file(s) = ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_mapping_resources/ALL.wgs.dbsnp.build135.snps.sites.vcf.gz
#0-based
for rec in bam:
    if rec.cigarstring.find('M') > -1 and rec.cigarstring.find('S') > -1 and not rec.is_unmapped and not rec.mate_is_unmapped:
        break
print(rec.query_name, rec.reference_id, bam.getrname(rec.reference_id), rec.reference_start, rec.reference_end)
print(rec.cigarstring)
print(rec.query_alignment_start, rec.query_alignment_end, rec.query_alignment_length)
print(rec.next_reference_id, rec.next_reference_start, rec.template_length)
print(rec.is_paired, rec.is_proper_pair, rec.is_unmapped, rec.mapping_quality)
print(rec.query_qualities)
print(rec.query_alignment_qualities)
print(rec.query_sequence)SRR100025.62130839 19 20 59996 60048
52M24S
0 52 52
19 60228 295
True True False 60
array('B', [33, 34, 36, 33, 39, 34, 33, 38, 39, 34, 40, 35, 40, 40, 32, 40, 38, 33, 35, 38, 33, 39, 40, 34, 37, 39, 36, 30, 36, 37, 34, 35, 34, 40, 37, 34, 38, 28, 40, 40, 38, 32, 33, 32, 36, 34, 37, 24, 34, 35, 31, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])
array('B', [33, 34, 36, 33, 39, 34, 33, 38, 39, 34, 40, 35, 40, 40, 32, 40, 38, 33, 35, 38, 33, 39, 40, 34, 37, 39, 36, 30, 36, 37, 34, 35, 34, 40, 37, 34, 38, 28, 40, 40, 38, 32, 33, 32, 36, 34, 37, 24, 34, 35, 31, 2])
CTCAGATCCAGAGGTGGAAGAGGAAGGAAGCTTGGAACCCTATAGAGTTGCTGAGTGCCAGGACCAGATACTGGGC
counts = [0] * 76
for n, rec in enumerate(bam.fetch('20', 0, 10000000)):
    for i in range(rec.query_alignment_start, rec.query_alignment_end):
        counts[i] += 1
freqs = [x / (n + 1.) for x in counts]
fig, ax = plt.subplots(figsize=(16,9))
ax.plot(range(1, 77), freqs)_____no_output_____phreds = defaultdict(list)
for rec in bam.fetch('20', 0, None):
    for i in range(rec.query_alignment_start, rec.query_alignment_end):
        phreds[i].append(rec.query_qualities[i])_____no_output_____maxs = [max(phreds[i]) for i in range(76)]
tops = [np.percentile(phreds[i], 95) for i in range(76)]
medians = [np.percentile(phreds[i], 50) for i in range(76)]
bottoms = [np.percentile(phreds[i], 5) for i in range(76)]
medians_fig = [x - y for x, y in zip(medians, bottoms)]
tops_fig = [x - y for x, y in zip(tops, medians)]
maxs_fig = [x - y for x, y in zip(maxs, tops)]_____no_output_____fig, ax = plt.subplots(figsize=(16,9))
ax.stackplot(range(1, 77), (bottoms, medians_fig, tops_fig, maxs_fig))
ax.plot(range(1, 77), maxs, 'k-')_____no_output_____


# Notebook from yscyang1/Thinkful_Data_Science
Path: Exercises/Data_Visualization/2 - Challenge - Data Cleaning.ipynb

### Task
1.  Determine 5 most common journals
    1. Find total articles for each
2.  Calculate mean, median, std dev of open access cost per article for each journal
3.  Bonus Round:  identify open access prices paid by subj area_____no_output_____#### Import libraries and data_____no_output_____

import seaborn as sns
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

import re
import string_____no_output_____df = pd.read_csv('WELLCOME_APCspend2013_forThinkful.csv', encoding = 'Latin-1')_____no_output_____df.describe()_____no_output_____df.head()_____no_output_____

#### Missing Info
About 10% missing from PMID/PMCID column.  But we are mostly concerned with the journal titles, article titles, and cost.  Of these, only one 0.05% (1 journal title) is missing.  _____no_output_____

for col in df.columns:
    print('{}:  {}'.format(col, df[col].isnull().sum()/df[col].shape[0]*100))PMID/PMCID:  9.35590032910202
Publisher:  0.0
Journal title:  0.04701457451810061
Article title:  0.0
COST (£) charged to Wellcome (inc VAT when charged):  0.0

### Determine 5 most common journals_____no_output_____From the raw data, there are ~985 unique journal titles.  But looking at the first 10 journal titles in alphabetical order, we see that ACS NANO and ACS Nano are the same journal, but categorized as different titles.  A similar situation arises with PLoS One and PLoS ONE, seen below in the top 15 most common journals as seen in the raw data.  An easy way to fix this is to make all letters lowercase and strip the white space.  Other small things to decrease repeated journals is to remove punctuation, convert '&' to 'and', and convert '1' to 'one'.
 _____no_output_____

df['Journal title'].nunique()_____no_output_____sorted(df[df['Journal title'].notnull()]['Journal title'].unique())[:10]_____no_output_____df['Journal title'].value_counts()[:15]_____no_output_____def initial_conversion(col):
# Convert string to lower case and remove white space
# Convert '&' to 'and'
# Convert '1', to 'one
# Remove punctuation
    try:
        col = col.lower().strip()
        col = col.replace('&', 'and')
        col = col.replace('1', 'one')
        col = col.translate(str.maketrans('', '', string.punctuation))
        return col
    except AttributeError:
        pass_____no_output_____df['Journal title'] = df['Journal title'].apply(initial_conversion)_____no_output_____df['Journal title'].nunique()_____no_output_____

Purely converting all titles to lowercase and stripping of white space reduces the number of unique journals by ~100.  The other manipulations decreased the count by 10.

Below I've looked at the top 15 most common journal titles to see where to concentrate most of my efforts to find the 5 most common journals.  _____no_output_____

df['Journal title'].value_counts()[:15]_____no_output_____

#### Plos one
Plos one is by far the most popular journal so far.  plos is a pretty unique spelling, so lets see if we can find other variations of plos one._____no_output_____

for j in df[df['Journal title'].notnull()]['Journal title'].unique():
    if 'plos' in j:
        print(j)plos computational biology
plos genetics
plos ntd
plos one
plos pathogens
plos
plos  computational biology
plos  one
plos biology
plos medicine
plos medicine journal
plos negected tropical diseases
plos neglected tropical diseases
plosone

From the above list, I see 'plosone' and 'plos  one'.  Fixing these typos increases the number from 197 to 207.  _____no_output_____

def fix_plos_one(col):
    try:
        if col == 'plosone':
            return 'plos one'
        elif col == 'plos  one':
            return 'plos one'
        else:
            return col
    except AttributeError:
        pass_____no_output_____df['Journal title'] = df['Journal title'].apply(fix_plos_one)_____no_output_____

#### Journal of biological chemistry
Biology, bio, chem, and biochem are all very common phrases.  I had to go through the alphabetized list by eye to pick out the following variations.  Fixing these variations increased the number from 52 to 59._____no_output_____*List of journal of biological chemistry abbreviations:*
'j biol chem.'
'j biol chemistry'
'jnl biological chemistry'
'journal biological chemistry'
'journal of biological chemistry'
'journal of biol chem'
'journal of biological chemistry'  _____no_output_____

def fix_bio_chem(col):
    try:
        if col == 'j biol chem.':
            return 'journal of biological chemistry'
        elif col == 'j biol chemistry':
            return 'journal of biological chemistry'
        elif col == 'jnl biological chemistry':
            return 'journal of biological chemistry'
        elif col == 'journal biological chemistry':
            return 'journal of biological chemistry'
        elif col == 'journal of biol chem':
            return 'journal of biological chemistry'
        else:
            return col
    except AttributeError:
        pass_____no_output_____df['Journal title'] = df['Journal title'].apply(fix_bio_chem)_____no_output_____

#### Nucleic acid research
I only found one misspelling of 'nucleic acid research'.  Fixing this increased the count from 26 to 29.  _____no_output_____

for j in df[df['Journal title'].notnull()]['Journal title'].unique():
    if 'nucleic' in j:
        print(j)nucleic acids research
nucleic acid research
def fix_nar(col):
    try:
        if col == 'nucleic acid research':
            return 'nucleic acids research'
        else:
            return col
    except AttributeError:
        pass_____no_output_____df['Journal title'] = df['Journal title'].apply(fix_nar)_____no_output_____

#### Proceedings of the national academy of sciences
This is another one that required scanning by eye to find all the variations, which is listed as follows:

*List of Proceedings of the national academy of sciences abbreviations:*
'pnas'
'pnas (proceedings of the national academy of sciences of the united states of america'
'pnas usa'
'proc natl acad sci u s a.'
'proceddings of the national academy of sciences of usa'
'proceedings of the national academy of sciences'
'proceedings of the national academy of sciences (pnas)'
'proceedings of the national academy of sciences of the united states of america'
'proceedings of the national academy of sciences of the usa'

Fixing these abbreviations increased the number from 22 to 34.  _____no_output_____

def fix_pnas(col):
    try:
        if col == 'pnas':
            return 'proceedings of the national academy of sciences'
        elif col == 'pnas (proceedings of the national academy of sciences of the united states of america':
            return 'proceedings of the national academy of sciences'
        elif col == 'pnas usa':
            return 'proceedings of the national academy of sciences'
        elif col == 'proc natl acad sci u s a.':
            return 'proceedings of the national academy of sciences'
        elif col == 'proceddings of the national academy of sciences of usa':
            return 'proceedings of the national academy of sciences'
        elif col == 'proceedings of the national academy of sciences (pnas)':
            return 'proceedings of the national academy of sciences'
        elif col == 'proceedings of the national academy of sciences of the united states of america':
            return 'proceedings of the national academy of sciences'
        elif col == 'proceedings of the national academy of sciences of the usa':
            return 'proceedings of the national academy of sciences'
        else:
            return col
    except AttributeError:
        pass_____no_output_____df['Journal title'] = df['Journal title'].apply(fix_pnas)_____no_output_____

#### Answer:
top 5 journals are:  _____no_output_____

df['Journal title'].value_counts()[:5]_____no_output_____

As expected, plos one by far was the most frequent journal with almost 200 articles.  The next most frequent journal was the Journal of Biological Chemistry, which had ~70 articles.  The rest were between 25 and 35 articles.  _____no_output_____

top_five = df[(df['Journal title'] == df['Journal title'].value_counts()[:1].index[0]) |
  (df['Journal title'] == df['Journal title'].value_counts()[1:2].index[0]) |
  (df['Journal title'] == df['Journal title'].value_counts()[2:3].index[0]) |
  (df['Journal title'] == df['Journal title'].value_counts()[3:4].index[0]) |
  (df['Journal title'] == df['Journal title'].value_counts()[4:5].index[0]) ]_____no_output_____answer1 = pd.DataFrame()
answer1['Journal title'] = top_five['Journal title'].value_counts().index
answer1['Title count'] = top_five['Journal title'].value_counts().values_____no_output_____sns.barplot(x = 'Title count', y = 'Journal title',  data = answer1)
plt.title('5 Most Frequent Journals and Number of Articles')
plt.ylabel('Journal')
plt.xlabel('Title Count')_____no_output_____

### Mean, Median, Std Dev of Cost for per article for each journal
To find the cost per article, I had to do some string manipulation because there is a '£' symbol.  Then, to plot it in a histogram, I needed to convert it to a float._____no_output_____

# Remove '£' symbol and convert to float
top_five['COST (£) charged to Wellcome (inc VAT when charged)'] = top_five['COST (£) charged to Wellcome (inc VAT when charged)'].apply(lambda x:  float(x[1:]))C:\Users\Crystal\AppData\Local\Continuum\Anaconda3\lib\site-packages\ipykernel\__main__.py:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  if __name__ == '__main__':

Below are the histograms of the costs.  On the left is the cost distribution for all articles.  As you can see, some of the prices get ridiculously high, at 1 million pounds.  To the right is a cost distribution for all articles under £25000.  As you can see, the highest cost in this figure is ~£3000, so there aren't any articles that cost between £3000 and £25000.  From this plot, we can see that most of the articles cost ~£800. _____no_output_____

fig, (axis1, axis2) = plt.subplots(1,2,figsize=(15,4))

sns.distplot(top_five['COST (£) charged to Wellcome (inc VAT when charged)'], ax = axis1)
sns.distplot(top_five[top_five['COST (£) charged to Wellcome (inc VAT when charged)'] < 25000]['COST (£) charged to Wellcome (inc VAT when charged)'], ax = axis2)

axis1.set_title('Price Distribution For All Data')
axis2.set_title('Price Distribution Under £25000')

for ax in [axis1, axis2]:
    ax.set_xlabel('Cost (£)')
    ax.set_ylabel('Density')_____no_output_____

Instead of removing these outlier prices, I'll try a 90% winsorization on them.  The data now ranges between £600 and £2500._____no_output_____

import scipy.stats_____no_output_____top_five['COST (£) charged to Wellcome (inc VAT when charged)'] = scipy.stats.mstats.winsorize(top_five['COST (£) charged to Wellcome (inc VAT when charged)'], limits = [0.05, 0.05])C:\Users\Crystal\AppData\Local\Continuum\Anaconda3\lib\site-packages\ipykernel\__main__.py:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  if __name__ == '__main__':
sns.distplot(top_five['COST (£) charged to Wellcome (inc VAT when charged)'])
plt.title('Winsorized Data')
plt.xlabel('Cost (£)')
plt.ylabel('Density')_____no_output_____

I can now find the mean, median, and standard deviations per article for each journal, seen below._____no_output_____

plos_one_mean = top_five[top_five['Journal title'] == 'plos one']['COST (£) charged to Wellcome (inc VAT when charged)'].mean()
plos_one_median = top_five[top_five['Journal title'] == 'plos one']['COST (£) charged to Wellcome (inc VAT when charged)'].median()
plos_one_std = top_five[top_five['Journal title'] == 'plos one']['COST (£) charged to Wellcome (inc VAT when charged)'].std()_____no_output_____jbc_mean = top_five[top_five['Journal title'] == 'journal of biological chemistry']['COST (£) charged to Wellcome (inc VAT when charged)'].mean()
jbc_median = top_five[top_five['Journal title'] == 'journal of biological chemistry']['COST (£) charged to Wellcome (inc VAT when charged)'].median()
jbc_std = top_five[top_five['Journal title'] == 'journal of biological chemistry']['COST (£) charged to Wellcome (inc VAT when charged)'].std()_____no_output_____pnas_mean = top_five[top_five['Journal title'] == 'proceedings of the national academy of sciences']['COST (£) charged to Wellcome (inc VAT when charged)'].mean()
pnas_median = top_five[top_five['Journal title'] == 'proceedings of the national academy of sciences']['COST (£) charged to Wellcome (inc VAT when charged)'].median()
pnas_std = top_five[top_five['Journal title'] == 'proceedings of the national academy of sciences']['COST (£) charged to Wellcome (inc VAT when charged)'].std()_____no_output_____neural_mean = top_five[top_five['Journal title'] == 'neuroimage']['COST (£) charged to Wellcome (inc VAT when charged)'].mean()
neural_median = top_five[top_five['Journal title'] == 'neuroimage']['COST (£) charged to Wellcome (inc VAT when charged)'].median()
neural_std = top_five[top_five['Journal title'] == 'neuroimage']['COST (£) charged to Wellcome (inc VAT when charged)'].std()_____no_output_____nar_mean = top_five[top_five['Journal title'] == 'nucleic acids research']['COST (£) charged to Wellcome (inc VAT when charged)'].mean()
nar_median = top_five[top_five['Journal title'] == 'nucleic acids research']['COST (£) charged to Wellcome (inc VAT when charged)'].median()
nar_std = top_five[top_five['Journal title'] == 'nucleic acids research']['COST (£) charged to Wellcome (inc VAT when charged)'].std()_____no_output_____top_five_stats = pd.DataFrame(np.array([[plos_one_mean, plos_one_median, plos_one_std],
                                      [jbc_mean, jbc_median, jbc_std],
                                      [pnas_mean, pnas_median, pnas_std],
                                      [neural_mean, neural_median, neural_std],
                                      [nar_mean, nar_median, nar_std]]),
                             columns = ['Mean', 'Median', 'Standard Deviation'],
                             index = ['plos one', 'journal of biological chemistry','pnas', 'neuralimage', 'nucleic acids research'])_____no_output_____top_five_stats_____no_output_____

Here, we see the journal with the cheapest per article is from pnas, with a mean at ~£835.  The most expensive (and with the lowest standard deviation) is from neuralimage, with a mean of ~£2200.  _____no_output_____### Bonus Round: identify open access prices paid by subject area
I've chosen to not do the bonus round for now.  Next time..._____no_output_____


# Notebook from mgrubisic/coronavirus-2020
Path: archive/model2.ipynb

# Coronavirus outbreak 2020

- Taken world-wide data from

- Source: https://www.worldometers.info/coronavirus/

- Fit growth model of type $t^p$ for infections and deaths

- Predict infection and deaths for the next day based on that model

- starting 21 January
_____no_output_____

%%file data.txt
date,infected,deaths
21/jan/2020,446,9
22/jan/2020,579,17
23/jan/2020,844,25
24/jan/2020,1312,41
25/jan/2020,2015,56
26/jan/2020,2801,80
27/jan/2020,4579,106
28/jan/2020,6061,132
29/jan/2020,7816,170
30/jan/2020,9821,213
31/jan/2020,11948,258
01/feb/2020,14551,304
02/feb/2020,17387,362
03/feb/2020,20626,426
04/feb/2020,24553,492
05/feb/2020,28276,565
06/feb/2020,31439,638
07/feb/2020,34875,724
08/feb/2020,37552,813
09/feb/2020,40553,910
10/feb/2020,43099,1018
11/feb/2020,44919,1114
12/feb/2020,60327,1369
13/feb/2020,65246,1491
14/feb/2020,67100,1526
15/feb/2020,69197,1669
16/feb/2020,71329,1775
17/feb/2020,73332,1873
18/feb/2020,75184,2009
19/feb/2020,75700,2126
20/feb/2020,76676,2247
21/feb/2020,77673,2360
22/feb/2020,78651,2460
23/feb/2020,79205,2618
24/feb/2020,80088,2699Overwriting data.txt
%config InlineBackend.figure_formats = ['svg']
import pandas as pd
import scipy.optimize
import numpy as np
import seaborn
import datetime
import matplotlib
import matplotlib.pyplot as plt
seaborn.set()
%matplotlib inline
!mkdir -p figures
figsize = (8, 6)   # need bigger size otherwise xlabels are cut-off in svg files_____no_output_____data = pd.read_csv('data.txt',
                   parse_dates=[0],
                   infer_datetime_format=True)
print(f"Last executed: {datetime.date.today().strftime('%d %b %Y')}, "\
      f"last data point from {max(data['date']).strftime('%d %b %Y')}.")
# remove last 2 rows or debugging
# data = data.iloc[:-2]Last executed: 25 Feb 2020, last data point from 24 Feb 2020.
fig, ax = plt.subplots(figsize=figsize)
data.plot('date', 'infected', style='og', ax=ax);
ax.get_figure().savefig('figures/infections.svg')_____no_output_____fig, ax = plt.subplots(figsize=figsize)
data.plot('date', 'deaths', style='og', ax=ax)
ax.get_figure().savefig('figures/deaths.svg')_____no_output_____

Compute some more useful columns_____no_output_____

data['days-td'] = data['date'] - data['date'][0]  # td = TimeDelta
data['days'] = data['days-td'].astype("timedelta64[D]").astype(float)
data['new-infections'] = data['infected'].diff()
data['new-deaths'] = data['deaths'].diff()
# grow factor (see https://www.worldometers.info/coronavirus/coronavirus-cases/#cases-growth-factor)
data['growth-infected'] = data['infected'].pct_change()+1
data['growth-deaths'] = data['deaths'].pct_change()+1
data.to_html('figures/table-1.html')
data_____no_output_____fig, ax = plt.subplots(figsize=figsize)
ax.bar(data['date'], data['new-deaths'])
ax.xaxis.set_major_locator(matplotlib.dates.DayLocator())
ax.xaxis.set_major_formatter(matplotlib.dates.DateFormatter('%b %d'))
ax.tick_params(axis='x', rotation=90)
ax.set_title("New deaths per day")
fig.savefig('figures/new-deaths.svg')_____no_output_____fig, ax = plt.subplots(figsize=figsize)
ax.bar(data['date'], data['new-infections'])
ax.xaxis.set_major_locator(matplotlib.dates.DayLocator())
ax.xaxis.set_major_formatter(matplotlib.dates.DateFormatter('%b %d'))
ax.tick_params(axis='x', rotation=90)
ax.set_title("New infections per day")
fig.savefig('figures/new-infections.svg')_____no_output_____

Update 5 February: The number of new infections on 05 Feb was smaller than that on 04 Feb. This could be the beginning of a slow down in spread.

Update 6 February: the trend continues.

It will take some time until this is reflected in the daily number of deaths.

Update 12 Feb: Looks like the reporting / identification of cases and deaths was saturated before._____no_output_____

def day_to_date(day):
    """Given the day from day0 (21 Jan 2020), return the data as a string.
    Example output: '26 Jan 2020'

    Needed to obtain days for predictions into the future, which are not given in our data frame 'data' yet.
    """
    assert isinstance(day, (int, float)), day

    # Compute date
    date = pd.datetime(year=2020, month=1, day=21) + pd.Timedelta(day, "D")
    # format nicely
    s = date.strftime("%d %b %Y")
    return s_____no_output_____

# Use a trivial growth model to represent the data

## Model 1_____no_output_____We assume a model of type $n(t) = c(t-t_0)^p + a_0$ to describe the growth of the number of infections (or deaths) $n$ as a function of days $t$, and we start counting days from 21 Jan 2020 as day0. The choice of day0 is somewhat random; it was the first data point shown in the data at https://www.worldometers.info/coronavirus/._____no_output_____

def model1(t, c, t0, p, a0):
    return c*(t-t0)**p + a0    _____no_output_____

The parameters t0 and a0 are relatively uninteresting; they help to fit the data better but are unimportant for the growth behaviour. The key parameters are the exponent $p$ and then the prefactor $c$ in $n(t) = c t^p$.

If $p=2$, we know that the number of infections grows quadratically as a function of time. That means for example that the change from 5 days after day0 to 10 days after day0 (i.e. double it), the number $n$ will have increase by a factor of $2^2 = 4$.

For larger values of $p$, the growth will be faster.

The number $n$ will only go down, if $p$ is smaller than 1._____no_output_____We use `curve_fit` from `scipy.optimize` to determine parameters._____no_output_____## Why this model?
_____no_output_____- it is a relatively simple model
- it fits the data so far (~10 days from 21 Jan onwards) well
- model allows predictions for infections and deaths *if the spread of the virus carries on as it has so far*
- We thus very much hope that the model stops working, and overestimates infections and deaths

Update: 10 Feb
- the model deviates from reported numbers from about 7 Feb: reported infections are smaller than predicted
- this could be a good sign, or a change in how tow infections are classified, or saturation of testing capabilities or something else.

Update: 12 Feb
- clearly the data reported up to 12 Feb was inaccurate (for whatever reason)
- To account for this (and make the curve_fitting for the simple model possible), we take a subset of data points, with focus on those we might consider most accurate. This includes 12 Feb and a few days at the beginning (21 Jan to 24 Jan). It's just a guess of what might work
_____no_output_____## Model 2

We use a sigmoidal function (https://en.wikipedia.org/wiki/Sigmoid_function). This often works for processes that start from zero, then accelerate towards a maximum and the decrease to zero again. It is a guess of course.

_____no_output_____

def model2(t, t0, nmax, c):
    """t0 - days to maximum increase
    nmax - final number
    c - steepness: c=0 is linear function , c >> 1 is step function"""
    return nmax/(1 + np.exp(-c*(t-t0)))_____no_output_____# t = np.linspace(0, 50, 100)
# y = model2(t, 25, 100, 1)
# import pylab
# pylab.plot(t, y, 'o')_____no_output_____
# subset of data points used in fit
if False:
    dates_used = ['2020-01-21', '2020-01-22', '2020-01-23', '2020-01-24', '2020-02-12', '2020-02-13', '2020-02-14',
                 '2020-02-15']
    dates_used_string = str(dates_used)

    # Take a gamble: numbers from 14 Feb look underreported; ignore that data point for now in fits
    # dates_used = ['2020-01-21', '2020-01-22', '2020-01-23', '2020-01-24', '2020-02-12', '2020-02-13']

    data2 = data[data['date'].isin(dates_used)]
else:
    dates_used_string = 'data from all days from 2020-01-21 to ' + str(max(data['date']))[0:10]
    data2 = data
dates_used_string_____no_output_____

# Global infections_____no_output_____

days = data['days']
days2 = data2['days']
infections = data2['infected']
param_infections, cov_infections = \
    scipy.optimize.curve_fit(model2, days2,
                             infections,
                             p0=(25, 100000, 0.5))  # guess initial values for fit parameters_____no_output_____def pretty_params(param, end="\n"):
    c, t0, p, a0 = param
    s = "p = {:.3f}".format(p) + end
    s += "c = {:.3g}".format(c) + end
    s += "t0= {:.3f}".format(t0) + end
    s += "a0= {:.3f}".format(a0)
    return s_____no_output_____def pretty_params2(param, end="\n"):
    t0, nmax, c = param
    s = "t0    = {:.3f}".format(t0) + end
    s += "nmax = {:.1f}".format(nmax) + end
    s += "c.   = {:.3f}".format(c) + end
    return s_____no_output_____t0, nmax, c = param_infections
fitted_infections = model2(days, t0, nmax, c)
print(pretty_params2(param_infections))t0    = 18.574
nmax = 83727.1
c.   = 0.219

# Estimated standard deviation of parameters
with np.printoptions(precision=2, suppress=True):
    print(np.sqrt(cov_infections.diagonal()))[   0.28 1516.83    0.01]
# Whole covariance matrix
with np.printoptions(precision=2, suppress=True):
    print(cov_infections)[[      0.08     357.84      -0.  ]
 [    357.84 2300767.13     -10.86]
 [     -0.       -10.86       0.  ]]
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=figsize)
ax.plot(data['date'], data['infected'], 'og')
ax.plot(data['date'], fitted_infections, '-b')
ax.legend(["data", "fit: $n(t) = c(t-t_0)^p+a_0$"])
ax.set_title(f"Infections $n(t)$ ({pretty_params2(param_infections, end=', ')}) \n"+
             f"dates used for fit: {dates_used_string}")
ax.xaxis.set_major_locator(matplotlib.dates.DayLocator())
ax.xaxis.set_major_formatter(matplotlib.dates.DateFormatter('%b %d'))
ax.tick_params(axis='x', rotation=90)
fig.savefig('figures/infections-with-model-fit.svg')_____no_output_____

### Prediction of infection numbers for tomorrow

Assuming infections keep carry on growing as they have done according to the model:_____no_output_____

print(f"Infected people tomorrow: {int(round(model2(max(days)+1, t0, nmax, c)))} ({day_to_date(max(days)+1)})")Infected people tomorrow: 81477 (25 Feb 2020)

More speculative predictions further into the future:_____no_output_____

for future_days in range(1, 11):
    d = int(max(data['days'])) + future_days
    print(f"Infections in {future_days:2} days: ",
          f"{int(round(model2(d, t0, nmax, c))):6}",
          f" ({day_to_date(d)})")Infections in  1 days:   81477  (25 Feb 2020)
Infections in  2 days:   81909  (26 Feb 2020)
Infections in  3 days:   82259  (27 Feb 2020)
Infections in  4 days:   82543  (28 Feb 2020)
Infections in  5 days:   82773  (29 Feb 2020)
Infections in  6 days:   82959  (01 Mar 2020)
Infections in  7 days:   83108  (02 Mar 2020)
Infections in  8 days:   83229  (03 Mar 2020)
Infections in  9 days:   83326  (04 Mar 2020)
Infections in 10 days:   83405  (05 Mar 2020)

# Deaths_____no_output_____Fit (the same) model, and plot.

Update 12 Feb: can't fit all data points for this model with the increase from 11 to 12 Feb.

New attempt: use the initial few days and 12 Feb as fitting points._____no_output_____

data2_____no_output_____days = data['days']
days2 = data2['days']
deaths = data2['deaths']

param_death, cov_deaths = scipy.optimize.curve_fit(model1, days2, deaths,
                                                   p0=(0.9, -2., 2.3, 3.6))
print(pretty_params(param_death))p = 1.852
c = 4.12
t0= -0.000
a0= -12.201
# Estimated standard deviation of parameters
with np.printoptions(precision=2, suppress=True):
    print(" c:   t0:  p:   a0:")
    print(np.sqrt(cov_deaths.diagonal())) c:   t0:  p:   a0:
[ 3.16  2.45  0.19 35.78]
fitted_days = days
c, t0, p, a0 = param_death
fitted_deaths = model1(fitted_days, c, t0, p, a0)_____no_output_____fig, ax = plt.subplots(figsize=figsize)
# data.plot('date', 'deaths', style='og',ax=ax)
ax.plot(data['date'], data['deaths'], 'og')
ax.plot(data['date'], fitted_deaths, '-b')
ax.legend(["data", "fit: $n(t) = c(t-t_0)^p+a_0$"])
ax.set_title(f"Deaths $n(t)$ ({pretty_params(param_death, end=', ')})\n"+
             f"dates used for fit: {dates_used_string}")
# ax.set_xlabel('$t$ (days)')
ax.xaxis.set_major_locator(matplotlib.dates.DayLocator())
ax.xaxis.set_major_formatter(matplotlib.dates.DateFormatter('%b %d'))
ax.tick_params(axis='x', rotation=90)
fig.savefig('figures/deaths-with-model-fit.svg')_____no_output_____

## Prediction for tomorrow

Assuming increase of deaths carries on as before_____no_output_____

print(f"Deaths tomorrow: {int(round(model1(max(days)+1, c, t0, p, a0)))} ({day_to_date(max(days)+1)})")Deaths tomorrow: 2971 (25 Feb 2020)

and more speculative beyond tomorrow_____no_output_____

for future_days in range(1, 11):
    d = int(max(data['days'])) + future_days
    print(f"Deaths in {future_days:2} days: ",
          f"{int(round(model1(d, c, t0, p, a0))):6}",
          f" ({day_to_date(d)})")Deaths in  1 days:    2971  (25 Feb 2020)
Deaths in  2 days:    3130  (26 Feb 2020)
Deaths in  3 days:    3294  (27 Feb 2020)
Deaths in  4 days:    3461  (28 Feb 2020)
Deaths in  5 days:    3633  (29 Feb 2020)
Deaths in  6 days:    3808  (01 Mar 2020)
Deaths in  7 days:    3986  (02 Mar 2020)
Deaths in  8 days:    4169  (03 Mar 2020)
Deaths in  9 days:    4355  (04 Mar 2020)
Deaths in 10 days:    4545  (05 Mar 2020)
print(f"Last executed: {datetime.date.today().strftime('%d %b %Y')}, "\
      f"last data point from {max(data['date']).strftime('%d %b %Y')}.")Last executed: 25 Feb 2020, last data point from 24 Feb 2020.

# Discussion_____no_output_____- disclaimer: the whole discussion above is not particularly sophisticated, does not take into account expert knowledge in epidemiology.
- we fit a trivial (monomial-type) model to the numbers, and the fit appears (chi-by-eye) pretty good over the period from 21 Jan to early February. This implies that
- the effect of measures to contain the spread of the disease (lock downs, cancelled flights, ...) is not clearly visible in the data yet

What would be interesting to do?
- evaluate quality of fit quantitatively ($\chi^2$?)
- take simplified growth model ($n = ct^p$?), compute fit for shorter periods of time (5 days?) as a sliding window and see if $p$ changes? (We need it to go down.)

What do we see?
- the $t^p$ model stops working for the number of infections (was a good fit [judged by looking at it - chi-by-eye] until about 7 February, then the reported infection numbers slow down faster than the in the early stages of the outbreak)
- the $t^p$ model remains to be pretty accurate for reported deaths. Maybe not suprising as these should lag the number of infections by 2 weeks or so.
_____no_output_____# Other somewhat random data explorations_____no_output_____### With the current growth, when would humans face extinction?

Inspired by [this suggestion from Joseph C Slater](https://twitter.com/JosephCSlater/status/1223739178468413440), we can work out how long it takes until all 7.8 billion people on the planet would be *if the growth rate does not change*. _____no_output_____

# work out solution numerically (in case model is changed later)
def survivors_planet(t, params):
    # Assume 7.8 billion people on planet, https://en.wikipedia.org/wiki/World_population
    return 7.8e9 - model1(t, *params)

ndays = scipy.optimize.bisect(survivors_planet, 0, 2100000, args=(param_death,))
print(f"Days until extinction: {ndays:.0f}, that's {ndays/365:0.1f} years.") Days until extinction: 101968, that's 279.4 years.

## Compare average new infections from model with day-resolved data_____no_output_____

fix, ax = plt.subplots()
# increase per day should be derivative dn/dt = p*c*(t-t0)^(p-1)
derivative = p*c*(data['days']-t0)**(p-1)
ax.plot(data['days'], data['new-deaths'], 'x-', label='actual data')
ax.plot(data['days'], derivative, 'o-', label="'averaged' new deaths from model")
ax.legend()_____no_output_____# The plot above doesn't look quite right. What went wrong?_____no_output_____data_____no_output_____