# Notebook from Camilo-HG/PDF_with_Python
Path: PDF_with_Python27.ipynb

# PDFs with python 2.7

This is a documentation about **how to extract information from PDFs using python 2.7**.

Date: 13 Dec (2017)

I needed to record this process because it was difficult to find a good package which let me extract text from any PDF file. Maybe I was expecting too much from them.

First, I needed to understand how the pdf file format works.

In quora, I found the next [comment](https://www.quora.com/Is-there-an-easy-to-use-Python-library-to-read-a-PDF-file-and-extract-its-text):

    We all expect to be able to use a library to parse some file format for text and be able to iterate through the text line by line, but what if the text has no line characters?

    How would the library know what constitutes a line? Most libraries won’t try to guess at that, and honestly we wouldn’t want them to, because if the line isn’t represented by a line character, then the concept of line isn’t really part of the text (is it?) and we are using the library to extract *text*.

    In pdf, text is laid out, meaning that a particular text object get displayed at a particular x,y position on the page. So what you might think of as 3 lines would actually be 3 text objects, displayed at (x,y), (x, y-20), (x, y-40) respectively, so a text extraction library would just pull out the text, but you’d have no line data. (IRRC pdfminer hands you String as output, just a big String, not a (line) iterable, it was because PDFMiner didn’t work for me that I had to study up and learn a bit about pdf to get what I wanted out of the files).

    The upside is this — You finally get a chance to ‘roll your own.’

    Fortunately, extracting the text out of a pdf is very well defined and simple goal. And fortuanately, PDF is a very well documented and very well understood file format, so google is going to be very helpful. If push comes to shove, the text rendering part of the spec is less than 200 pages, but you won’t need to go there.

    Start here:

    https://web.archive.org/web/20141010035745/http://gnupdf.org/Introduction_to_PDF

    Then read the wikipedia article which is super well written. Then you will have to open the file in text editor and study it, which won’t be hard if you are interested only in text. Use this as a tool to understand the stream writing operators:

    http://www.verypdf.com/document/pdf-format-reference/pg_0985.htm

    The accepted answer to the following SO tells you what you need to investigate to understand how text is encoded within the pdf:

    https://stackoverflow.com/questions/4047953/programatically-rip-text-from-a-pdf-file-by-hand-missing-some-text

    Google anything you wish to understand, and you will be brought to cool sites like planetpdf, where they have great articles.

    It should take you a day or two to hand write your parser and you will learn a lot in the process about something pretty common. The libraries have to be general, so they are going to be limited.

    (Perhaps irrelevant, the pdfs I was working with are linearized—see the linked references—which made studying the text in the pdf and mapping to the layout on the screen super simple, I didn’t study an non-linearized files because i didn’t have to, but if it makes things harder there’s a ton of code out there to linearize a pdf but not a lot out there that can go the otherway)

   **Naftali Michalowsky**

***
Another comment I found in the same page said:

    This may come to you as a surprise, but PDF was never actually intended as a format for easy text extraction. Indeed, its primary purpose is to make sure that whatever is in the document would be displayed in a consistent manner across multiple platforms, as well as print identically (as much as that is possible) everywhere. This is achieved in many different ways, and sometimes goes as far as literally saying "this letter goes here, this letter goes here, and this letter goes here" in its internal language.

    So yes, the libraries to extract text from PDF are insanely complicated by necessity, and they sometimes do not even function properly, because PDF is not meant for that.

   **Paulina Jonušaitė**

***

So I decided to take some packages and take a look of their performance "reading" PDF files._____no_output_____## PDF Samples

In order to explore the tool's functionalities, I have downloaded three pdfs.

One of them named sample.pdf which was obtained from this tutorial: http://programacion.net/articulo/como_trabajar_con_documentos_pdf_utilizando_python_1398.

Other one named pdf-sample.pdf which was obtained from: http://unec.edu.az/application/uploads/2014/12/pdf-sample.pdf.

Last one is a paper.

I have created one directory named **pdfs** to contain the documents. My work directory looks like:

    PDF_with_Python/
    ├── LICENSE
    ├── pdfs
    │   ├── paper.pdf
    │   ├── pdf-sample.pdf
    │   └── sample.pdf
    ├── PDF_with_Python.ipynb
    └── README.md_____no_output_____## Tools information

There exists different python tools for manipulating PDFs. Our interest in this notebook is to address this issue using **python 2.7**. Next tools are reported to work with python 2.

**But remember that _Python 2.7_ will not be maintained past 2020**.

### [PDFMiner](http://euske.github.io/pdfminer/index.html "PDFMiner")
- Last Modified: Fri Mar 28 09:17:06 UTC 2014.
- **Python 2.x** only.
- MIT License.
- To install [this package](https://anaconda.org/conda-forge/pdfminer "pdfminer conda") with [conda](https://anaconda.org/ "anaconda cloud") run:

        conda install -c conda-forge pdfminer

- When I did using conda, the scripts are not globally recognized. So I had to install following next steps:
    + Install Python 2.4 or newer. (Python 3 is not supported.)
    + Download the PDFMiner source: https://pypi.python.org/pypi/pdfminer/
    + Unpack it.
    + Run setup.py to install:

            sudo python setup.py install

    + Do the following test:

            pdf2txt.py samples/simple1.pdf

            Hello

            World

            Hello

            World

            H e l l o

            W o r l d

            H e l l o

            W o r l d
    + Done!

PDFMiner is a tool for extracting information from PDF documents. Unlike other PDF-related tools, it focuses entirely on getting and analyzing text data.

PDFMiner allows one to obtain the exact location of text in a page, as well as other information such as fonts or lines. It includes a PDF converter that can transform PDF files into other text formats (such as HTML). It has an extensible PDF parser that can be used for other purposes than text analysis.

#### Tutorials

- [PDFMiner API Doc](http://unixuser.org/~euske/python/pdfminer/programming.html)
- [PDFMiner tutorial 1](http://denis.papathanasiou.org/posts/2010.08.04.post.html)
- [PDFMiner tutorial 2](https://www.binpress.com/tutorial/manipulating-pdfs-with-python/167)
- [PDFMiner tutorial 3](https://wiki.carleton.edu/display/itskb/PDFMiner%3A+Extracting+Text+from+a+PDF+File)

#### Features

- Written entirely in Python. (for version 2.4 or newer)
- Parse, analyze, and convert PDF documents.
- PDF-1.7 specification support. (well, almost)
- CJK languages and vertical writing scripts support.
- Various font types (Type1, TrueType, Type3, and CID) support.
- Basic encryption (RC4) support.
- PDF to HTML conversion (with a sample converter web app).
- Outline (TOC) extraction.
- Tagged contents extraction.
- Reconstruct the original layout by grouping text chunks.

Though PDFminer is simple it has its own DRAWBACKS:

   + Getting simple things done, like extracting the text is quite complex. The program is not designed to return Python objects, which makes interfacing things irritating.
   + It’s an extremely complete set of tools, with multiple and moderately steep learning curves.
   + It’s not written with hackability in mind.

### [Textract](http://textract.readthedocs.io/en/latest/)
- Latest commit on 21 Jul (2017).
- Requires **PDFMiner**, pdf2text and other solutions.
- **Python 2**.
- It is supposed to work in **Python 3** too using **pdfminer.six**, but when I tried to install it with conda (for python 3) it generated conflicts.
- Unspecified License.
- Includes **command line interface**:

        textract path/to/file.extension

- Project's **GitHub**: https://github.com/deanmalmgren/textract
- **Documentation**: http://textract.readthedocs.io/en/latest/
- **Documentation for Python Package**: http://textract.readthedocs.io/en/latest/python_package.html
- Current versión (up to date): **1.6.1** (manually)
- To install [this package](https://anaconda.org/conda-forge/textract "Textract conda") with [conda](https://anaconda.org/ "anaconda cloud") run one of the following (version 1.5.0):

        conda install -c conda-forge textract

### [PdfQuery](https://github.com/jcushman/pdfquery "PdfQuery")
- Latest commit on 29 Jul (2017).
- Requires **PDFMiner**, pyquery and lxml libraries.
- Includes sample code, documentation.
- Seems to be **Python 2.x**. It is not clear if this tool is Python 2.x or Python 3.x.
- MIT License.
- To install [this package](https://anaconda.org/jacksongs/pdfquery "PdfQuery conda") with [conda](https://anaconda.org/ "anaconda cloud") run:

        conda install -c jacksongs pdfquery_____no_output_____# Textract

First we have to import the module_____no_output_____

import textract_____no_output_____

Now let's try to read a document.

***
Let's try to read sample.pdf:_____no_output_____

text = textract.process("./pdfs/sample.pdf")_____no_output_____

What happened? ... Let's print **text** content:_____no_output_____

print(text)This is a sample PDF document Iâ€™m using to follow along with the tutorial

It seems pretty good.

***
Let's try with the file pdf-sample.pdf:_____no_output_____

text = textract.process("./pdfs/pdf-sample.pdf")
print(text)Adobe Acrobat PDF Files
Adobe® Portable Document Format (PDF) is a universal file format that preserves all
of the fonts, formatting, colours and graphics of any source document, regardless of
the application and platform used to create it.
Adobe PDF is an ideal format for electronic document distribution as it overcomes the
problems commonly encountered with electronic file sharing.
•

Anyone, anywhere can open a PDF file. All you need is the free Adobe Acrobat
Reader. Recipients of other file formats sometimes can't open files because they
don't have the applications used to create the documents.

•

PDF files always print correctly on any printing device.

•

PDF files always display exactly as created, regardless of fonts, software, and
operating systems. Fonts, and graphics are not lost due to platform, software, and
version incompatibilities.

•

The free Acrobat Reader is easy to download and can be freely distributed by
anyone.

•

Compact PDF files are smaller than their source files and download a
page at a time for fast display on the Web.

Last result was so good although it seems disordered.

***
What happen when we try to read more complex PDFs like the **paper** we have downloaded?

Let's try with the file paper.pdf:_____no_output_____

text = textract.process("./pdfs/paper.pdf")
print(text)Leukemia Research 34 (2010) 677–681

Contents lists available at ScienceDirect

Leukemia Research
journal homepage: www.elsevier.com/locate/leukres

Brief communication

Micro-RNA-15a and micro-RNA-16 expression and chromosome 13 deletions in
multiple myeloma
Sophie L. Corthals a , Mojca Jongen-Lavrencic a , Yvonne de Knegt a , Justine K. Peeters a ,
H. Berna Beverloo b , Henk M. Lokhorst c , Pieter Sonneveld a,∗
a
b
c

Department of Hematology, Erasmus Medical Centre Rotterdam, Rotterdam, The Netherlands
Department of Clinical Genetics, Erasmus Medical Centre Rotterdam, Rotterdam, The Netherlands
Department of Hematology, University Medical Center Utrecht, Utrecht, The Netherlands

a r t i c l e

i n f o

Article history:
Received 15 October 2009
Received in revised form 28 October 2009
Accepted 28 October 2009
Available online 23 December 2009
Keywords:
Multiple myeloma
Micro-RNA
Chromosome 13 deletion

a b s t r a c t
We have used copy number variation (CNV) analysis with SNP mapping arrays for miRNA-15a and
miRNA-16-1 expression analysis in patients with multiple myeloma (MM) with or without deletion of
chromosome 13q14. MiRNA-15a and miRNA-16 display a range of expression patterns in MM patients,
independent of the chromosome 13 status. These ﬁndings suggest that genes other than miR-15a and
miR-16 may explain the prognostic signiﬁcance of 13q14 deletions.
© 2009 Elsevier Ltd. All rights reserved.

1. Introduction
Multiple myeloma (MM) is a dismal disease with a median survival of 3–5 years [1].
MM is characterized by genetic instability and chromosomal
translocations early in disease progression play an important role
in the pathogenesis of MM. Chromosome 13 aberrations occur in
50% and the most commonly deleted marker in the 13q14 region is
D13S319, located distal to the retinoblastoma-1 (RB1) gene [2,3]. It
is now common practice to determine the presence of chromosome
13 deletions by ﬂuorescence in situ hybridization (FISH) [4–10].
The adverse prognostic role of these deletions has been attributed
to the presence of MM tumor suppressor gene(s) on chromosome
13q which as yet have not been identiﬁed.
Micro-RNAs (miRs) are a class of small non-coding single
stranded RNAs of approximately 22 nucleotides in length [11]. MiRs
negatively regulate gene expression by binding to partially complementary sites in messenger RNAs (mRNAs) [12]. miRs may play
a role in the pathogenesis of MM [13,14]. In chronic lymphocytic
leukemia (CLL), miR-15a and miR-16-1 are frequently downregu-

∗ Corresponding author at: Department of Hematology, Erasmus Medical Centre
Rotterdam, Room No. L439, PO Box 2040, 3000 CA Rotterdam, The Netherlands.
Tel.: +31 10 7033123; fax: +31 10 7035814.
E-mail address: p.sonneveld@erasmusmc.nl (P. Sonneveld).
0145-2126/$ – see front matter © 2009 Elsevier Ltd. All rights reserved.
doi:10.1016/j.leukres.2009.10.026

lated [15]. We evaluated the expression of miR-15a and miR-16-1
in MM.
2. Materials and methods
We obtained bone marrow samples from newly diagnosed patients with MM.
Myeloma plasma cells were puriﬁed >80% using CD138 magnetic microbeads
(MACS system, Miltenyi Biotec, Bergisch Gladbach, Germany). Small RNA transcripts less than 200 nucleotides in length including miRs, were isolated from
CD138 magnetic cell selected (MACS) MM plasma cells using miRVanaTM miRNA
Isolation Kit (Ambion, Austin, TX). RNA levels and quality were assessed using
the Agilent 2100 Bioanalyzer (Agilent Technologies). Mature miR-15a and miR16 expression levels in MM plasma cells were determined by real-time PCR
using TaqMan miR assay (Applied Biosystems, P/N: 4373123 and 4373121)
[16]. All experiments were performed in duplicate and miRs were considered
as present when CT -values were lower than 35. Genomic DNA was isolated
from peripheral blood by salting out precipitation. Tumor DNA was extracted
from CD138 MACS MM plasma cells using Qiagen RNAeasy kit (Qiagen, Valencia, CA). Genome-wide SNP genotyping was performed using the Illumina
Inﬁnium HumanHap550 Genotyping BeadChip, containing over 550,000 unique
tag SNP markers, according to manufacturer’s instructions (Illumina, San Diego,
CA).
Genotypes for all arrays were calculated using BeadStudio’s genotyping module (v2.0, Illumina). Data was imported into Partek Genomics Suite 6.4 software
(Partek Inc., St Louis, MO) for further analysis; allele intensities were calculated
for 25 genomic DNA MM samples and 26 tumor DNA MM samples, of which 20
had paired normal samples. For these 20 samples, paired analysis was performed.
Allele speciﬁc copy number variation (AsCN) was calculated by estimating the number of copies for each allele, rather than total copies of each chromosome. Loss of
heterozygosity (LOH) was estimated from the imported sample genotypes by a Hidden Markov Model, using the same baseline as in the segmentation algorithm (25

678

Table 1
Chromosome 13 aberrations of the region ﬂanking miR-15a and miR-16-1 identiﬁed by FISH and segmentation analysis using SNP mapping arrays.
FISH

Segmentation

Analysis
type

Start

End

Cytoband

Length (bps)

Copy
Number

# Markers

Genes

MM 1
MM 2
MM 3
MM 4
MM 5
MM 6
MM 7
MM 8
MM 9
MM 10
MM 11
MM 12
MM 13
MM 14
MM 15
MM 16
MM 17
MM 18
MM 19
MM 20
MM 21
MM 22
MM 23
MM 24
MM 25
MM 26

13
Del
Del
Del
Del
Del
Del
Del
Del
Del
Del
Del
N
N
N
N
N
N
N
N
N
N
N
N
N
N

Del
Del
Del
Del
Del
Del
Del
Del
Del
Del
Del
Del
Del
Del
Del
Del
Del
Del
Del
Del
N
N
N
N
N
N

Paired
Paired
Paired
Paired
Paired
Paired
Paired
Paired
Paired
Paired
Unpaired
Unpaired
Paired
Paired
Paired
Paired
Paired
Unpaired
Unpaired
Unpaired
Paired
Unpaired
Paired
Paired
Paired
Paired

42503650
27313597
47862787
45560358
48984705
43781771
40498617
47786017
39203461
40531599
45324420
22372135
49431816
49024071
26462920
34866084
49358246
37338941
48984705
48951859

57487878
57610331
114121253
52963785
53079298
70193498
70309383
104600044
66142848
53088969
52963785
78494745
49631113
52210918
104775995
97736842
49631113
52972100
52958855
68490076

13ql4.11–13q21.1
13ql2.2–13q21.2
13ql4.2–13q34
13ql4.12–13q21.1
13ql4.3–13q21.1
13ql4.11–13q21.33
13ql4.11–13q21.33
13ql4.2–13q33.2
13ql3.3–13q21.32
13ql4.11–13q21.1
13ql4.12–13q21.1
13ql2.12–13q31.1
13ql4.3
13ql4.3–13q21.1
13ql2.13–13q33.2
13ql3.3–13q32.2
13ql4.3
13ql3.3–13q21.1
13ql4.3–13q21.1
13ql4.3–13q21.33

14984228
30296734
66258466
7403427
4094593
26411727
29810766
56814027
26939387
12557370
7639365
56122610
199297
3186847
78313075
62870758
272867
15633159
3974150
19538217

0.6930
1.0028
0.9633
0.8485
0.7389
0.6852
0.5797
1.2421
0.9434
0.9634
0.9586
1.0474
1.0060
1.2947
0.8103
1.4238
0.5809
1.0293
1.3794
1.5462

2789
6342
13801
1488
828
4764
5616
11446
5040
2661
1529
11979
19
572
16439
12446
23
3373
796
3374

ARL11, C13orﬂ, DLEU1, EBPL, KCNRG, KPNA3, PHF11, RCBTB1, SETDB2, TRIM13
ARL11, C13orﬂ, DLEU1, EBPL, KCNRG, KPNA3, PHF11, RCBTB1, SETDB2, TRIM13
ARL11, C13orﬂ, DLEU1, EBPL, KCNRG, KPNA3, PHF11, RCBTB1, SETDB2, TRIM13
ARL11, C13orﬂ, DLEU1, EBPL, KCNRG, KPNA3, PHF11, RCBTB1, SETDB2, TRIM13
ARL11, C13orﬂ, DLEU1, EBPL, KCNRG, KPNA3, PHF11, RCBTB1, SETDB2, TRIM13
ARL11, C13orﬂ, DLEU1, EBPL, KCNRG, KPNA3, PHF11, RCBTB1, SETDB2, TRIM13
ARL11, C13orﬂ, DLEU1, EBPL, KCNRG, KPNA3, PHF11, RCBTB1, SETDB2, TRIM13
ARL11, C13orﬂ, DLEU1, EBPL, KCNRG, KPNA3, PHF11, RCBTB1, SETDB2, TRIM13
ARL11, C13orﬂ, DLEU1, EBPL, KCNRG, KPNA3, PHF11, RCBTB1, SETDB2, TRIM13
ARL11, C13orﬂ, DLEU1, EBPL, KCNRG, KPNA3, PHF11, RCBTB1, SETDB2, TRIM13
ARL11, C13orﬂ, DLEU1, EBPL, KCNRG, KPNA3, PHF11, RCBTB1, SETDB2, TRIM13
ARL11, C13orﬂ, DLEU1, EBPL, KCNRG, KPNA3, PHF11, RCBTB1, SETDB2, TRIM13
KCNRG, TRIM13
ARL11, C13orﬂ, EBPL, KCNRG, KPNA3, RCBTB1, TRIM13
ARL11, C13orﬂ, DLEU1, EBPL, KCNRG, KPNA3, PHF11, RCBTB1, SETDB2, TRIM13
ARL11, C13orﬂ, DLEU1, EBPL, KCNRG, KPNA3, PHF11, RCBTB1, SETDB2, TRIM13
C13orﬂ, KCNRG, TRIM13
ARL11, C13orﬂ, DLEU1, EBPL, KCNRG, KPNA3, PHF11, RCBTB1, SETDB2, TRIM13
ARL11, C13orﬂ, EBPL, KCNRG, KPNA3, PHF11, RCBTB1, TRIM13
ARL11, C13orﬂ, DLEU1, EBPL, KCNRG, KPNA3, PHF11, RCBTB1, SETDB2, TRIM13

N indicates normal; Del, deletion; A13, deletion; Amp, ampliﬁcation; ND, not done.

S.L. Corthals et al. / Leukemia Research 34 (2010) 677–681

Patient

S.L. Corthals et al. / Leukemia Research 34 (2010) 677–681

679

Fig. 1. Copy number alterations of chromosome 13 as determined by paired segmentation analysis. SNP mapping arrays (Illumina Inﬁnium HumanHap550 Genotyping
BeadChip) were used to determine the exact deleted regions at chromosome 13 of multiple myeloma (MM) cases. (A) In panel 1; the genome browser illustrates the gene
transcripts (NCBI) located on chromosome 13. The results of the paired segmentation analysis of chromosome 13 (20/26 MM patients) are shown in the panel 2. (B) A region
on chromosome 13; 13q14.3, including the miR-15a and miR-16 locus, is depicted at larger magniﬁcation. In panel 1 the miR-15a and miR-16 locus (star) is shown within the
genome browser. Panel 2 shows the results of the paired segmentation analysis in more detail. A deletion (orange and purple) was detected in 15/20 MM patients. Uniparental
disomy (UPD; green) was detected in patient MM 17, in the regions ﬂanking miR-15a and miR-16-1. Copy number (CN) values for SNPs (red dots) are plotted in panel 3. CN
values of 2 represent diploid copy number, whereas values of 1 and 0 represent heterozygous and homozygous deletion, respectively. A representative patient (MM 7) shows
values of 1 or ≤1, indicating deletion. In panel 4 the B allele frequency is displayed. Each point (green dot) represents the B allele frequency of one SNP. B allele frequencies of 0
and 1 represent homozygous signals (AA or BB), while an allele frequency of ∼0.5 represents an equal contribution from both alleles (AB). A representative MM patient sample
(MM 7) shows a deletion with LOH (black box in segmentation panel) which produces a characteristic pattern that lacks heterozygote signal. For visualization purposes the
order of the samples is MM 26, MM 10, MM 4, MM 1, MM 21, MM 8, MM 14, MM 7. MM 16, MM 15, MM 9, MM 13, MM 2, MM 23, MM 5, MM 24, MM 25, MM 3, and MM 17.
genomic blood DNA MM samples). CN, AsCN, LOH and allelic ratios were integrated
and visualized in a genomic browser within Partek Genomics Suite.
Chromosome analysis was performed using both QFQ- and RBA-banding [17].
Fluorescence in situ hybridization (FISH) was performed using standard protocols
[18].
In this study, the stability of ﬁve candidate reference small nuclear RNAs was
examined using the validation program GeNorm [19]. The relative expression levels
of miR-15a and miR-16 compared to CD138 sorted plasma cells from normal bone
marrow were determined using the 2−Ct method [20].
The Mann–Whitney U-test was applied to determine differences of the miR-15a
or miR-16 gene expression levels.

3. Results
FISH analysis showed chromosome 13 deletions in 11/26
patients (Table 1 and Table S1). Using karyotyping, 1 additional patient had a chromosome 13 deletion, resulting in 12/26
patients (46.2%) having a chromosome 13 aberration. CNV analysis using SNP mapping arrays performed on 20/26 paired samples
indicated that a minimally deleted region on chromosome 13
(49431816–49631113 bp), in which miR-15a and miR-16-1 are

680

S.L. Corthals et al. / Leukemia Research 34 (2010) 677–681

Fig. 2. Copy number of chromosome 13 in correlation with miR-15a gene expression in multiple myeloma (MM) patient samples. A TaqMan miR assay was used to determine
the relative miR-15a expression levels in myeloma plasma of 26 MM patients. The normalized relative expression of miR-15a in 26 MM patient samples is displayed on
the X-axis. MiR-15a is expressed in all 26 MM samples, although the level of expression varies across the samples. SNP mapping arrays (Illumina Inﬁnium HumanHap550
Genotyping BeadChip) were used to determine copy number values of chromosome 13, which are shown on the Y-axis. Copy number values of 2 were detected in 6/26
MM patients, indicating diploid copy number. Values less than 2 represent heterozygous and homozygous deletions, which were found in 20/26 MM patients. Patients are
coloured by chromosome 13 status as determined by FISH; being either normal (black) or deletion (gray). Segmentation analysis has shown 8 additional patients with a
chromosome 13 aberration in the region containing miR-15a and miR-16-1. A linear regression line reveals there is no correlation between the miR-15a expression and the
chromosome 13 status as determined by segmentation analysis. The picture insert shows a box plot representation of the normalized relative expression of miR-15a in 26
MM patients samples (X-axis), in correlation with the chromosome 13 status determined by segmentation (Y-axis). The box plot shows no signiﬁcant difference in miR-15a
gene expression (P = 0.38, Mann–Whitney U-test) between patients with a deletion (gray, n = 20) and normal chromosome 13 status (black, n = 6).

located, was observed in 20/26 patients, 15 of which were from
the paired analysis (P < 0.00001) (Table 1 and Tables S1 and S2 and
Fig. 1, Fig. S1). Segmentation analysis revealed 8 additional patients
with a chromosome 13 deletion compared to FISH analysis. AsCN
analysis further revealed the absence of homozygous deletions in
these patients. No uniparental disomy (UPD; copy number-neutral
LOH), was detected in the exact location of miR-15a and miR-161 (Fig. 1 and Fig. S1). A highly correlated cluster of miRs; miR-15b
and miR-16-2, is located on chromosome 3. MiR-15a and miR-15b
are derived from the same seed sequence, however, they differ in
their mature sequence and therefore cannot be distinguished. For
this reason, we are not able to separate miR-16-1 and 16-2 expression using the real-time PCR assay, since the mature miR sequence
primes the assay. As these highly correlated miRs are located on
chromosome 3, we have also examined this region. Analysis of this
chromosome was performed using the segmentation algorithm to
determine the ampliﬁed and deleted regions (miR-15b and miR16-2 location 161605070–161605307 bp). Deletion of this region
on chromosome 3 was found in 2 MM and ampliﬁcation in 6 MM
patients (Table S2).
Both miR-15a and miR-16 were expressed in all 26 MM samples.
The median expression value of miR-15a and miR-16 was 0.84 and
1.16, respectively (both values in log2 scale, relative to the geometric mean). No signiﬁcant association was found between the
chromosome 13 deletion status and the miR-15a expression levels
using FISH (P = 0.38) and CNV (P = 0.25) (Fig. 2). In addition, statistical analysis showed no signiﬁcant association between miR-16
expression and chromosome 13 status by FISH (P = 0.40) or CNV

(P = 0.27). We also evaluated the expression levels of miR-15a and
miR-16 in CD138 sorted plasma cells from normal individuals. MiR16 was expressed at 1.4-fold lower levels in normal plasma cells
when compared with myeloma plasma cells.
These ﬁndings demonstrate that miR-15a and miR-16 are displayed at a range of expression levels in MM patients which are
higher than in normal plasma cell counterparts. The expression of
these miRs varies independent of the chromosome 13 status.

4. Discussion
Deletion of chromosome 13 is detected in 50% of patients by
FISH and in 10–20% by karyotyping [5,21–23]. Patients with a whole
chromosome 13 deletion detected by cytogenetics, have a worse
prognosis than 13q deletions detected by FISH [24–27]. Loss of
chromosome 13 occurs, like in MM, in approximately 50% of CLL
patients, although CLL patients with 13q deletions have favorable
survival [28]. Differences of the length of the involved region may
explain these observations [29].
Downregulation of miR-15a and miR-16-1 has been reported in
cases of MM, CLL and diffuse B Cell Lymphomas [14,15,30]. In this
study, the two miRs were expressed in all 26 MM patients examined, even when chromosome 13 was deleted. Our analysis showed
an high speciﬁcity for detecting allelic imbalances in heterogeneous
samples [31,32]. It is unlikely that chromosome 13 deletions that
are not detected by CNV analysis are partial deletions. Compensation by the non-deleted allele could explain the expression level

S.L. Corthals et al. / Leukemia Research 34 (2010) 677–681

of miR-16 since all chromosome 13 deletions in this study are heterozygous.
This study clearly shows the discrepancy between FISH and
CNV analysis and the efﬁcacy of SNP mapping arrays in detecting
chromosomal aberration. It is important that a direct comparison
between tumor DNA and matched germline DNA was performed
[33]. Because paired CN analysis was carried out for 20/26 MM
samples, this makes the data highly reliable. In conclusion, high
resolution, genome-wide SNP mapping arrays may provide an
excellent tool to identify partial chromosomal aberrations and
genes.
Conﬂict of interest statement
There are no conﬂict of interests in relation to this work.
Appendix A. Supplementary data
Supplementary data associated with this article can be found, in
the online version, at doi:10.1016/j.leukres.2009.10.026.
References
[1] Kyle RA, Rajkumar SV. Multiple myeloma. N Engl J Med 2004;351(18):1860–73.
[2] Avet-Louseau H, Daviet A, Sauner S, Bataille R. Chromosome 13 abnormalities in multiple myeloma are mostly monosomy 13. Br J Haematol
2000;111(4):1116–7.
[3] Shaughnessy J, Tian E, Sawyer J, Bumm K, Landes R, Badros A, et al. High incidence of chromosome 13 deletion in multiple myeloma detected by multiprobe
interphase FISH. Blood 2000;96(4):1505–11.
[4] Debes-Marun CS, Dewald GW, Bryant S, Picken E, Santana-Davila R, GonzalezPaz N, et al. Chromosome abnormalities clustering and its implications for
pathogenesis and prognosis in myeloma. Leukemia 2003;17(2):427–36.
[5] Facon T, Avet-Loiseau H, Guillerm G, Moreau P, Genevieve F, Zandecki M,
et al. Chromosome 13 abnormalities identiﬁed by FISH analysis and serum
beta2-microglobulin produce a powerful myeloma staging system for patients
receiving high-dose therapy. Blood 2001;97(6):1566–71.
[6] Fonseca R, Harrington D, Oken MM, Dewald GW, Bailey RJ, Van Wier SA, et
al. Biological and prognostic signiﬁcance of interphase ﬂuorescence in situ
hybridization detection of chromosome 13 abnormalities (delta13) in multiple myeloma: an eastern cooperative oncology group study. Cancer Res
2002;62(3):715–20.
[7] Seong C, Delasalle K, Hayes K, Weber D, Dimopoulos M, Swantkowski J,
et al. Prognostic value of cytogenetics in multiple myeloma. Br J Haematol
1998;101(1):189–94.
[8] Shaughnessy J, Barlogie B. Chromosome 13 deletion in myeloma. Curr Top
Microbiol Immunol 1999;246:199–203.
[9] Tricot G, Barlogie B, Jagannath S, Bracy D, Mattox S, Vesole DH, et al. Poor prognosis in multiple myeloma is associated only with partial or complete deletions
of chromosome 13 or abnormalities involving 11q and not with other karyotype
abnormalities. Blood 1995;86(11):4250–6.
[10] Zojer N, Konigsberg R, Ackermann J, Fritz E, Dallinger S, Kromer E, et al. Deletion of 13q14 remains an independent adverse prognostic variable in multiple
myeloma despite its frequent detection by interphase ﬂuorescence in situ
hybridization. Blood 2000;95(6):1925–30.
[11] Grifﬁths-Jones S, Grocock RJ, van Dongen S, Bateman A, Enright AJ. miRBase: microRNA sequences, targets and gene nomenclature. Nucleic Acids Res
2006;34:D140–4 [Database issue].
[12] Bartel DP. MicroRNAs: genomics, biogenesis, mechanism, and function. Cell
2004;116(2):281–97.

681

[13] Pichiorri F, Suh SS, Ladetto M, Kuehl M, Palumbo T, Drandi D, et al. MicroRNAs
regulate critical genes associated with multiple myeloma pathogenesis. Proc
Natl Acad Sci USA 2008;105(35):12885–90.
[14] Roccaro AM, Sacco A, Thompson B, Leleu X, Azab AK, Azab F, et al. microRNAs
15a and 16 regulate tumor proliferation in multiple myeloma. Blood 2009.
[15] Calin GA, Dumitru CD, Shimizu M, Bichi R, Zupo S, Noch E, et al. Frequent deletions and down-regulation of micro-RNA genes miR15 and miR16 at 13q14 in
chronic lymphocytic leukemia. Proc Natl Acad Sci USA 2002;99(24):15524–9.
[16] Chen C, Ridzon DA, Broomer AJ, Zhou Z, Lee DH, Nguyen JT, et al. Realtime quantiﬁcation of microRNAs by stem-loop RT-PCR. Nucleic Acids Res
2005;33(20):e179.
[17] Schaffer LG, Tommerup N, ISCN. An International System for Human Cytogenetic Nomenclature. S. Karger; 2005.
[18] van Zutven LJ, Velthuizen SC, Wolvers-Tettero IL, van Dongen JJ, Poulsen TS,
MacLeod RA, et al. Two dual-color split signal ﬂuorescence in situ hybridization
assays to detect t(5;14) involving HOX11L2 or CSX in T-cell acute lymphoblastic
leukemia. Haematologica 2004;89(6):671–8.
[19] Vandesompele J, De Preter K, Pattyn F, Poppe B, Van Roy N, De Paepe A,
et al. Accurate normalization of real-time quantitative RT-PCR data by geometric averaging of multiple internal control genes. Genome Biol 2002;3(7)
[RESEARCH0034].
[20] Livak KJ, Schmittgen TD. Analysis of relative gene expression data using
real-time quantitative PCR and the 2(-Delta Delta C(T)) method. Methods
2001;25(4):402–8.
[21] Chng WJ, Santana-Davila R, Van Wier SA, Ahmann GJ, Jalal SM, Bergsagel PL,
et al. Prognostic factors for hyperdiploid-myeloma: effects of chromosome 13
deletions and IgH translocations. Leukemia 2006;20(5):807–13.
[22] Chang H, Bouman D, Boerkoel CF, Stewart AK, Squire JA. Frequent monoallelic
loss of D13S319 in multiple myeloma patients shown by interphase ﬂuorescence in situ hybridization. Leukemia 1999;13(1):105–9.
[23] Fassas AB, Spencer T, Sawyer J, Zangari M, Lee CK, Anaissie E, et al. Both
hypodiploidy and deletion of chromosome 13 independently confer poor prognosis in multiple myeloma. Br J Haematol 2002;118(4):1041–7.
[24] Kaufmann H, Kromer E, Nosslinger T, Weltermann A, Ackermann J, Reisner
R, et al. Both chromosome 13 abnormalities by metaphase cytogenetics and
deletion of 13q by interphase FISH only are prognostically relevant in multiple
myeloma. Eur J Haematol 2003;71(3):179–83.
[25] Shaughnessy Jr J, Tian E, Sawyer J, McCoy J, Tricot G, Jacobson J, et al. Prognostic
impact of cytogenetic and interphase ﬂuorescence in situ hybridizationdeﬁned chromosome 13 deletion in multiple myeloma: early results of total
therapy II. Br J Haematol 2003;120(1):44–52.
[26] Dewald GW, Therneau T, Larson D, Lee YK, Fink S, Smoley S, et al. Relationship
of patient survival and chromosome anomalies detected in metaphase and/or
interphase cells at diagnosis of myeloma. Blood 2005;106(10):3553–8.
[27] Chiecchio L, Protheroe RK, Ibrahim AH, Cheung KL, Rudduck C, Dagrada GP,
et al. Deletion of chromosome 13 detected by conventional cytogenetics is a
critical prognostic factor in myeloma. Leukemia 2006;20(9):1610–7.
[28] Dohner H, Stilgenbauer S, Benner A, Leupolt E, Krober A, Bullinger L, et al.
Genomic aberrations and survival in chronic lymphocytic leukemia. N Engl J
Med 2000;343(26):1910–6.
[29] Rowntree C, Duke V, Panayiotidis P, Kotsi P, Palmisano GL, Hoffbrand AV, et
al. Deletion analysis of chromosome 13q14.3 and characterisation of an alternative splice form of LEU1 in B cell chronic lymphocytic leukemia. Leukemia
2002;16(7):1267–75.
[30] Eis PS, Tam W, Sun L, Chadburn A, Li Z, Gomez MF, et al. Accumulation of
miR-155 and BIC RNA in human B cell lymphomas. Proc Natl Acad Sci USA
2005;102(10):3627–32.
[31] Staaf J, Lindgren D, Vallon-Christersson J, Isaksson A, Goransson H, Juliusson
G, et al. Segmentation-based detection of allelic imbalance and loss-ofheterozygosity in cancer cells using whole genome SNP arrays. Genome Biol
2008;9(9):R136.
[32] Lai WR, Johnson MD, Kucherlapati R, Park PJ. Comparative analysis of
algorithms for identifying ampliﬁcations and deletions in array CGH data.
Bioinformatics 2005;21(19):3763–70.
[33] Feng S, Liang Q, Kinser RD, Newland K, Guilbaud R. Testing equivalence between
two laboratories or two methods using paired-sample analysis and interval
hypothesis testing. Anal Bioanal Chem 2006;385(5):975–81.

text_____no_output_____type(text)_____no_output_____text.find('miR-15')_____no_output_____text.find('bioanal')_____no_output_____text.find('Bioanal')_____no_output_____text.find('physics')_____no_output_____str1 = "this is string example....wow!!!";
str2 = "is";

print str1.find(str2)
2

gene expression
RNA
genomics
STAR
bioinformatics


# Notebook from MarcusFra/workshop
Path: 07_train/archive/03_Predict_Reviews_BERT_Transformers_TensorFlow_AdHoc.ipynb

# TODO: Predict Reviews with BERT in the Notebook (AdHoc)

# Note:  this no longer works properly as it's using the `transformers/fine-tuned` model_____no_output_____

import boto3
import sagemaker
import pandas as pd

sess   = sagemaker.Session()
bucket = sess.default_bucket()
role = sagemaker.get_execution_role()
region = boto3.Session().region_name

sm = boto3.Session().client(service_name='sagemaker', region_name=region)_____no_output_____

# Retrieve the Training Job Name_____no_output_____

%store -r training_job_name_____no_output_____try:
    training_job_name
except NameError:
    print('++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++')
    print('[ERROR] Please wait for the previous notebook to finish.')
    print('++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++')_____no_output_____print('Previous training_job_name: {}'.format(training_job_name))Previous training_job_name: tensorflow-training-2021-01-03-05-49-48-016

# Download the Trained Model_____no_output_____

models_dir = './models'_____no_output_____# Download the model and output artifacts from AWS S3
!aws s3 cp s3://$bucket/$training_job_name/output/model.tar.gz $models_dir/model.tar.gzdownload: s3://sagemaker-us-east-1-835319576252/tensorflow-training-2021-01-03-05-49-48-016/output/model.tar.gz to models/model.tar.gz
import tarfile
import pickle as pkl

#!ls -al ./models

tar = tarfile.open('{}/model.tar.gz'.format(models_dir))
tar.extractall(path=models_dir)
tar.close()_____no_output_____!ls -al $models_dirtotal 835436
drwxrwxr-x  8 ec2-user ec2-user      4096 Jan  3 06:47 .
drwxrwxr-x 17 ec2-user ec2-user      4096 Jan  3 06:47 ..
drwxr-xr-x  2 ec2-user ec2-user      4096 Jan  3 06:07 code
drwxr-xr-x  2 ec2-user ec2-user      4096 Jan  3 06:09 metrics
-rw-rw-r--  1 ec2-user ec2-user 855452891 Jan  3 06:11 model.tar.gz
drwxr-xr-x  4 ec2-user ec2-user      4096 Jan  3 05:58 tensorboard
drwxr-xr-x  3 ec2-user ec2-user      4096 Jan  3 05:56 tensorflow
drwxr-xr-x  2 ec2-user ec2-user      4096 Jan  3 06:07 test_data
drwxr-xr-x  3 ec2-user ec2-user      4096 Jan  3 05:56 transformers

# Load the Model_____no_output_____

transformer_model_dir = '{}/transformers/fine-tuned/'.format(models_dir)

!ls -al $transformer_model_dirtotal 259360
drwxr-xr-x 2 ec2-user ec2-user      4096 Jan  3 05:58 .
drwxr-xr-x 3 ec2-user ec2-user      4096 Jan  3 05:56 ..
-rw-r--r-- 1 ec2-user ec2-user       648 Jan  3 05:58 config.json
-rw-r--r-- 1 ec2-user ec2-user 265569984 Jan  3 05:58 tf_model.h5
cat $transformer_model_dir/config.json{
  "_name_or_path": "distilbert-base-uncased",
  "activation": "gelu",
  "architectures": [
    "DistilBertForMaskedLM"
  ],
  "attention_dropout": 0.1,
  "dim": 768,
  "dropout": 0.1,
  "hidden_dim": 3072,
  "id2label": {
    "0": 1,
    "1": 2,
    "2": 3,
    "3": 4,
    "4": 5
  },
  "initializer_range": 0.02,
  "label2id": {
    "1": 0,
    "2": 1,
    "3": 2,
    "4": 3,
    "5": 4
  },
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 12,
  "n_layers": 6,
  "pad_token_id": 0,
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "tie_weights_": true,
  "vocab_size": 30522
}
import os
from transformers import TFDistilBertForSequenceClassification

loaded_model = TFDistilBertForSequenceClassification.from_pretrained(transformer_model_dir,
                                                                     id2label={
                                                                       0: 1,
                                                                       1: 2,
                                                                       2: 3,
                                                                       3: 4,
                                                                       4: 5
                                                                     },
                                                                     label2id={
                                                                       1: 0,
                                                                       2: 1,
                                                                       3: 2,
                                                                       4: 3,
                                                                       5: 4
                                                                     })All model checkpoint layers were used when initializing TFDistilBertForSequenceClassification.

Some layers of TFDistilBertForSequenceClassification were not initialized from the model checkpoint at ./models/transformers/fine-tuned/ and are newly initialized: ['pre_classifier', 'classifier', 'dropout_19']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
from transformers import DistilBertTokenizer

tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')

inference_device = -1 # CPU: -1, GPU: 0
print('inference_device {}'.format(inference_device))inference_device -1
from transformers import TextClassificationPipeline

inference_pipeline = TextClassificationPipeline(model=loaded_model,
                                                tokenizer=tokenizer,
                                                framework='tf',
                                                device=inference_device) # -1 is CPU, >= 0 is GPU_____no_output_____

# THESE PREDICTIONS WILL NOT BE ACCURATE

Here are the reasons:
* we are not using the full dataset
* we are only training for 50 steps
* we are using a small training cluster

_This same BERT model achieves close to 97% accuracy when training longer, using a larger training cluster, and using more data.______no_output_____

review = """I loved it!"""
print(review, inference_pipeline(review))I loved it! [{'label': 1, 'score': 0.21490779519081116}]
review = """Not that good."""
print(review, inference_pipeline(review))Not that good. [{'label': 1, 'score': 0.21912693977355957}]

# Test Model_____no_output_____

import csv

df_test_reviews = pd.read_csv('./data/amazon_reviews_us_Digital_Software_v1_00.tsv.gz',
                                delimiter='\t',
                                quoting=csv.QUOTE_NONE,
                                compression='gzip')[['review_body', 'star_rating']]

df_test_reviews = df_test_reviews.sample(n=100)

df_test_reviews.shape_____no_output_____

# Actual Star Rating_____no_output_____

df_test_reviews.head()_____no_output_____y_actual = df_test_reviews['star_rating']

y_actual_____no_output_____

# Predicted Star Rating_____no_output_____

import pandas as pd

def predict(review_body):
    prediction_map = inference_pipeline(review_body[0:512])
    return prediction_map[0]['label']

y_test = df_test_reviews['review_body'].map(predict)

y_test_____no_output_____

# Classification Report:  Compare Actual to Predicted_____no_output_____

from sklearn.metrics import classification_report

print(classification_report(y_true=y_test, y_pred=y_actual))              precision    recall  f1-score   support

           1       0.30      0.11      0.16        56
           2       0.07      1.00      0.13         1
           3       0.00      0.00      0.00         0
           4       0.24      0.12      0.16        43
           5       0.00      0.00      0.00         0

    accuracy                           0.12       100
   macro avg       0.12      0.24      0.09       100
weighted avg       0.27      0.12      0.16       100

# Accuracy_____no_output_____

from sklearn.metrics import accuracy_score

print('Test accuracy: ', accuracy_score(y_true=y_test, y_pred=y_actual))Accuracy:  0.12

# Confusion Matrix_____no_output_____

import matplotlib.pyplot as plt
import pandas as pd

def plot_conf_mat(cm, classes, title, cmap = plt.cm.Greens):
    print(cm)
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    fmt = 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
        horizontalalignment="center",
        color="black" if cm[i, j] > thresh else "black")

        plt.tight_layout()
        plt.ylabel('True label')
        plt.xlabel('Predicted label')_____no_output_____import itertools
import numpy as np
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt
%matplotlib inline
%config InlineBackend.figure_format='retina'

cm = confusion_matrix(y_true=y_test, y_pred=y_actual)

plt.figure()
fig, ax = plt.subplots(figsize=(10,5))
plot_conf_mat(cm,
              classes=['1', '2', '3', '4', '5'],
              title='Confusion Matrix')
plt.show()[[ 6  7  3 16 24]
 [ 0  1  0  0  0]
 [ 0  0  0  0  0]
 [14  6  2  5 16]
 [ 0  0  0  0  0]]
%%javascript
Jupyter.notebook.save_checkpoint();
Jupyter.notebook.session.delete();_____no_output_____


# Notebook from edencfc/LearningPaddle
Path: notebooks/脚本任务示例合集：PaddleDetection、PGL、Parakeet等.ipynb

> 更新说明：
> 2020.09.26 新增PaddleDetection脚本任务示例：[PaddleDetection：PCB瑕疵检测](https://aistudio.baidu.com/aistudio/clusterprojectdetail/1029223)，暂时只跑通P40单卡训练
# 前言

- 如果任务要训练很久如何不掉线？
- 能不能同时开多个项目训练？
- 想试用多卡训练训练的功能却不知道脚本项目怎么入门？

使用AI Studio平台时，绝大部分时间我们都在Notebook环境下开发，但是Notebook项目有数量和离线时间限制。

 本文整理了一些适合使用脚本任务训练的示例，同时提供了Session-based Recommendation SR-GNN在自定义数据集上脚本任务简单使用说明，后续会持续补充各场景下更多脚本任务示例。

如果您想使用自己业务场景下的数据，只要令数据满足对应脚本任务数据格式要求即可开始训练。
_____no_output_____# 脚本任务结构解读
![file](https://ai-studio-static-online.cdn.bcebos.com/c93f730754234f0a8b5e68ddf669817393d1722a4c524d77b1f8234e44282f16)

STEP 1 模拟终端

由于脚本任务没有终端，因此将标记为【main】的`train.py`改名为`run.py`，起到和终端输入命令相同的功能。

STEP 2 指定百度源

从日志看脚本任务内置的镜像源存在问题，因此需要手动指定百度源等国内镜像，下载好必须的依赖。

STEP 3 安装正确的飞桨版本

脚本任务内置的框架较少，一些任务需要用到1.6、1.7等，建议直接`pip`安装对应版本。

STEP 4 准备数据

1. 直接上传：脚本任务可以上传<30M的数据。
2. 数据集挂载：超过30M的数据集，可以先上传到数据集中再挂载，注意设置正确的路径。

STEP 5 训练并下载输出

1. 在`run.py`中指定要运行的python文件开始训练。
2. 训练完成后，将需要下载的文件移动到输出目录，待脚本任务完成即可下载。

## $\color{red}{一些注意事项}$
1. 脚本任务不能创建空目录，即使创建了，运行时也会消失。
2. 脚本任务可能会出现I/O报错，一些对策如下：
-  如果需要从数据集解压缩文件到data等指定目录，该data目录需要是在`run.py`中用`mkdir`命令创建的
-  尽量等训练全部完成再将需要下载的文件移动到下载输出目录
3.脚本任务目前尚不能解析YAML和Markdown等文件格式，但是各有个小技巧，将文件后缀改成.py就可以查看内容了_____no_output_____# PaddleDetection脚本任务示例

PaddleDetection飞桨目标检测开发套件，旨在帮助开发者更快更好地完成检测模型的组建、训练、优化及部署等全开发流程。

PaddleDetection模块化地实现了多种主流目标检测算法，提供了丰富的数据增强策略、网络模块组件（如骨干网络）、损失函数等，并集成了模型压缩和跨平台高性能部署能力。

经过长时间产业实践打磨，PaddleDetection已拥有顺畅、卓越的使用体验，被工业质检、遥感图像检测、无人巡检、新零售、互联网、科研等十多个行业的开发者广泛应用。

## 使用说明

1. PaddleDetection的脚本任务情况相对特殊，需要先通过数据集挂载加载好PaddleDetecion源码，再解压缩。
![file](https://ai-studio-static-online.cdn.bcebos.com/be3096d2f0df458590a981ae0372964b32cc0d8f4af449b5a956d404a7c91abf)

2. 对于PaddleDetection训练需要指定的自定义config文件，需要在本地调好，上传到编辑界面，然后指定绝对路径开始训练
![file](https://ai-studio-static-online.cdn.bcebos.com/cf7b5074e58047a7a4f436af6737d8b5f2fd07ceb8624b02a784a3adde393e78)_____no_output_____## [PaddleDetection：PCB瑕疵检测](https://aistudio.baidu.com/aistudio/clusterprojectdetail/1029223)_____no_output_____# PGL：图学习脚本任务示例合集

Paddle Graph Learning (PGL)是一个基于PaddlePaddle的高效易用的图学习框架，本项目中的示例整理自文档的[examples目录](https://github.com/paddlepaddle/PGL/tree/master/examples)，均已测试跑通。

关于PGL的入门教程，[自尊心3](https://aistudio.baidu.com/aistudio/personalcenter/thirdview/218586)大佬的[PGL：Paddle带你走进图学习](https://aistudio.baidu.com/aistudio/projectdetail/413386)系列详细代码解读一定不可错过。

![file](https://gitee.com/paddlepaddle/PGL/raw/master/docs/source/_static/framework_of_pgl.png)_____no_output_____## [GCN：图卷积网络](https://aistudio.baidu.com/aistudio/clusterprojectdetail/881609)
图卷积神经网络[Graph Convolutional Network (GCN)](https://arxiv.org/abs/1609.02907)是图学习中的一个基本的网络结构。

> ### 数据集
> 3个常用的图学习数据集，CORA, PUBMED, CITESEER。数据集的相关介绍参考[论文：Semi-Supervised Classification with Graph Convolutional Networks](https://arxiv.org/abs/1609.02907)。

参考资料：
- [PGL系列4：Graph Convolutional Networks](https://aistudio.baidu.com/aistudio/projectdetail/409400)
- [何时能懂你的心——图卷积神经网络（GCN）](https://zhuanlan.zhihu.com/p/71200936)
## [GAT：基于Attention的图卷积网络](https://aistudio.baidu.com/aistudio/projectdetail/881633)

[Graph Attention Networks (GAT)](https://arxiv.org/abs/1710.10903)网络是GCN网络结构改进版本，通过学习自注意层，给节点分配权重，以解决图卷积或其近似方法的缺点。

> ### 数据集
> CORA, PUBMED, CITESEER。

参考资料：
- [PGL系列5：Graph Attention Networks](https://aistudio.baidu.com/aistudio/projectdetail/411016)

## [GraphSage：基于邻居采样的大规模图卷积网络](https://aistudio.baidu.com/aistudio/clusterprojectdetail/891560)

GCN要求在一个确定的图中去学习顶点的embedding，无法直接泛化到在训练过程没有出现过的顶点，即属于一种直推式(transductive)的学习。GraphSAGE则是一种能够利用顶点的属性信息高效产生未知顶点embedding的一种归纳式(inductive)学习的框架，其核心思想是通过学习一个对邻居顶点进行聚合表示的函数来产生目标顶点的embedding向量。

> ### 数据集Reddit Dataset
>
> 使用的是Reddit数据集。这个数据集包含了不同Reddit帖子和所属的社区，用来对帖子所属社区进行分类的任务。Reddit是一个大型在线讨论论坛，用户可以在其中发布和评论不同主题社区中的内容。这个数据集手机了2014年9月发布的帖子，构建了一个图形数据集。在这个数据集中，节点标签是在帖子所属的社区或者“subreddit”。数据集对50个大型社区进行了抽样，并构建了一个帖子到帖子的连接图表。如果一个用户在两个帖子上都发表评论，则将这两个帖子连接起来。该数据及总共包含了232965个帖子，每个节点的平均度为294.

参考资料：
- [【Graph Neural Network】GraphSAGE: 算法原理，实现和应用](https://zhuanlan.zhihu.com/p/79637787)
- [PGL系列8：GraphSAGE](https://aistudio.baidu.com/aistudio/projectdetail/519067)

## [unSup-GraphSage：无监督学习的GraphSAGE](https://aistudio.baidu.com/aistudio/projectdetail/891796)

GraphSAGE在无监督学习任务上的应用。

## [LINE：基于一阶、二阶邻居的表示学习](https://aistudio.baidu.com/aistudio/projectdetail/891878)

LINE: Large-scale Information Network Embedding主要解决了**大规模**网络嵌入到低维向量。而低维向量能够更加有助于visualization, node classification, 和link prediction。这个方法能够处理各种各样的网络，有向无向图，带不带权等等。论文作者认为文章主要有如下两个亮点：

- 优化的目标函数设计的比较好
- 边采样方法能够克服(address)传统的随机梯度下降方法的缺陷，同时提高最后结果的效率和效果（efficiency and effectiveness）。

### 数据集

> LINE案例中使用的数据集是[Flickr社交网络](http://socialnetworks.mpi-sws.org/data-imc2007.html)，该数据集是论文[Measurement and Analysis of Online Social Networks Systems](https://link.springer.com/referenceworkentry/10.1007/978-1-4939-7131-2_242)的发布的三个数据集之一，在会议Internet Measurement Conference2017中发表。
>
> Flickr是雅虎旗下图片分享网站，为一家提供免费及付费数位照片储存、分享方案之线上服务，也提供网络社群服务的平台。其重要特点就是基于社会网络的人际关系的拓展与内容的组织。这个网站的功能之强大，已超出了一般的图片服务，比如图片服务、联系人服务、组群服务。更多可以查看[百度百科flickr](https://baike.baidu.com/item/flickr/714868)
>
> 在这个Flickr数据集中，包含 1,715,256个节点，22,613,981条边（无向且不加权）。节点的平均度 26.37。
>
> 所有节点分为5个类别（组），每个节点包含一个或多个标签，既每个节点可能包含于一个或多个组。

参考资料：
- [LINE:Large-scale Information Network Embedding阅读笔记](https://zhuanlan.zhihu.com/p/27037042)
- [LINE：大规模信息网络嵌入](https://www.shintaku.top/posts/line/)
- [【论文笔记】LINE：大规模信息网络嵌入](https://blog.csdn.net/wizardforcel/article/details/95016540)
- [LINE：Large-scale Information Network Embedding翻译](https://www.cnblogs.com/bianque/articles/10771029.html)
- [PGL系列12：LINE大型信息网络嵌入](https://aistudio.baidu.com/aistudio/projectdetail/655814)

## [DeepWalK：DFS随机游走的表示学习](https://aistudio.baidu.com/aistudio/clusterprojectdetail/895869)

Deepwalk是一种将随机游走(random walk)和word2vec两种算法相结合的图结构数据挖掘算法。该算法主要分为随机游走和生成表示向量两个部分。首先利用随机游走算法(Random walk)从图中提取一些顶点序列；然后借助自然语言处理的思路，将生成的定点序列看作由单词组成的句子，所有的序列可以看作一个大的语料库(corpus)，最有利用自然语言处理工具word2vec将每一个顶点表示为一个维度为d的向量。

### 数据集
> BlogCatalog数据集是一个社会关系网络，图是由博主和他（她）的社会关系（比如好友）组成，labels是博主的兴趣爱好。Reddit数据集是由来自Reddit论坛的帖子组成，如果两个帖子被同一人评论，那么在构图的时候，就认为这两个帖子是相关联的，labels就是每个帖子对应的社区分类。Epinions是一个从一个在线商品评论网站收集的多图数据集，里面包含了多种关系，比如评论者对于另一个评论者的态度（信任/不信任），以及评论者对商品的评级。
>
> 文件构成
> BlogCatalog数据集的结点数为10312，边条数为333983，label维度为39，数据集包含两个文件：
> - Nodes.csv：以字典的形式存储用户的信息，但是只包含节点id。
> - Edges.csv：存储博主的社交网络（好友等），以此来构图。

参考资料：
感谢[没入门的研究生](https://aistudio.baidu.com/aistudio/personalcenter/thirdview/300157)分享的论文解读：
- [DeepWalk：火爆的图神经网络前夜](https://aistudio.baidu.com/aistudio/projectdetail/532867)
- [DeepWalk（续）：如何处理大型网络](https://aistudio.baidu.com/aistudio/projectdetail/534062)

## [MetaPath2Vec：基于metapath的表示学习](https://aistudio.baidu.com/aistudio/projectdetail/897008)

MetaPath2Vec: Scalable Representation Learning for Heterogeneous Networks研究的是关于异构网络的特征表示学习。异构网络的最大挑战来源于不同种类的节点与连接，因此限制了传统network embedding的可行性。论文提出了两种特征学习模型：metapath2vec以及metapath2vec++，它们的具体做法是基于元路径的随机游走来指定一个节点的邻居，之后利用异构skip-gram模型来实现embedding。

### 数据集
> DBLP——Digital Bibliography & Library Project的缩写。这里是[DBLP的主页](http://www.informatik.uni-trier.de/~ley/db/)
>
> DBLP是计算机领域内对研究的成果以作者为核心的一个计算机类英文文献的集成数据库系统，按年代列出了作者的科研成果。包括国际期刊和会议等公开发表的论文。DBLP没有提供对中文文献的收录和检索功能，国内类似的权威期刊及重要会议论文集成检索系统有C-DBLP。
>
> 这个项目是德国特里尔大学的Michael Ley负责开发和维护。它提供计算机领域科学文献的搜索服务，但只储存这些文献的相关元数据，如标题，作者，发表日期等。和一般流行的情况不同，DBLP并没有使用数据库而是使用XML存储元数据。
>
> 示例项目使用了其中部分数据。

参考资料：
- [论文链接](https://ericdongyx.github.io/papers/KDD17-dong-chawla-swami-metapath2vec.pdf)
- [metapath2vec: Scalable Representation Learning for Heterogeneous Networks 阅读笔记](https://zhuanlan.zhihu.com/p/32598703)
## [Node2Vec：结合DFS及BFS的表示学习](https://aistudio.baidu.com/aistudio/projectdetail/897176)

Node2vec是用来产生网络中节点向量的模型，输入是网络结构（可以无权重），输出是每个节点的向量。

### 数据集
> Node2Vec示例中提供了BlogCatalog和Arxiv两种数据集的实现。

参考资料：
- [PGL系列7：node2vec](https://aistudio.baidu.com/aistudio/projectdetail/481549)

## [Struct2Vec：基于结构相似的表示学习](https://aistudio.baidu.com/aistudio/projectdetail/897467)

与很多现有的做网络表示的工作不同，Struct2Vec关注的是不同的节点在网络中的所处的角色。两个不近邻的顶点也可能拥有很高的相似性，可能这些节点在邻域中的角色相似，例如星型结构的中心节点，社区结构之间的桥接节点等。struc2vec就是针对捕捉节点的结构化角色相似度（structural role proximity）提出的模型。

> ### 数据集
>
> 美国空中交通网络数据集 usa-airports.edgelist labels-usa-airports.txt 数据收集来自[Bureau of Transportation Statistics](https://www.transtats.bts.gov/)，记录了美国2016年一月和十月之间的航班活动情况。网络数据有1190个节点，13599个边。机场的繁忙程度（label）使用机场的出入人数，按照排序分级。

参考资料：
- [【论文笔记】struc2vec](https://zhuanlan.zhihu.com/p/63175042)
- [PGL系列14：Struc2Vec](https://aistudio.baidu.com/aistudio/projectdetail/897514)

## [SGC：简化的图卷积网络](https://aistudio.baidu.com/aistudio/projectdetail/897570)

SGC(Simplifying Graph Convolutional Networks)是基于图卷积网络GCN简化而来。SGC的目的就是要把非线性的GCN转化成一个简单的线性模型，通过消除GCN层之间的非线性操作和缩小权重矩阵，将得到的函数折叠成一个线性变换来减少GCNs的额外复杂度。论文证明了SGC相当于一个固定的低通道滤波器和一个线性分类器，SGC中的特征提取等价在每个特征的维度上应用了单个固定的filter。实验结果表明，这些简化不会对许多下游应用的准确性产生负面影响。此外，得到的SGC模型可扩展到更大的数据集，并且比FastGCN产生两个数量级的加速。

> ### 数据集
> CORA, PUBMED, CITESEER。

参考资料：
- [PGL系列13：SGC: 简化的GCNs](https://aistudio.baidu.com/aistudio/projectdetail/739861)

## [DGI：基于图卷积网络的无监督表示学习](https://aistudio.baidu.com/aistudio/clusterprojectdetail/898132)

Deep Graph Infomax (DGI)是一种通用的无监督图学习方法，它可以用于学习节点的嵌入向量。DGI通过最大化深度神经网络编码器的输入和输出之间的互信息，研究无监督表示学习。

> ### 数据集
> CORA, PUBMED, CITESEER。

参考资料：
- [PGL系列9： DGI: Deep Graph Infomax](https://aistudio.baidu.com/aistudio/projectdetail/548667)

## [GATNE：一种针对 Multi-Edge 的大规模异构图嵌入模型](https://aistudio.baidu.com/aistudio/clusterprojectdetail/872114)

清华大学和达摩院合作的一篇论文《Representation Learning for Attributed Multiplex Heterogeneous Network》，发表于 KDD 2019。

目前很多 Graph Embedding 应用广泛，但大部分都只是同构网络或者是小尺度网络，而真实世界往往大都是数以亿计的不同类型的节点和边，且节点往往包含多种属性。

为此，作者提出了 GATNE 框架用于解决大规模多元异构属性网络（Attributed Multiplex Heterogeneous Network，AMHEN），该框架支持 transductive 和 inductive 的学习范式。

此外，作者也进行了理论分析证明了 GATNE 具有良好的表达能力，并通过四种不同的数据集和 A/B 测试验证了模型的性能。

![https://ucc.alicdn.com/pic/developer-ecology/d1799b1f3ca74a3b9dbf8cb68c51d0ea.png](https://ucc.alicdn.com/pic/developer-ecology/d1799b1f3ca74a3b9dbf8cb68c51d0ea.png)

参考资料：
- [Representation Learning for Attributed Multiplex Heterogeneous Network](https://arxiv.org/pdf/1905.01669.pdf)
- [KDD 2019 | GATNE：一种针对 Multi-Edge 的大规模异构图嵌入模型](https://developer.aliyun.com/article/714557)
- [PGL系列15：GATNE](https://aistudio.baidu.com/aistudio/projectdetail/978916)
_____no_output_____# Parakeet：语音合成脚本任务示例
飞桨语音合成套件，提供了灵活、高效、先进的文本到语音合成工具，帮助开发者更便捷高效地完成语音合成模型的开发和应用。本项目中的示例整理自文档的[examples目录](https://github.com/PaddlePaddle/Parakeet/tree/develop/examples)，均已测试跑通。
## [脚本任务示例：Parakeet——WaveFlow语音合成模型训练](https://aistudio.baidu.com/aistudio/projectdetail/629531)
关于该项目的详细说明，可参考:
- [Parakeet：手把手教你训练语音合成模型（脚本任务、Notebook）](https://aistudio.baidu.com/aistudio/projectdetail/639029)
- [Parakeet：飞桨，你是个成熟的框架了，要学会自己读论文](Parakeet：飞桨，你是个成熟的框架了，要学会自己读论文)_____no_output_____# 推荐系统：[SR-GNN脚本任务示例应用](https://aistudio.baidu.com/aistudio/clusterprojectdetail/913252)

## 用户下个月会买什么？

> 该示例参考自：[基于PaddlePaddle的SR-GNN推荐算法](https://aistudio.baidu.com/aistudio/projectdetail/124382)

原项目中复现了论文效果，在DIGINETICA数据集上P@20可以达到50.7；本文则使用用户购买预测常规赛的数据集[MarTech Track1](https://aistudio.baidu.com/aistudio/datasetdetail/19383)，将其迁移为一个预测下月用户购买场景的任务，测试集上P@3可以达到24.3。

脚本任务项目中目录结构及说明：

```text
.
├── run.py            # 运行脚本
├── train.py             # 训练脚本
├── eval.py             # 验证脚本
├── infer.py             # 预测脚本
├── network.py           # 网络结构
├── reader.py            # 和读取数据相关的函数
├── data/martech
    ├── config.txt        # 用户购买商品数量
    ├── train.txt       # 生成的二进制训练集文件
    ├── test.txt       # 生成的二进制测试集文件
```

运行效果：

![file](https://ai-studio-static-online.cdn.bcebos.com/6944648abb58449c8ee0dc0e30d57d31aa447fcd2be443fdbce61ce9925c6509)

## 简介

SR-GNN模型的介绍可以参阅论文[Session-based Recommendation with Graph Neural Networks](https://arxiv.org/abs/1811.00855)。

本文解决的是Session-based Recommendation这一问题,过程大致分为以下四步：

是对所有的session序列通过有向图进行建模。

然后通过GNN，学习每个node（item）的隐向量表示

然后通过一个attention架构模型得到每个session的embedding

最后通过一个softmax层进行全表预测

模型结构如下图：
![file](https://ai-studio-static-online.cdn.bcebos.com/3395a79e26434c18bbe798bfe66c7a73b3df92d7bd9041f6a275ba5b8ebfa245)
_____no_output_____SR-GNN最初处理的是用户点击session预测的问题，运行数据预处理之后在data文件夹下会产生diginetica文件夹，里面包含config.txt、test.txt、train.txt三个文件

生成的数据格式为:(session_list, label_list)。

其中session_list是一个session的列表，其中每个元素都是一个list，代表不同的session。label_list是一个列表，每个位置的元素是session_list中对应session label。

例子：session_list=[[1,2,3], [4], [7,9]]。代表这个session_list包含3个session，第一个session包含的item序列是1,2,3，第二个session只有1个item 4，第三个session包含的item序列是7，9。

label_list = [6, 9, 1]。代表[1,2,3]这个session的预测label值应该为6，后两个以此类推。

在本文中，则将用户在一段时间内购买的商品按顺序排列为一个session，用户下月购买的商品作为label，将所有用户的全部购买情况整理成对应的(session_list, label_list)数据格式。

处理后的数据文件config.txt、test.txt、train.txt三个文件下载后即可上传到脚本任务中训练。_____no_output_____### 数据预处理_____no_output_____

import numpy as np
import pandas as pd
import pickle
from sklearn.preprocessing import RobustScaler, LabelEncoder
tmp = pd.read_csv('data/data19383/train.csv')_____no_output_____tmp = tmp[tmp.order_pay_time > '2013-02-01'][tmp.order_pay_time < '2013-09-01']
tmp['goods_id_ori'] = tmp['goods_id']/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/ipykernel_launcher.py:1: UserWarning: Boolean Series key will be reindexed to match DataFrame index.
  """Entry point for launching an IPython kernel.
# 对商品id进行编码
tmp['goods_id'] = LabelEncoder().fit_transform(tmp['goods_id'])_____no_output_____df_1 = tmp[['customer_id', 'goods_id', 'order_pay_time']][tmp.order_pay_time > '2013-01-01'][tmp.order_pay_time < '2013-04-01']
df_1 = df_1.sort_values(['customer_id', 'order_pay_time'])
df_1 = df_1['goods_id'].groupby(df_1['customer_id']).aggregate(lambda x:list(x)).reset_index()
label = tmp[['customer_id', 'goods_id']][tmp.order_pay_time > '2013-04-01'][tmp.order_pay_time < '2013-05-01']
label.rename(columns={'customer_id':'customer_id','goods_id':'label'}, inplace=True)
df_1 = pd.merge(df_1,label,how='left')
df_1.dropna(0,inplace=True)
df_1['label'] = df_1['label'].astype("int")/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/ipykernel_launcher.py:4: UserWarning: Boolean Series key will be reindexed to match DataFrame index.
  after removing the cwd from sys.path.
df_1.head()_____no_output_____df_2 = tmp[['customer_id', 'goods_id', 'order_pay_time']][tmp.order_pay_time > '2013-02-01'][tmp.order_pay_time < '2013-05-01']
df_2 = df_2.sort_values(['customer_id', 'order_pay_time'])
df_2 = df_2['goods_id'].groupby(df_2['customer_id']).aggregate(lambda x:list(x)).reset_index()
label = tmp[['customer_id', 'goods_id']][tmp.order_pay_time > '2013-05-01'][tmp.order_pay_time < '2013-06-01']
label.rename(columns={'customer_id':'customer_id','goods_id':'label'}, inplace=True)
df_2 = pd.merge(df_2,label,how='left')
df_2.dropna(0,inplace=True)
df_2['label'] = df_2['label'].astype("int")/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/ipykernel_launcher.py:4: UserWarning: Boolean Series key will be reindexed to match DataFrame index.
  after removing the cwd from sys.path.
df_3 = tmp[['customer_id', 'goods_id', 'order_pay_time']][tmp.order_pay_time > '2013-03-01'][tmp.order_pay_time < '2013-06-01']
df_3 = df_3.sort_values(['customer_id', 'order_pay_time'])
df_3 = df_3['goods_id'].groupby(df_3['customer_id']).aggregate(lambda x:list(x)).reset_index()
label = tmp[['customer_id', 'goods_id']][tmp.order_pay_time > '2013-06-01'][tmp.order_pay_time < '2013-07-01']
label.rename(columns={'customer_id':'customer_id','goods_id':'label'}, inplace=True)
df_3 = pd.merge(df_3,label,how='left')
df_3.dropna(0,inplace=True)
df_3['label'] = df_3['label'].astype("int")/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/ipykernel_launcher.py:1: UserWarning: Boolean Series key will be reindexed to match DataFrame index.
  """Entry point for launching an IPython kernel.
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/ipykernel_launcher.py:4: UserWarning: Boolean Series key will be reindexed to match DataFrame index.
  after removing the cwd from sys.path.
df_4 = tmp[['customer_id', 'goods_id', 'order_pay_time']][tmp.order_pay_time > '2013-04-01'][tmp.order_pay_time < '2013-07-01']
df_4 = df_4.sort_values(['customer_id', 'order_pay_time'])
df_4 = df_4['goods_id'].groupby(df_4['customer_id']).aggregate(lambda x:list(x)).reset_index()
label = tmp[['customer_id', 'goods_id']][tmp.order_pay_time > '2013-07-01'][tmp.order_pay_time < '2013-08-01']
label.rename(columns={'customer_id':'customer_id','goods_id':'label'}, inplace=True)
df_4 = pd.merge(df_4,label,how='left')
df_4.dropna(0,inplace=True)
df_4['label'] = df_4['label'].astype("int")/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/ipykernel_launcher.py:1: UserWarning: Boolean Series key will be reindexed to match DataFrame index.
  """Entry point for launching an IPython kernel.
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/ipykernel_launcher.py:4: UserWarning: Boolean Series key will be reindexed to match DataFrame index.
  after removing the cwd from sys.path.
df_5 = tmp[['customer_id', 'goods_id', 'order_pay_time']][tmp.order_pay_time > '2013-05-01'][tmp.order_pay_time < '2013-08-01']
df_5 = df_5.sort_values(['customer_id', 'order_pay_time'])
df_5 = df_5['goods_id'].groupby(df_5['customer_id']).aggregate(lambda x:list(x)).reset_index()
label = tmp[['customer_id', 'goods_id']][tmp.order_pay_time > '2013-08-01'][tmp.order_pay_time < '2013-09-01']
label.rename(columns={'customer_id':'customer_id','goods_id':'label'}, inplace=True)
df_5 = pd.merge(df_5,label,how='left')
df_5.dropna(0,inplace=True)
df_5['label'] = df_5['label'].astype("int")/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/ipykernel_launcher.py:1: UserWarning: Boolean Series key will be reindexed to match DataFrame index.
  """Entry point for launching an IPython kernel.
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/ipykernel_launcher.py:4: UserWarning: Boolean Series key will be reindexed to match DataFrame index.
  after removing the cwd from sys.path.
df = df_1.append([df_2,df_3,df_4])_____no_output_____# 生成的用户购买序列和标签
df.head()_____no_output_____df = df[['goods_id', 'label']]
df_5 = df_5[['goods_id', 'label']]_____no_output_____tra = (list(df.goods_id), list(df.label))
tes = (list(df_5.goods_id), list(df_5.label))_____no_output_____pickle.dump(tra, open('gnn/data/martech/train.txt', 'wb'))_____no_output_____pickle.dump(tes, open('gnn/data/martech/test.txt', 'wb'))_____no_output_____goods_seq = tmp[['goods_id','order_count']][tmp.order_pay_time > '2013-01-01'][tmp.order_pay_time < '2013-09-01']_____no_output_____with open("gnn/data/martech/config.txt", "w") as fout:
    fout.write(str(len(goods_seq['goods_id'].unique())) + "\n")_____no_output_____

### 训练和验证_____no_output_____

#使用GPU训练
!cd gnn && python train.py --train_path './data/martech/train.txt' --config_path './data/martech/config.txt' --use_cuda 1 --epoch_num 20 --model_path './saved_model'_____no_output_____!cd gnn && python eval.py --model_path './saved_model' --test_path './data/martech/test.txt' --use_cuda 12020-09-17 13:37:10,041-WARNING: paddle.fluid.layers.create_py_reader_by_data() may be deprecated in the near future. Please use paddle.fluid.io.DataLoader.from_generator() instead.
W0917 13:37:10.067281 11818 device_context.cc:252] Please NOTE: device: 0, CUDA Capability: 70, Driver API Version: 9.2, Runtime API Version: 9.0
W0917 13:37:10.071609 11818 device_context.cc:260] device: 0, cuDNN Version: 7.6.
2020-09-17 13:37:15,392-INFO: TEST -->Recall@20: 0.2432
#预测,此处仅展示一例，按升序显示下月用户购买可能性最高的前三商品
!cd gnn && python infer.py --model_path './saved_model/' --test_path './data/martech/test.txt' --config_path './data/martech/config.txt' --use_cuda 12020-09-17 13:37:56,643-WARNING: paddle.fluid.layers.create_py_reader_by_data() may be deprecated in the near future. Please use paddle.fluid.io.DataLoader.from_generator() instead.
W0917 13:37:56.669899 12030 device_context.cc:252] Please NOTE: device: 0, CUDA Capability: 70, Driver API Version: 9.2, Runtime API Version: 9.0
W0917 13:37:56.674283 12030 device_context.cc:260] device: 0, cuDNN Version: 7.6.
pred: [134 437 156]
# 打印正确的标签
pickle.load(open('gnn/data/martech/test.txt', 'rb'))[1][14]_____no_output_____


# Notebook from pydae/pydae
Path: examples/grids/smib_milano_ex8p1/smib_milano_ex8p1_4ord_vsg_1/smib_milano_ex8p1_4ord_vsg_1_builder.ipynb

import numpy as np
import sympy as sym
import numba
import pydae.build as db_____no_output_____

### Electromechanical differential equations
\begin{eqnarray}
f_1 &=& \dot \delta = \Omega_b \left( \omega - \omega_s  \right) \\
f_2 &=& \dot \omega = \frac{1}{2H} \left( p_m - p_e - D   \left( \omega - \omega_s \right) \right)
\end{eqnarray}

### Electric rotor differential equations
\begin{eqnarray}
f_3 &=& \dot e_q' =  \frac{1}{T'_{d0}} \left( -e'_q - \left(X_d - X'_d \right) i_d + v_f^\star \right) \\
f_4 &=& \dot e'_d = \frac{1}{T'_{q0}} \left( -e'_d - \left(X_q - X'_q \right) i_q \right)
\end{eqnarray}

### Park transform
\begin{eqnarray}
g_1 &=&-v_d + v_t \sin\left(\delta - \theta_t\right) \\
g_2 &=&-v_q + v_t \cos\left(\delta - \theta_t\right)
\end{eqnarray}

### Stator equations
\begin{eqnarray}
g_3 &=& v_q + R_a i_q + X'_d i_d - e'_q\\
g_4 &=& v_d + R_a i_d - X'_q i_q - e'_d\\
\end{eqnarray}

### Powers
\begin{eqnarray}
g_5 &=& -p_e + \left( v_q + R_a i_q  \right) i_q + \left( v_d + R_a i_d  \right) i_d \\
g_6 &=&  i_d v_d + i_q v_q - p_t \\
g_7 &=&  i_d v_q - i_q v_d - q_t
\end{eqnarray}

### Network equations
\begin{eqnarray}
g_8 &=&   p_t - \left(v_t V_0 \sin\left(\theta_t - \Theta_0\right)\right)/X_l\\
g_9 &=&   q_t + \left(v_t V_0 \cos\left(\theta_t - \Theta_0\right)\right)/X_l - v_t^2/X_l
\end{eqnarray}

_____no_output_____## System definition _____no_output_____

params_dict = {'X_d':1.81,'X1d':0.3, 'T1d0':8.0,  # synnchronous machine d-axis parameters
               'X_q':1.76,'X1q':0.65,'T1q0':1.0,  # synnchronous machine q-axis parameters
               'R_a':0.003,
               'X_l': 0.02,
               'H':3.5,'D':0.0,
               'Omega_b':2*np.pi*50,'omega_s':1.0,
               'V_0':1.0,'Theta_0':0.0}

u_ini_dict = {'p_m':0.8,'v_f':1.0}  # for the initialization problem
u_run_dict = {'p_m':0.8,'v_f':1.0}  # for the running problem (here initialization and running problem are the same)

x_list = ['delta','omega','e1q','e1d']    # dynamic states
y_ini_list = ['i_d','i_q','p_t','q_t','v_t','theta_t']
y_run_list = ['i_d','i_q','p_t','q_t','v_t','theta_t']

sys_vars = {'params':params_dict,
            'u_list':u_run_dict,
            'x_list':x_list,
            'y_list':y_run_list}

exec(db.sym_gen_str())  # exec to generate the required symbolic varables and constants_____no_output_____# auxiliar equations
v_d = v_t*sin(delta - theta_t)  # park
v_q = v_t*cos(delta - theta_t)  # park

p_e = i_d*(v_d + R_a*i_d) + i_q*(v_q + R_a*i_q) # electromagnetic power

# dynamic equations
ddelta = Omega_b*(omega - omega_s) - 1e-4*delta  # load angle
domega = 1/(2*H)*(p_m - p_e - D*(omega - omega_s)) # speed
de1q = 1/T1d0*(-e1q - (X_d - X1d)*i_d + v_f)
de1d = 1/T1q0*(-e1d + (X_q - X1q)*i_q)

# algrbraic equations
g_1 = v_q + R_a*i_q + X1d*i_d - e1q # stator
g_2 = v_d + R_a*i_d - X1q*i_q - e1d # stator
g_3 = i_d*v_d + i_q*v_q - p_t # active power
g_4 = i_d*v_q - i_q*v_d - q_t # reactive power
g_5 = p_t - (v_t*V_0*sin(theta_t - Theta_0))/X_l  # network equation (p)
g_6 = q_t + (v_t*V_0*cos(theta_t - Theta_0))/X_l - v_t**2/X_l  # network equation (q)                _____no_output_____sys = {'name':'smib_milano_ex8p1_4ord',
       'params_dict':params,
       'f_list':[ddelta,domega,de1q,de1d],
       'g_list':[g_1,g_2,g_3,g_4,g_5,g_6],
       'x_list':x_list,
       'y_ini_list':y_ini_list,
       'y_run_list':y_run_list,
       'u_run_dict':u_run_dict,
       'u_ini_dict':u_ini_dict,
       'h_dict':{'p_m':p_m,'p_e':p_e, 'v_f':v_f}}

sys = db.system(sys)
db.sys2num(sys)_____no_output_____


# Notebook from pobonomo/modeling-examples
Path: protein_folding/protein_folding.ipynb

# Protein Folding

## Objective and Prerequisites

Hone your modeling skills with this challenging Protein Folding problem. We’ll show you how to create a binary optimization model of the problem with the Gurobi Python API and then solve it using the Gurobi Optimizer.

This model is example 28 from the fifth edition of Model Building in Mathematical Programming by H. Paul Williams on pages 289-290 and 344-345.

This modeling example is at the advanced level, where we assume that you know Python and the Gurobi Python API and that you have advanced knowledge of building mathematical optimization models. Typically, the objective function and/or constraints of these examples are complex or require advanced features of the Gurobi Python API.

**Download the Repository**

You can download the repository containing this and other examples by clicking [here](https://github.com/Gurobi/modeling-examples/archive/master.zip).

**Gurobi License**

In order to run this Jupyter Notebook properly, you must have a Gurobi license. If you do not have one, you can request an [evaluation license](https://www.gurobi.com/downloads/request-an-evaluation-license/?utm_source=3PW&utm_medium=OT&utm_campaign=WW-MU-MUI-OR-O_LEA-PR_NO-Q3_FY20_WW_JPME_PROTEIN_FOLDING_COM_EVAL_GitHub&utm_term=Protein_Folding&utm_content=C_JPM) as a *commercial user*, or download a [free license](https://www.gurobi.com/academia/academic-program-and-licenses/?utm_source=3PW&utm_medium=OT&utm_campaign=WW-MU-EDU-OR-O_LEA-PR_NO-Q3_FY20_WW_JPME_PROTEIN_FOLDING_ACADEMIC_EVAL_GitHub&utm_term=Protein_Folding&utm_content=C_JPM) as an *academic user*._____no_output_____---
## Problem Description

The problem described in this Jupyter Notebook is based on a molecular biology problem that is discussed in a paper entitled “Quadratic Binary Programming Models in Computational Biology” by Forrester and Greenberg (2008). The problem pertains to a protein, which consists of a chain of amino acids. In this problem, the amino acids come in two forms: hydrophilic (waterloving) and hydrophobic (water hating). An example of such a chain is given in the following figure with the hydrophobic acids marked in bold.

![chain](chain.PNG)

Such a chain naturally folds so as to bring as many hydrophobic acids as possible close together. A folding for the chain, in two dimensions, is given in the following figure, with the new matches marked by dashed lines. Our objective is to predict the optimum folding.

![folding](folding.PNG)

To solve the problem posed here, we must find the optimum folding for a chain of 50 amino acids with hydrophobic acids at positions 2, 4, 5, 6, 11, 12, 17, 20, 21, 25, 27, 28, 30, 31, 33, 37, 44 and 46._____no_output_____---
## Model Formulation

### Sets and Indices

$k \in A =\{1,2,...,50\}$: Chain of aminoacids.

$i,j \in H =\{2,4,5,6,11,12,17,20,21,25,27,28,30,31,33,37,44,46\} \subseteq Aminoacids $: Subset of aminoacids that are hydrophobic.

### Decision variables

$\text{match}_{i,j} \equiv x_{i,j} = 1$, iff hydrophobic acid $i$ is matched with acid $j$, for all hydrophobic acids $ i < j \in H$. This matching does not include those matchings that are predefined by virtue of the acids being contiguous in the chain, i.e. $j > i+1$).

$\text{fold}_{k} \equiv y_{k} = 1$, iff a fold occurs between the $i$th and $(i +1)$st acids in the chain.

### Constraints

For each pair of hydrophobic acids $i$ and $j$, we can match them if:
* they are not contiguous, that is, already matched,
* they have an even number of acids between them in the chain,
* and there is exactly one fold between $i$ and $j$ .

This gives rise to the following constraints.

1. $y_{k} + x_{i,j} \leq 1, \; \forall k \in A, (i,j) \in H, \; \text{such that} \; i \leq k < j, \; \text{and} \;
k \neq (i+j-1)/2$.
2. $x_{i,j} \leq y_{k}, \; \text{where} \; k = (i+j-1)/2 $.

Let $\text{H_fold} = \{(i,j) \in H: x_{i,j} \leq y_{k}, \; k = (i+j-1)/2  \}$ be the set of hydrophobic acids for which there is a folding that enables the matching.

### Objective function
The objective is to maximize the number of matchings of hydrophobic acids.

$$
\sum_{i,j \in \text{H_fold}} x_{i,j}
$$_____no_output_____---
## Python Implementation

We import the Gurobi Python Module._____no_output_____

import gurobipy as gp
from gurobipy import GRB

# tested with Python 3.7.0 & Gurobi 9.0_____no_output_____

## Input Data _____no_output_____

# list of aminoacids and hydrophobic acids

acids = [*range(1,51)]

h_phobic = [2,4,5,6,11,12,17,20,21,25,27,28,30,31,33,37,44,46]_____no_output_____

## Preprocessing_____no_output_____

# Creating the data structures to generate the model
list_ij = []

# Indices of hydrophobic acids that can be matched
for i in h_phobic:
    for j in h_phobic:
        if j > i + 1:
            tp = i,j
            list_ij.append(tp)

ij = gp.tuplelist(list_ij)

###
list_ik1j = []

list_ik2j = []

for i,j in ij:
    for k in range(i,j):
        if (k == (i+j-1)/2  ):
            tp = i,j,k
            list_ik2j.append(tp)
        else:
            tp = i,j,k
            list_ik1j.append(tp)

# Indices for constraints of type 2
ik2j = gp.tuplelist(list_ik2j)

# Indices for constraints of type 1
ik1j = gp.tuplelist(list_ik1j)

# Matchings that are enabled by a folding
list_ijfold = []

for i,j,k in ik2j:
    tp = i,j
    list_ijfold.append(tp)

ijfold = gp.tuplelist(list_ijfold)_____no_output_____

## Model Deployment
We create a model and the decision variables. There are two types of decision variables: the variables that determine which hydrophobic acids to match, and the variables that determine at which amino acid the folding of the protein happens._____no_output_____

model = gp.Model('ProteinFolding')

# Matching variables
match = model.addVars(ij, vtype=GRB.BINARY, name="match")

# Folding variables
fold = model.addVars(acids, vtype=GRB.BINARY, name="fold")Using license file c:\gurobi\gurobi.lic

### Folding and matching constraints

1. $y_{k} + x_{i,j} \leq 1, \; \forall k \in A, (i,j) \in H, \; \text{such that} \; i \leq k < j, \; \text{and} \;
k \neq (i+j-1)/2$.
2. $x_{i,j} \leq y_{k}, \; \text{where} \; k = (i+j-1)/2 $.
_____no_output_____

# Constraint 1:

C1 = model.addConstrs( (fold[k] + match[i,j] <= 1 for i,j,k in ik1j ) , name='C1')

# Constraint 2:

C2 = model.addConstrs( ( match[i,j] <= fold[k]  for i,j,k in ik2j ) , name='C2')_____no_output_____

### Objective function

Maximize the matchings of hydrophobic acids. _____no_output_____

# Objective function

model.setObjective(gp.quicksum(match[i,j] for i,j in ijfold) , GRB.MAXIMIZE )
_____no_output_____# Verify model formulation

model.write('ProteinFolding.lp')

# Run optimization engine

model.optimize()Gurobi Optimizer version 9.1.0 build v9.1.0rc0 (win64)
Thread count: 4 physical cores, 8 logical processors, using up to 8 threads
Optimize a model with 2441 rows, 197 columns and 4882 nonzeros
Model fingerprint: 0x7a3a8e69
Variable types: 0 continuous, 197 integer (197 binary)
Coefficient statistics:
  Matrix range     [1e+00, 1e+00]
  Objective range  [1e+00, 1e+00]
  Bounds range     [1e+00, 1e+00]
  RHS range        [1e+00, 1e+00]
Found heuristic solution: objective -0.0000000
Presolve removed 1596 rows and 105 columns
Presolve time: 0.02s
Presolved: 845 rows, 92 columns, 1779 nonzeros
Variable types: 0 continuous, 92 integer (92 binary)

Root relaxation: objective 3.200000e+01, 90 iterations, 0.00 seconds

    Nodes    |    Current Node    |     Objective Bounds      |     Work
 Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

     0     0   32.00000    0   89   -0.00000   32.00000      -     -    0s
H    0     0                       8.0000000   32.00000   300%     -    0s
H    0     0                       9.0000000   32.00000   256%     -    0s
     0     0   21.00000    0   69    9.00000   21.00000   133%     -    0s
     0     0   13.50000    0   81    9.00000   13.50000  50.0%     -    0s
     0     0   12.50000    0   82    9.00000   12.50000  38.9%     -    0s
     0     0   12.50000    0   82    9.00000   12.50000  38.9%     -    0s
     0     0   12.00000    0   82    9.00000   12.00000  33.3%     -    0s
     0     0   12.00000    0   82    9.00000   12.00000  33.3%     -    0s
H    0     0                      10.0000000   11.00000  10.0%     -    0s
     0     0   11.00000    0   76   10.00000   11.00000  10.0%     -    0s
     0     0   11.00000    0   67   10.00000   11.00000  10.0%     -    0s
     0     2   11.00000    0   67   10.00000   11.00000  10.0%     -    0s

Cutting planes:
  Gomory: 2
  Zero half: 17
  RLT: 38

Explored 55 nodes (1222 simplex iterations) in 0.21 seconds
Thread count was 8 (of 8 available processors)

Solution count 4: 10 9 8 -0

Optimal solution found (tolerance 1.00e-04)
Best objective 1.000000000000e+01, best bound 1.000000000000e+01, gap 0.0000%
# Output report

print(f"Optimal number of hydrophobic acids matchings: {model.objVal}")

print("_______________________________________")
print(f"Optimal matching of hydrophobic acids.")
print("_______________________________________")

for i,j,k in ik2j:
    if (match[i,j].x > 0.5):
        print(f"Hydrophobic acid matching {i,j} with folding at amonacid {k}.")

Optimal number of hydrophobic acids matchings: 10.0
_______________________________________
Optimal matching of hydrophobic acids.
_______________________________________
Hydrophobic acid matching (2, 5) with folding at amonacid 3.
Hydrophobic acid matching (5, 12) with folding at amonacid 8.
Hydrophobic acid matching (6, 11) with folding at amonacid 8.
Hydrophobic acid matching (12, 17) with folding at amonacid 14.
Hydrophobic acid matching (17, 20) with folding at amonacid 18.
Hydrophobic acid matching (20, 25) with folding at amonacid 22.
Hydrophobic acid matching (25, 28) with folding at amonacid 26.
Hydrophobic acid matching (28, 31) with folding at amonacid 29.
Hydrophobic acid matching (31, 46) with folding at amonacid 38.
Hydrophobic acid matching (33, 44) with folding at amonacid 38.

---
## References

H. Paul Williams, Model Building in Mathematical Programming, fifth edition.

Forrester, R.J. and Greenberg, H.J. (2008) Quadratic Binary Programming Models in Computational Biology. Algorithmic Operations Research, 3, 110–129.

Copyright © 2020 Gurobi Optimization, LLC_____no_output_____

protein folding
biology
computational biology