# Notebook from VarunM24/data-science-portfolio
Path: Python/Wrangling/Xml-Text-preprocessing.ipynb

Environment: Python 2.7.11 and Jupyter notebook

Libraries used:
* os(deleting files)
* BeautifulSoup(for extracting data from xml files)
* pandas (for dataframe, included in Anaconda Python 2.7)
* re (for regular expression, included in Anaconda Python 2.7)
* numpy (for saving to text file)
* Counter (for counting frequency)
* nltk (for natural language processing)
* re (regular expression)
* itertools (iterating through data structures)

## 1. Introduction
In this notebook, we pre-process text data from a xml file containing patent related information. This involves extracting the  information, storing them in text files, cleaning up and processing the abstract data contained in the xml to create a vocabulary and sparse vector file for possible citation analysis task. This work was done as part of academic requirement.

Assumptions:

- patent.xml is present in same directory as the python file for this code to work properly.
- patentId are stored inside publication-reference tag rather than application-reference tag

Various challenges were faced in this task such as :

- extracting data from patent.xml : The given file is an invalid xml as it has multiple xmls in a single xml file and therefore xml parsing libraries have trouble parsing all the xmls inside it.

  Solution - split the xml content into multiple xml files and then parse them with beautifulsoup library

- Choosing the patent id from 2 different doc-number tags : There are 2 tags which contain the doc-number so it makes deciding the patent id a bit confusing.

 Solution: I will be assuming the doc-number inside publication-reference to be the correct patent id as the word publication seems to be referring to patent publication while application seems to be referring to application instead.

-  Pre-processing abstract data : Order of steps of pre-processing steps after extracting the abstract text and the decision to lemmatize : I chose the following order:

 Sentence segmentation - This is required for lemmatization so it is performed first

 Lemmatization - It will reduce the number of variations in features for analysis, also its 1st in order to facilitate creation of less number of bigrams due to reduction in forms of words

 Tokenization - This will be be done as part of whole lemmatization process and is very essential for all further steps.

 Lower case conversion - Will help in removing some stop words as well as normalizes text for bigram creation, frequent and infrequent word investigation as well as end task analysis. It is performed after lemmatization so that it does not affect that process.

 Bigram creation - Required as part of assignment specification as well as required to create features for text analysis task.
 Removing Stop words, non alphabetical words and symbols - these tokens do not contribute to text analysis task such as classification and are removed after bigram creation to prevent incorrect bigram creation.

 Removal of top 20 frequent tokens and tokens that only appear in one abstract - These are not very valuable as features for text analysis as the frequent tokens will not help in distinguishing the context of citation and extremely rare tokens will not be adding to the features unnecesarily as they will only be present in one citation and will not contribute to context.

_____no_output_____## 2.  Import libraries _____no_output_____

# Code to import libraries
from bs4 import BeautifulSoup
import os
import pandas as pd
import numpy as np
from collections import Counter
import nltk.data
import nltk
from nltk.corpus import stopwords
from nltk.corpus import wordnet
from nltk.stem import WordNetLemmatizer
import re
from nltk.tokenize import MWETokenizer
from nltk.probability import FreqDist
from itertools import chain_____no_output_____

## 3. Examining and loading data_____no_output_____First we will take a look at the xml file given to us in this assignment by using BeautifulSoup library
_____no_output_____

# Code to inspect the provided data file
# Couple of notes for all code block in this notebook

#Opening patents.xml
ifilename = 'patents.xml'
ofilename = 'myfile'
tree = BeautifulSoup(open(ifilename),"lxml-xml")
#print to see the xml
print (tree.prettify())<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE us-patent-grant SYSTEM "us-patent-grant-v42-2006-08-23.dtd">
<us-patent-grant country="US" date-produced="20110208" date-publ="20110222" dtd-version="v4.2 2006-08-23" file="USPP021722-20110222.XML" id="us-patent-grant" lang="EN" status="PRODUCTION">
 <us-bibliographic-data-grant>
  <publication-reference>
   <document-id>
    <country>
     US
    </country>
    <doc-number>
     PP021722
    </doc-number>
    <kind>
     P3
    </kind>
    <date>
     20110222
    </date>
   </document-id>
  </publication-reference>
  <application-reference appl-type="plant">
   <document-id>
    <country>
     US
    </country>
    <doc-number>
     12316880
    </doc-number>
    <date>
     20081216
    </date>
   </document-id>
  </application-reference>
  <us-application-series-code>
   12
  </us-application-series-code>
  <us-term-of-grant>
   <us-term-extension>
    108
   </us-term-extension>
  </us-term-of-grant>
  <classifications-ipcr>
   <classification-ipcr>
    <ipc-version-indicator>
     <date>
      20060101
     </date>
    </ipc-version-indicator>
    <classification-level>
     A
    </classification-level>

     A

    <class>
     01
    </class>
    <subclass>
     H
    </subclass>

     5

    <subgroup>
     00
    </subgroup>
    <symbol-position>
     F
    </symbol-position>
    <classification-value>
     I
    </classification-value>
    <action-date>
     <date>
      20110222
     </date>
    </action-date>
    <generating-office>
     <country>
      US
     </country>
    </generating-office>
    <classification-status>
     B
    </classification-status>
    <classification-data-source>
     H
    </classification-data-source>
   </classification-ipcr>
  </classifications-ipcr>
  <classification-national>
   <country>
    US
   </country>

    PLT161

  </classification-national>
  <invention-title id="d2e53">
   Apple tree named ‘Daligris’
  </invention-title>
  <us-botanic>
   <latin-name>
    Malus domestica Borkh
   </latin-name>
   <variety>
    Daligris
   </variety>
  </us-botanic>
  <references-cited>
   <citation>
    <patcit num="00001">
     <document-id>
      <country>
       US
      </country>
      <doc-number>
       PP17672
      </doc-number>
      <kind>
       P3
      </kind>
      <name>
       Hofmann
      </name>
      <date>
       20070500
      </date>
     </document-id>
    </patcit>
    <category>
     cited by examiner
    </category>
    <classification-national>
     <country>
      US
     </country>

      PLT161

    </classification-national>
   </citation>
   <citation>
    <patcit num="00002">
     <document-id>
      <country>
       US
      </country>
      <doc-number>
       PP18482
      </doc-number>
      <kind>
       P3
      </kind>
      <name>
       Ligonniere
      </name>
      <date>
       20080200
      </date>
     </document-id>
    </patcit>
    <category>
     cited by examiner
    </category>
    <classification-national>
     <country>
      US
     </country>

      PLT161

    </classification-national>
   </citation>
   <citation>
    <patcit num="00003">
     <document-id>
      <country>
       US
      </country>
      <doc-number>
       PP18483
      </doc-number>
      <kind>
       P3
      </kind>
      <name>
       Ligonniere
      </name>
      <date>
       20080200
      </date>
     </document-id>
    </patcit>
    <category>
     cited by examiner
    </category>
    <classification-national>
     <country>
      US
     </country>

      PLT161

    </classification-national>
   </citation>
  </references-cited>
  <number-of-claims>
   1
  </number-of-claims>
  <us-exemplary-claim>
   1
  </us-exemplary-claim>
  <us-field-of-classification-search>
   <classification-national>
    <country>
     US
    </country>

     PLT161

   </classification-national>
  </us-field-of-classification-search>
  <figures>
   <number-of-drawing-sheets>
    4
   </number-of-drawing-sheets>
   <number-of-figures>
    4
   </number-of-figures>
  </figures>
  <us-related-documents>
   <related-publication>
    <document-id>
     <country>
      US
     </country>
     <doc-number>
      20100154089
     </doc-number>
     <kind>
      P1
     </kind>
     <date>
      20100617
     </date>
    </document-id>
   </related-publication>
  </us-related-documents>
  <parties>
   <applicants>
    <applicant app-type="applicant-inventor" designation="us-only" sequence="001">
     <addressbook>
      <last-name>
       Ligonnière
      </last-name>
      <first-name>
       Guy
      </first-name>
      <address>
       <city>
        Angers
       </city>
       <country>
        FR
       </country>
      </address>
     </addressbook>
     <nationality>
      <country>
       omitted
      </country>
     </nationality>
     <residence>
      <country>
       FR
      </country>
     </residence>
    </applicant>
   </applicants>
   <agents>
    <agent rep-type="attorney" sequence="01">
     <addressbook>
      <last-name>
       Bos
      </last-name>
      <first-name>
       Michelle
      </first-name>
      <address>
       <country>
        unknown
       </country>
      </address>
     </addressbook>
    </agent>
   </agents>
  </parties>
  <assignees>
   <assignee>
    <addressbook>
     <orgname>
      S.N.C. Elaris
     </orgname>
     <role>
      03
     </role>
     <address>
      <city>
       Angers
      </city>
      <country>
       FR
      </country>
     </address>
    </addressbook>
   </assignee>
  </assignees>
  <examiners>
   <primary-examiner>
    <last-name>
     McCormick Ewoldt
    </last-name>
    <first-name>
     Susan B
    </first-name>
    <department>
     1661
    </department>
   </primary-examiner>
  </examiners>
 </us-bibliographic-data-grant>
 <abstract id="abstract">

   A new apple tree named ‘Daligris’ is disclosed. The fruit of the new variety is particularly notable for its eating quality and distinctive flavor and appearance. The fruit is very sweet and has a pronounced aniseed flavor, and takes on a distinctive red orange coloration as it ripens on the tree.

 </abstract>
 <drawings id="DRAWINGS">

 </drawings>
 <description id="description">
  <?BRFSUM description="Brief Summary" end="lead"?>

   Latin name:

    Malus domestica

   Borkh.

   Variety denomination: ‘Daligris’.

  <heading id="h-0001" level="1">
   CROSS REFERENCE TO RELATED APPLICATIONS
  </heading>

   None.

  <heading id="h-0002" level="1">
   PRIORITY CLAIM
  </heading>

   None.

  <heading id="h-0003" level="1">
   STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT
  </heading>

   None.

  <heading id="h-0004" level="1">
   BACKGROUND OF THE VARIETY
  </heading>

   ‘Daligris’ is a new and distinct cultivar of apple tree

    Malus domestica

   Borkh. ‘Daligris’ is a whole tree mutation of the apple variety ‘Pinova’ (U.S. Plant Pat. No. 11,601). The original whole tree mutation was discovered during the 1998 harvest in a commercial block of ‘Pinova’ trees near Lezigne-France, planted on M9 rootstocks in 1997. It was first asexually propagated in March 2000 for testing purposes by grafting under the breeder's reference number DDR14-51 at Doue la Fontaine, France. ‘Daligris’ was propagated for further experimentation in 2002.

   The asexually propagated progeny of ‘Daligris’ has consistently been found to be homogeneous and stable, retaining the unique characteristics of the original ‘Daligris’ tree and fruit, and has remained true to type over successive asexually propagated generations. Plant Variety Rights have been applied for ‘Daligris’ in the European Union in 2006 under the application number 2006/1697.

  <?BRFSUM description="Brief Summary" end="tail"?>
  <?brief-description-of-drawings description="Brief Description of Drawings" end="lead"?>
  <description-of-drawings>
   <heading id="h-0005" level="1">
    BRIEF DESCRIPTION OF THE VARIETY
   </heading>

    Trees of ‘Daligris’ have been virus tested, and have been found to be free from the most common apple viruses in Europe (stem pitting, stem grooving, rubbery wood, proliferation, spy decline, chlorotic leaf spot, russet ring, star crack, russet wart, mosaic). Virus free certified material will be available in 2009.

    ‘Daligris’ is quite similar to ‘Pinova’ with respect to tree behavior, flowers, and leaves. However, the fruit of ‘Daligris’ is notably distinct from ‘Pinova,’ and from ‘Dalinip’ (U.S. Plant Pat. No. 18,482), ‘Dalirail’ (U.S. Plant Pat. No. 18,483), and ROHO 3615(U.S. Plant Pat. No. 17,672), each of which is also a mutation of ‘Pinova’.

    <tables id="TABLE-US-00001" num="00001">

      <tgroup align="left" cols="1" colsep="0" rowsep="0">
       <colspec align="center" colname="1" colwidth="217pt"/>

        <row>
         <entry nameend="1" namest="1" rowsep="1">
          TABLE 1
         </entry>
        </row>

        <row>
         <entry align="center" nameend="1" namest="1" rowsep="1"/>
        </row>
        <row>
         <entry>
          Comparison of ‘Daligris’ to ‘Pinova’ and Related Varieties
         </entry>
        </row>
        <row>
         <entry>
          (measurements given are approximate)
         </entry>
        </row>

      </tgroup>
      <tgroup align="left" cols="6" colsep="0" rowsep="0">
       <colspec align="left" colname="1" colwidth="35pt"/>
       <colspec align="left" colname="2" colwidth="35pt"/>
       <colspec align="left" colname="3" colwidth="42pt"/>
       <colspec align="left" colname="4" colwidth="35pt"/>
       <colspec align="left" colname="5" colwidth="35pt"/>
       <colspec align="left" colname="6" colwidth="35pt"/>

        <row>
         <entry>
          Charac-
         </entry>
         <entry/>
         <entry/>
         <entry/>
         <entry/>
         <entry/>
        </row>
        <row>
         <entry>
          teristic of
         </entry>
         <entry/>
         <entry/>
         <entry/>
         <entry/>
         <entry>
          ROHO
         </entry>
        </row>
        <row>
         <entry>
          the fruit
         </entry>
         <entry>
          Pinova
         </entry>
         <entry>
          Daligris
         </entry>
         <entry>
          Dalinip
         </entry>
         <entry>
          Dalirail
         </entry>
         <entry>
          3615
         </entry>
        </row>
        <row>
         <entry align="center" nameend="6" namest="1" rowsep="1"/>
        </row>
        <row>
         <entry>
          Maturity
         </entry>
         <entry>
          Third
         </entry>
         <entry>
          Fourth
         </entry>
         <entry>
          First
         </entry>
         <entry>
          Second
         </entry>
         <entry>
          Third
         </entry>
        </row>
        <row>
         <entry/>
         <entry>
          week of
         </entry>
         <entry>
          week of
         </entry>
         <entry>
          week of
         </entry>
         <entry>
          week of
         </entry>
         <entry>
          week of
         </entry>
        </row>
        <row>
         <entry/>
         <entry>
          September
         </entry>
         <entry>
          September
         </entry>
         <entry>
          September
         </entry>
         <entry>
          September
         </entry>
         <entry>
          September
         </entry>
        </row>
        <row>
         <entry>
          Amount
         </entry>
         <entry>
          Absent
         </entry>
         <entry>
          Typical
         </entry>
         <entry>
          Absent
         </entry>
         <entry>
          Absent
         </entry>
         <entry>
          Absent
         </entry>
        </row>
        <row>
         <entry>
          of
         </entry>
         <entry>
          to very
         </entry>
         <entry>
          light
         </entry>
         <entry>
          to very
         </entry>
         <entry>
          to very
         </entry>
         <entry>
          to very
         </entry>
        </row>
        <row>
         <entry>
          russeting
         </entry>
         <entry>
          small
         </entry>
         <entry>
          brown-
         </entry>
         <entry>
          small
         </entry>
         <entry>
          small
         </entry>
         <entry>
          small
         </entry>
        </row>
        <row>
         <entry/>
         <entry/>
         <entry>
          yellow
         </entry>
         <entry/>
         <entry/>
         <entry/>
        </row>
        <row>
         <entry/>
         <entry/>
         <entry>
          russeting on
         </entry>
         <entry/>
         <entry/>
         <entry/>
        </row>
        <row>
         <entry/>
         <entry/>
         <entry>
          25% to 40%
         </entry>
         <entry/>
         <entry/>
         <entry/>
        </row>
        <row>
         <entry/>
         <entry/>
         <entry>
          of the skin
         </entry>
         <entry/>
         <entry/>
         <entry/>
        </row>
        <row>
         <entry/>
         <entry/>
         <entry>
          surface
         </entry>
         <entry/>
         <entry/>
         <entry/>
        </row>
        <row>
         <entry>
          Type of
         </entry>
         <entry>
          Red 41A
         </entry>
         <entry>
          Red 41A
         </entry>
         <entry>
          Red 44A
         </entry>
         <entry>
          Red 44A
         </entry>
         <entry>
          Intense
         </entry>
        </row>
        <row>
         <entry>
          fore-
         </entry>
         <entry/>
         <entry>
          to Red-
         </entry>
         <entry>
          to Red-
         </entry>
         <entry/>
         <entry>
          red
         </entry>
        </row>
        <row>
         <entry>
          ground
         </entry>
         <entry/>
         <entry>
          Orange
         </entry>
         <entry>
          Orange
         </entry>
         <entry/>
         <entry>
          46A-46B
         </entry>
        </row>
        <row>
         <entry>
          Colour
         </entry>
         <entry/>
         <entry>
          42A
         </entry>
         <entry>
          42A
         </entry>
         <entry/>
         <entry/>
        </row>
        <row>
         <entry>
          Amount
         </entry>
         <entry>
          25-50%
         </entry>
         <entry>
          10-40%
         </entry>
         <entry>
          60-80%
         </entry>
         <entry>
          50-75%
         </entry>
         <entry>
          60-90%
         </entry>
        </row>
        <row>
         <entry>
          of fore-
         </entry>
         <entry/>
         <entry/>
         <entry/>
         <entry/>
         <entry/>
        </row>
        <row>
         <entry>
          ground
         </entry>
         <entry/>
         <entry/>
         <entry/>
         <entry/>
         <entry/>
        </row>
        <row>
         <entry>
          Colour
         </entry>
         <entry/>
         <entry/>
         <entry/>
         <entry/>
         <entry/>
        </row>
        <row>
         <entry>
          Sugar
         </entry>
         <entry>
          High:
         </entry>
         <entry>
          Very high:
         </entry>
         <entry>
          Similar to
         </entry>
         <entry>
          Similar to
         </entry>
         <entry>
          Similar to
         </entry>
        </row>
        <row>
         <entry>
          content
         </entry>
         <entry>
          13.4°
         </entry>
         <entry>
          15.5°
         </entry>
         <entry>
          Pinova
         </entry>
         <entry>
          Pinova
         </entry>
         <entry>
          Pinova
         </entry>
        </row>
        <row>
         <entry>
          Average
         </entry>
         <entry>
          brix
         </entry>
         <entry>
          brix
         </entry>
         <entry>
          (around
         </entry>
         <entry>
          (around
         </entry>
         <entry>
          (around
         </entry>
        </row>
        <row>
         <entry>
          on 20
         </entry>
         <entry/>
         <entry/>
         <entry>
          13° brix)
         </entry>
         <entry>
          13° brix)
         </entry>
         <entry>
          13° brix)
         </entry>
        </row>
        <row>
         <entry>
          fruits from
         </entry>
         <entry/>
         <entry/>
         <entry/>
         <entry/>
         <entry/>
        </row>
        <row>
         <entry>
          compara-
         </entry>
         <entry/>
         <entry/>
         <entry/>
         <entry/>
         <entry/>
        </row>
        <row>
         <entry>
          ble trees
         </entry>
         <entry/>
         <entry/>
         <entry/>
         <entry/>
         <entry/>
        </row>
        <row>
         <entry>
          Taste
         </entry>
         <entry>
          Balanced
         </entry>
         <entry>
          Pro-
         </entry>
         <entry>
          Balanced
         </entry>
         <entry>
          Balanced
         </entry>
         <entry>
          Balanced
         </entry>
        </row>
        <row>
         <entry/>
         <entry>
          in sugar
         </entry>
         <entry>
          nounced
         </entry>
         <entry>
          in sugar
         </entry>
         <entry>
          in sugar
         </entry>
         <entry>
          in sugar
         </entry>
        </row>
        <row>
         <entry/>
         <entry>
          and
         </entry>
         <entry>
          aniseed
         </entry>
         <entry>
          and
         </entry>
         <entry>
          and
         </entry>
         <entry>
          and
         </entry>
        </row>
        <row>
         <entry/>
         <entry>
          acidity
         </entry>
         <entry>
          flavor
         </entry>
         <entry>
          acidity
         </entry>
         <entry>
          acidity
         </entry>
         <entry>
          acidity
         </entry>
        </row>
        <row>
         <entry/>
         <entry>
          with
         </entry>
         <entry/>
         <entry>
          with
         </entry>
         <entry>
          with
         </entry>
         <entry>
          with
         </entry>
        </row>
        <row>
         <entry/>
         <entry>
          medium
         </entry>
         <entry/>
         <entry>
          medium
         </entry>
         <entry>
          medium
         </entry>
         <entry>
          medium
         </entry>
        </row>
        <row>
         <entry/>
         <entry>
          aroma
         </entry>
         <entry/>
         <entry>
          aroma
         </entry>
         <entry>
          aroma
         </entry>
         <entry>
          aroma
         </entry>
        </row>
        <row>
         <entry>
          Size of
         </entry>
         <entry>
          Small
         </entry>
         <entry>
          Very large
         </entry>
         <entry>
          Small
         </entry>
         <entry>
          Small
         </entry>
         <entry>
          Small
         </entry>
        </row>
        <row>
         <entry>
          Lenticels
         </entry>
         <entry>
          (average:
         </entry>
         <entry>
          and very
         </entry>
         <entry>
          (average:
         </entry>
         <entry>
          (average:
         </entry>
         <entry>
          (average:
         </entry>
        </row>
        <row>
         <entry/>
         <entry>
          0.5 mm)
         </entry>
         <entry>
          prominent
         </entry>
         <entry>
          0.5 mm)
         </entry>
         <entry>
          0.5 mm)
         </entry>
         <entry>
          0.5 mm)
         </entry>
        </row>
        <row>
         <entry/>
         <entry/>
         <entry>
          (average:
         </entry>
         <entry/>
         <entry/>
         <entry/>
        </row>
        <row>
         <entry/>
         <entry/>
         <entry>
          1.2 mm)
         </entry>
        </row>
        <row>
         <entry align="center" nameend="6" namest="1" rowsep="1"/>
        </row>

      </tgroup>

    </tables>

   <heading id="h-0006" level="1">
    The Tree
   </heading>

    The tree of ‘Daligris’ is a bit more dwarf than ‘Pinova’. The branches, leaves and flowers of ‘Daligris’ are similar to ‘Pinova’. The bloom is also similar to ‘Pinova’ in terms of bloom date range and intensity. Like its parent ‘Pinova’, ‘Daligris’ is very precocious and does not tend to alternate bearing. Productivity is comparable to ‘Pinova’.

   <heading id="h-0007" level="1">
    The Fruit
   </heading>

    ‘Daligris’ is very similar to ‘Pinova’ in term of fruit shape and size. Harvest of ‘Daligris’ has been observed to begin approximately one week later than ‘Pinova’. The storageability of ‘Daligris’ is very similar to ‘Pinova’.

    The fruit of ‘Daligris’ is very distinctive from ‘Pinova’ in appearance:

        it is partially covered with a smooth light brown to yellow russeting which fully covers the stem cavity and then spreads out on the cheeks of the fruit;

        the lenticels are much larger and much more prominent than on ‘Pinova’. They are russeted and bumpy; and

        ‘Daligris’ has a typical red-orange over-color, as distinguished from the red-pink over-color of ‘Pinova’.

    ‘Daligris’ also has a consistent distinctive eating quality:

        Quantitative analyses (as presented in the table below) show that ‘Daligris’ consistently has a higher sugar content than ‘Pinova’; and

        ‘Daligris’ has a pronounced aniseed flavor which, combined with the high sugar level, gives a unique taste to the variety.

    <tables id="TABLE-US-00002" num="00002">

      <tgroup align="left" cols="1" colsep="0" rowsep="0">
       <colspec align="center" colname="1" colwidth="259pt"/>

        <row>
         <entry nameend="1" namest="1" rowsep="1">
          TABLE 2
         </entry>
        </row>

        <row>
         <entry align="center" nameend="1" namest="1" rowsep="1"/>
        </row>
        <row>
         <entry>
          Comparison of ‘Daligris’ to ‘Pinova’ (measurements given are approximate)
         </entry>
        </row>
        <row>
         <entry>
          (Average of 20 fruits observed in comparative trials between ‘Daligris’ and ‘Pinova’
         </entry>
        </row>
        <row>
         <entry>
          trees on M9 rootstock, similar crop load, same age)
         </entry>
        </row>

      </tgroup>
      <tgroup align="left" cols="5" colsep="0" rowsep="0">
       <colspec align="center" colname="1" colwidth="35pt"/>
       <colspec align="center" colname="2" colwidth="56pt"/>
       <colspec align="center" colname="3" colwidth="56pt"/>
       <colspec align="center" colname="4" colwidth="56pt"/>
       <colspec align="center" colname="5" colwidth="56pt"/>

        <row>
         <entry>
          Year
         </entry>
         <entry>
          2005
         </entry>
         <entry>
          2006
         </entry>
         <entry>
          2007
         </entry>
         <entry>
          2008
         </entry>
        </row>

      </tgroup>
      <tgroup align="left" cols="9" colsep="0" rowsep="0">
       <colspec align="center" colname="1" colwidth="35pt"/>
       <colspec align="center" colname="2" colwidth="28pt"/>
       <colspec align="center" colname="3" colwidth="28pt"/>
       <colspec align="center" colname="4" colwidth="28pt"/>
       <colspec align="center" colname="5" colwidth="28pt"/>
       <colspec align="center" colname="6" colwidth="28pt"/>
       <colspec align="center" colname="7" colwidth="28pt"/>
       <colspec align="center" colname="8" colwidth="28pt"/>
       <colspec align="center" colname="9" colwidth="28pt"/>

        <row>
         <entry>
          Variety
         </entry>
         <entry>
          Pinova
         </entry>
         <entry>
          Daligris
         </entry>
         <entry>
          Pinova
         </entry>
         <entry>
          Daligris
         </entry>
         <entry>
          Pinova
         </entry>
         <entry>
          Daligris
         </entry>
         <entry>
          Pinova
         </entry>
         <entry>
          Daligris
         </entry>
        </row>
        <row>
         <entry align="center" nameend="9" namest="1" rowsep="1"/>
        </row>

      </tgroup>
      <tgroup align="left" cols="9" colsep="0" rowsep="0">
       <colspec align="center" colname="1" colwidth="35pt"/>
       <colspec align="char" char="." colname="2" colwidth="28pt"/>
       <colspec align="char" char="." colname="3" colwidth="28pt"/>
       <colspec align="char" char="." colname="4" colwidth="28pt"/>
       <colspec align="char" char="." colname="5" colwidth="28pt"/>
       <colspec align="char" char="." colname="6" colwidth="28pt"/>
       <colspec align="char" char="." colname="7" colwidth="28pt"/>
       <colspec align="char" char="." colname="8" colwidth="28pt"/>
       <colspec align="char" char="." colname="9" colwidth="28pt"/>

        <row>
         <entry>
          %
         </entry>
         <entry>
          2
         </entry>
         <entry>
          26
         </entry>
         <entry>
          4
         </entry>
         <entry>
          32
         </entry>
         <entry>
          10
         </entry>
         <entry>
          41
         </entry>
         <entry>
          4
         </entry>
         <entry>
          39
         </entry>
        </row>
        <row>
         <entry>
          Russet
         </entry>
        </row>
        <row>
         <entry>
          % Over-
         </entry>
         <entry>
          23
         </entry>
         <entry>
          19
         </entry>
         <entry>
          25
         </entry>
         <entry>
          27
         </entry>
         <entry>
          21
         </entry>
         <entry>
          30
         </entry>
         <entry>
          35
         </entry>
         <entry>
          28
         </entry>
        </row>
        <row>
         <entry>
          color
         </entry>
        </row>
        <row>
         <entry>
          Sugar
         </entry>
         <entry>
          12
         </entry>
         <entry>
          14.5
         </entry>
         <entry>
          13.2
         </entry>
         <entry>
          14.2
         </entry>
         <entry>
          14.3
         </entry>
         <entry>
          17.1
         </entry>
         <entry>
          13.4
         </entry>
         <entry>
          15.5
         </entry>
        </row>
        <row>
         <entry>
          (° brix)
         </entry>
        </row>
        <row>
         <entry>
          Starch 6.4
         </entry>
         <entry>
          7.1
         </entry>
         <entry>
          6.6
         </entry>
         <entry>
          7.0
         </entry>
         <entry>
          8
         </entry>
         <entry>
          7.2
         </entry>
         <entry>
          7.3
         </entry>
         <entry>
          5.9
         </entry>
        </row>
        <row>
         <entry>
          test
         </entry>
        </row>
        <row>
         <entry>
          Harvest
         </entry>
         <entry>
          Sept
         </entry>
         <entry>
          Sept
         </entry>
         <entry>
          Sept
         </entry>
         <entry>
          Sept
         </entry>
         <entry>
          Sept
         </entry>
         <entry>
          Sept
         </entry>
         <entry>
          Sept
         </entry>
         <entry>
          Sept
         </entry>
        </row>
        <row>
         <entry>
          Date
         </entry>
         <entry>
          19

           th

         </entry>
         <entry>
          29

           th

         </entry>
         <entry>
          14

           th

         </entry>
         <entry>
          19

           th

         </entry>
         <entry>
          20

           th

         </entry>
         <entry>
          28

           th

         </entry>
         <entry>
          22

           nd

         </entry>
         <entry>
          29

           th

         </entry>
        </row>
        <row>
         <entry align="center" nameend="9" namest="1" rowsep="1"/>
        </row>

      </tgroup>

    </tables>

  </description-of-drawings>
  <?brief-description-of-drawings description="Brief Description of Drawings" end="tail"?>
  <?DETDESC description="Detailed Description" end="lead"?>
  <heading id="h-0008" level="1">
   BRIEF DESCRIPTION OF THE PHOTOGRAPHS
  </heading>

   <figref idref="DRAWINGS">
    FIG. 1
   </figref>
   shows the fruit of ‘Daligris’ after harvest;

   <figref idref="DRAWINGS">
    FIG. 2
   </figref>
   shows ‘Daligris’ in bloom in an orchard;

   <figref idref="DRAWINGS">
    FIG. 3
   </figref>
   shows ‘Daligris’ fruit on the tree; and

   <figref idref="DRAWINGS">
    FIG. 4
   </figref>
   shows ‘Daligris’ fruit as compared to ‘Pinova’ fruit.

  <heading id="h-0009" level="1">
   DETAILED BOTANICAL DESCRIPTION OF THE VARIETY
  </heading>

   The following detailed botanical description is based on general observations made of the mother tree from 1998 to 2003 and on second and third generation trees from 2002 to 2008. Quantified measurements of the trees, flowers and leaves are based on observations of three year old trees grown on M9 rootstock in a high density planting in the temperate climate of the Loire Valley, France. All colors are described according to The Royal Horticultural Society Colour Chart. It should be understood that the characteristics described will vary somewhat depending upon cultural practices and climatic conditions, and will vary with location and season. The measurements of any individual plant, or any group of plants, of the new variety may vary from the stated average.

     Tree:

          Vigor.—

         Weak to medium (less than ‘Pinova’).

          Type.—

         Spur to ramified (a little less ramified than ‘Pinova’).

          Habit.—

         Upright.

          Height.—

         Low to medium (191 cm).

          Diameter.—

         Medium (101 cm).

          Trunk diameter

         (

          at

         30

          cm above the graft union

         ).—Medium (35 mm).

          Bark texture.—

         Smooth to medium.

          Bark coloration.—

         Grey 201C.

     Branch (fruiting branches located at around 1 m above the graft union):

          Length.—

         Short to medium (avg. 41 cm).

          Diameter.—

         Medium (avg. 19 mm).

          Crotch angle.—

         60° to 80°.

          Bark color.—

         Brown 200B.

     One year old shoot:

          Length.—

         Medium (avg. 16.3 cm).

          Color.—

         Brown 200B.

          Pubescence

         (

          on upper half of the shoot

         ).—Medium.

          Thickness.—

         Medium thick (avg. 6.4 mm).

          Internode length.—

         Short (avg. 10 mm).

          Number of lenticels.—

         Medium (6 lenticels per cm

          2

         ).

     Winter hardiness:

          Hardiness.—

         Similar to ‘Pinova’.

          Chilling requirement.—

         Similar to ‘Pinova’.

     Flower buds:

          Quantity per spur.—

         2 to 3.

          Shape.—

         Conical.

          Length.—

         Medium (avg. 12 mm).

          Diameter.—

         Medium (avg. 8 mm).

          Color.—

         Red purple 61A.

     Flowers:

          Color

         (

          balloon stage

         ).—Purple 77C.

          Diameter of open flower.—

         Large (avg. 3.6 cm).

          Relative position of petal margin.—

         Not touching.

          Number per cluster.—

         5 to 7.

     Bloom date:

          Date of first bloom.—

         April 15 in Angers, France (comparable to ‘Pinova’).

          Date of full bloom.—

         April 20 in Angers, France (comparable to ‘Pinova’).

     Petals:

          Number per flower.—

         5.

          Shape.—

         Oval.

          Length.—

         Medium long (avg. 20 mm).

          Width.—

         Medium wide (avg. 15 mm).

          Apex.—

         Round.

          Base.—

         Pointed.

          Margin.—

         Smooth.

          Coloration of upper surface.—

         White 155B with purple 76B included herein.

          Coloration of lower surface.—

         Purple 77D.

     Pistils:

          Size.—

         Long (13 mm).

          Color.—

         Yellow green 145D.

     Anthers:

          Quantity.—

         Numerous (avg. 17 per flower).

          Size.—

         Medium (avg. 2 mm).

          Presence of pollen.—

         Present.

          Color of pollen.—

         Yellow 4C.

     Stigma:

          Size.—

         Small to medium (avg. 0.5 mm).

          Color.—

         Pale yellow 9C.

     Style:

          Size.—

         Medium to long (avg. 9 mm).

          Color.—

         Yellow green N144D.

     Ovary:

          Size.—

         Medium (avg. 2 mm).

          Color.—

         Green 134B.

     Pedicel:

          Length.—

         Medium (avg. 2 cm).

          Diameter.—

         Thin (avg. 1.4 mm).

          Color.—

         Green 142C.

     Sepals:

          Quantity.—

         5.

          Coloration.—

         Green 143C.

          Shape.—

         Conical pointed.

     Leaves:

          Shape.—

         Oblanceolate.

          Length

         (

          petiole not included

         ).—Long (avg. 11.1 cm).

          Width.—

         Medium-wide (avg. 5.9 cm).

          Length/width ratio.—

         Medium (2).

          Blade margin.—

         Crenate.

          Apex.—

         Acuminate.

          Base shape.—

         Oblique.

          Color of upper surface.—

         Green 143A (in early summer).

          Color of lower surface.—

         Green 143C (in early summer).

          Attitude in relation to shoot.—

         Upward.

     Petiole:

          Length.—

         Long (avg. 3.5 cm).

          Diameter.—

         Thick (avg. 1.6 mm).

          Coloration.—

         Yellow green 145B.

     Fruit:

          Quantity per cluster.—

         3 to 4.

          Size.—

         Medium large (avg. 71 mm).

          Weight.—

         Medium (avg. 166 g).

          Ratio of height to width.—

         Small to medium (0.9).

          General shape in profile.—

         Conical to truncate conical.

          Position of maximum diameter.—

         Top third of the fruit.

          Ribbing.—

         Absent.

          Aperture of eye.—

         Closed.

          Size of eye.—

         Large (avg. 9.5 mm).

          Length of sepal.—

         Long (avg. 7.7 mm).

          Depth of eye basin.—

         Deep (avg. 11.5 mm).

          Width of eye basin.—

         Medium (avg. 29 mm).

          Length of stalk.—

         Medium long (avg. 29 cm).

          Thickness of stalk.—

         Medium (avg. 2.1 mm).

          Width of stalk cavity.—

         Medium (avg. 33 mm).

          Depth of stalk cavity.—

         Flat (avg. 7 mm).

          Size of lenticels.—

         Very large (avg. 1.2 mm).

          Number of lenticels.—

         Medium.

          Bloom of skin.—

         Absent or very weak.

          Greasiness of skin.—

         Weak to moderate.

          Background color of skin.—

         Yellow 13C.

          Amount of over color.—

         Low (10% to 40%).

          Over color of skin.—

         Red stripes 41A on red orange 42A.

          Intensity of over color.—

         Light to medium.

          Amount of russet around stalk cavity.—

         Very high.

          Amount of russet on cheeks.—

         Medium to high.

          Area of russet around eye basin.—

         Low.

          Relative area of fruit covered with russeting.—

         Medium to high (20% to 40%).

          Flesh texture.—

         Firm.

          Aroma.—

         Very intense.

          Flavor.—

         Sweet with an aniseed flavor.

          Juiciness.—

         Medium.

          Brix.—

         Very high (2° brix higher than ‘Pinova’).

          Flesh coloration.—

         Yellow 11D.

          Stem coloration.—

         Grey brown N199D.

     Seeds:

          Quantity per fruit.—

         5 on average.

          Shape.—

         Truncate ovoid.

          Coloration.—

         Brown 200D.

     Harvest:

          Beginning of harvest.—

         Last week of September (1 week after the beginning of ‘Pinova's’ harvest).

          Harvest window.—

         3 weeks.

          Number of picks.—

         Three picks.

          Amount of fruit produced per tree per harvest.—

         60 fruits on third leaf tree.

          Time for eating maturity.—

         Late.

  <?DETDESC description="Detailed Description" end="tail"?>
 </description>
 <us-claim-statement>
  I claim:
 </us-claim-statement>
 <claims id="claims">
  <claim id="CLM-00001" num="00001">
   <claim-text>
    1. A new and distinct apple tree substantially as described and illustrated herein.
   </claim-text>
  </claim>
 </claims>
</us-patent-grant>
<?xml version="1.0" encoding="UTF-8"?>

 Observation
Even though there are multiple xml files as can be confirmed from opening patent.xml in a text editor such as notepad ++ , beautifulsoup is only able to read just the first xml.
This is because an xml file is only supposed to have one root element.

So we will have to split up the xml files into multiple xml files to parse them one by one.

Also we can observe that the patent id could be stored under either
publication-reference tag -> document-id tag -> doc-number tag
or
 application-reference ->document-id tag -> doc-number tag.

We will assume the patent id is under publication reference tag._____no_output_____ Defining functions to split xml into multiple xml and deleting xml later _____no_output_____

# defining a function to split the huge xml file
# into multiple xml file for easier parsing
# this function will split the xml file into multiple xml files and return the number
# of files created. It accepts the input file to split as well as the output file prefix
def splitxml(inputfilename,outputfilename):
    n = 0
    out_fh = None

    with open(inputfilename) as in_fh:
        while True:
            line = in_fh.readline()
            if not line: break

            if line.startswith('<?xml'):
                if out_fh:
                    out_fh.close()
                n += 1
                out_fh = open('{}_{}.xml'.format(outputfilename,n),'w+')

            out_fh.write(line)

        out_fh.close()
    return n;

# defining function to remove split xml files
def rmxmls(inputfileprefix,no):
    for i in range(1,no+1):
        os.remove("{}_{}.xml".format(inputfileprefix,i))
    return;_____no_output_____

Splitting the big xml patent.xml file into multiple xml files_____no_output_____

#splitting given xml into multiple xml named myfile_xx
nosplits = splitxml(ifilename,ofilename)
#lets check the number of xml files generated
nosplits_____no_output_____

## 4. Parsing XML and Extracting all the required information

_____no_output_____Task 1: Extract the hierarchical IPC code

We can see from the above xml that the required information for IPC are stored as follows :

Id -> doc-number tag

section -> section tag

class -> class tag

subclass -> subclass tag

main_group -> main-group tag

subgroup -> subgroup tag
_____no_output_____We are going to store the content of each xml in xmltreearr list for later use

This code will loop through all the xml files and extract the required data from above discussed tags_____no_output_____

# Code to extract the requried information
#pId contains list of patent ids
pId=[]
#pId contains list of patent ids
psection = []
#pclass contains list of class
pclass = []
#psubclass contains list of subclass
psubclass = []
#pmgroup contains list of main_group
pmgroup = []
#psubgroup contains list of subgroup
psubgroup = []
#xmltreearr contains list of all extracted xml
xmltreearr = []
#looping through all xmls to extract Id,section,class,subclass,main_group,subgroup
for i in range(1,nosplits+1):
    xmltreearr.append( BeautifulSoup(open('{}_{}.xml'.format(ofilename,i)),"lxml-xml"))
    for patent in xmltreearr[i-1].find_all('publication-reference'):
        pId.append(patent('doc-number')[0].string)
    for patent in xmltreearr[i-1].find_all('classification-ipcr'):
        psection.append(patent.section.string)
    for patent in xmltreearr[i-1].find_all('classification-ipcr'):
        pclass.append(patent('class')[0].string )
    for patent in xmltreearr[i-1].find_all('classification-ipcr'):
        psubclass.append(patent('subclass')[0].string )
    for patent in xmltreearr[i-1].find_all('classification-ipcr'):
        pmgroup.append(patent('main-group')[0].string )
    for patent in xmltreearr[i-1].find_all('classification-ipcr'):
        psubgroup.append(patent('subgroup')[0].string)_____no_output_____#lets take a sneek peak at the ids to see if we are doing it correctly...
pId[1:10]_____no_output_____

Now that we have the information required for classification file
we can create a Dataframe with all the required info_____no_output_____

#Storing extracted data in a dictionary
d = {'pId':pId,'psection':psection,'pclass':pclass,'psubclass':psubclass,'pmgroup':pmgroup,'psubgroup':psubgroup}

#converting dictionary to dataframe
df = pd.DataFrame(d)

#rearranging order according to the required format
# code inspired from
# https://stackoverflow.com/questions/41968732/set-order-of-columns-in-pandas-dataframe
df = df[['pId','psection','pclass','psubclass','pmgroup','psubgroup']]

#lets take a look at our result
df.head(10)
_____no_output_____#Lets wrap up task 1 by inserting into classification.txt in the required format

np.savetxt('classification.txt',df,fmt="%s: %s, %s, %s, %s, %s")_____no_output_____

 Task 2: Extract the citation network.

So... in this task I have to find all the cited patent ids for each patent and

then extract that information to citations.txt file

 Observation

we can observe from the xml that the citations are under the tag references-cited inside the tag citation

So.. we will loop through the xml files again to gather the citations
We will be storing the patent id and the cited patent ids in a dictionary in the format
{patentId: citedpatentid1,citedpatentid2,citedpatentid3,citedpatentid4...}_____no_output_____

#looping through all xmls to extract citations
citarr = []
#We will be storing the cited patent ids as a dictionary with st
dataDict = {}

# code inspired from
# https://stackoverflow.com/questions/18072759/python-nested-list-comprehension
for i in range(1,nosplits+1):
    dataDict[pId[i-1]] = [[citation('document-id')[0]('doc-number')[0].string for citation in citations('citation')] for citations in xmltreearr[i-1].find_all('references-cited')][0]
_____no_output_____

Lets take a look at the citation data...._____no_output_____

dataDict['07910771']_____no_output_____

Now that we have the required data lets write it in the citations.txt file_____no_output_____

# We will loop through the dictionary and write the data in the required format
with open('citations.txt','w+') as o_fh:
    for key,value in dataDict.iteritems():
         o_fh.write('{}:{}\n'.format(key,','.join(value)))_____no_output_____

 Task 3 : Identify number of patent citation:

We need to figure out how many times a particular patent has been cited in the file. The citation we are referring to is the patentid that has been cited by the patent and can be found inside references-cited inside the tag citation

For this task we will be counting the frequency of all the patents that have been cited. This can be done using Counter module from collection package_____no_output_____

#code inspired from https://pymotw.com/2/collections/counter.html
# Here we will add all citations patent id in allcitations list
allcitations = []
for key,value in dataDict.iteritems():
    allcitations +=value
#citationfreq will provide us the frequency for each citation patent id
citationfreq = Counter(allcitations)
# write the citations into cited.txt
with open('cited.txt','w+') as o_fh:
    for patent1 in set(allcitations):
        o_fh.write('{}:{}\n'.format(patent1,citationfreq[patent1]))

#taking a look at the number of entries in all citations
len(set(allcitations))
_____no_output_____

......_____no_output_____## 5. Preprocessing the abstracts

Task 4:Extract and preprocess abstracts.
In this task we have to prepare the patent abstracts for analysis through a series of step.
First of all lets extract and store the abstract of all the patents in a dictionary. The key will be the patentId and value will be abstract string_____no_output_____

# abstractDict is the dictionary to store all the abstract
abstractDict = {}
for i in range(1,nosplits+1):
    abstractDict[pId[i-1]] = [p.p.string for p in xmltreearr[i-1].find_all('abstract')]

print abstractDict[pId[0]]
   [u'A new apple tree named \u2018Daligris\u2019 is disclosed. The fruit of the new variety is particularly notable for its eating quality and distinctive flavor and appearance. The fruit is very sweet and has a pronounced aniseed flavor, and takes on a distinctive red orange coloration as it ripens on the tree.']

Lemmatization  : It will reduce the number of features so that there will less number of features for analysis for each variation of a word which will improve the performance in task such as topic modelling and classification.

Source :
https://www.analyticsvidhya.com/blog/2015/10/6-practices-enhance-performance-text-classification-model/

https://arxiv.org/pdf/1608.03995.pdf_____no_output_____To lemmatize we will first segment into sentences using punkt sentence detector_____no_output_____

#code adapted from tutorial 3
# loading punkt sentence segmenter
sent_detector = nltk.data.load('tokenizers/punkt/english.pickle')
#We will store the sentences in a dictionary with key as patentId
#and value as list of sentences
sentences ={}
p = 0
for k, v in abstractDict.iteritems():
    sentences[k] = sent_detector.tokenize(v[0].strip())
sentences['07910771']_____no_output_____

Tokenization :  Tokenization will be done here before adding the POS tags

 Adding Part of Speech (POS) tags  :Now that we have the abstract segmented into sentences we shall tokenize and add POS tags to the tokens._____no_output_____

#loading stopwords from nltk corpus
stopwords_list = stopwords.words('english')
# tagged_sents_dict will be used for storing the tagged sentences for each patentId
tagged_sents_dict ={}
u = 0
for k, v in sentences.iteritems():
    tagged_sents = []
    for sent in v:
        uni_sent = nltk.word_tokenize(sent)
        tagged_sent = nltk.tag.pos_tag(uni_sent)
        tagged_sents.append(tagged_sent)
    tagged_sents_dict[k]=tagged_sents
#tagged_sents_dict_____no_output_____

 Using WorkdNetLemmatizer : We will use WordNetLemmatizer which makes use
 the POS tags of each word to decide the lexical base form and accept the
 following wordnet tags

wordnet.ADJ

wordnet.VERB

wordnet.NOUN

wordnet.ADV

The get_wordnet_pos function will return the

wordnet.ADJ

wordnet.VERB

wordnet.NOUN

wordnet.ADV

tags to be used in lemmatize function later_____no_output_____

#
# The code in this cell is adapted from the following website
# http://stackoverflow.com/questions/15586721/wordnet-lemmatization-and-pos-tagging-in-python
#
def get_wordnet_pos(treebank_tag):

    if treebank_tag.startswith('J'):
        return wordnet.ADJ
    elif treebank_tag.startswith('V'):
        return wordnet.VERB
    elif treebank_tag.startswith('N'):
        return wordnet.NOUN
    elif treebank_tag.startswith('R'):
        return wordnet.ADV
    else:
        return wordnet.NOUN_____no_output_____

 lemmatization code _____no_output_____

lemmatizer = WordNetLemmatizer()
# final_tokens_dict will store the lemmatized tokens  for every patentId in a dictionary
final_tokens_dict ={}
# we will loop through the dictionary and lemmatize each tagged word
for k, v in tagged_sents_dict.iteritems():
    final_tokens =[]
    for tagged_set in v:
        final_tokens = final_tokens + [lemmatizer.lemmatize(w[0], get_wordnet_pos(w[1])) for w in tagged_set ]
    final_tokens_dict[k] = final_tokens
final_tokens_dict['07910771']_____no_output_____

 Lower case conversion :

We shall convert the words into lowercase now in order to reduce the variations in the words.

We are converting it to lowercase after lemmatization because the Lemmatizer can use the case in its working._____no_output_____

#lfinal_tokens_dict is dictionary for storing lower case converted tokens
#lfinal_tokens_combined will contain all the tokens for all patents contacenated
# for finding bigrams
lfinal_tokens_dict ={}
lfinal_tokens_combined =[]
for k, v in final_tokens_dict.iteritems():
    lfinal_tokens =[]
    for tok in v:
        lfinal_tokens.append(tok.lower())
    lfinal_tokens_combined += lfinal_tokens
    lfinal_tokens_dict[k] = lfinal_tokens
lfinal_tokens_dict['07910771']_____no_output_____

 Creating meaningful Bigrams  : Now that we have the tokenized and lemmatized abstracts , we can obtain the meaningful bigrams  by using nltk collocation module. We will try to get 150 best bigrams from collocation module using Pointwise mutual information as a measure._____no_output_____

# we wil use BigramCollocationFinder to find collocationss based on pmi measure
bigram_measures = nltk.collocations.BigramAssocMeasures()
finder2 = nltk.collocations.BigramCollocationFinder.from_words(lfinal_tokens_combined)
bigrams1 = finder2.nbest(bigram_measures.pmi, 300)
bigrams1[0:10]_____no_output_____

Some of these bigrams may occur very infrequently so we will remove those which occur less than 5 times as they do not provide much usefulness in comparison of patents and are therefore not considered valueable and stable collocations due to their infrequency. Source : http://what-when-how.com/pattern-recognition-and-machine-intelligence/automatic-retrieval-of-parallel-collocations-natural-language-processing-and-text-and-data-mining/_____no_output_____

#applying filter to remove collocations that occur less than 5 times in the corpus
finder2.apply_freq_filter(5)
bigrams = finder2.nbest(bigram_measures.pmi, 300)
bigrams[0:10]_____no_output_____

There are stopwords , and non alphabetical words still existing in the list of tokens as well as list of bigram. Now we will remove them._____no_output_____Removing Stop words, non alphabetical words and symbols : We shall now remove the above words from bigrams as well as the unigram tokens as they reduce number of words that need to be processed in downstream processes and
 it will also improve analysis of patent citatation network as stop words do not contribute in any way to the analysis and only increase the load on the performance of analysis.
 We will allow those wordphrases with '-' in them by using regular expression but will remove punctuation as they are not required for analysis._____no_output_____

# this dictionary will contain the cleaned token dictionary for each patentId
clean_tagged_sent_dict = {}

a =0
#First we will clean the token dictionary of stopwords and symbols and non alphabetical words
for k, v in lfinal_tokens_dict.iteritems():
    clean_tagged_sent = []
    clean_tagged_sent.append([tokens for tokens in v if tokens not in stopwords_list and ((tokens.isalpha() and not re.search('\W', tokens)) or ('-' in tokens and not re.search('[^a-zA-Z-]', tokens)) )])
    clean_tagged_sent_dict[k] =  clean_tagged_sent[0]

# now we will clean the bigram list of Stop words, non alphabetical words and symbols
bigramlist = [bigrampair for bigrampair in bigrams if (bigrampair[0] not in stopwords_list and ((bigrampair[0].isalpha() and not re.search('\W', bigrampair[0])) or ('-' in bigrampair[0] and not re.search('[^a-zA-Z-]', bigrampair[0])) )) and bigrampair[1] not in stopwords_list and ((bigrampair[1].isalpha() and not re.search('\W', bigrampair[1])) or ('-' in bigrampair[1] and not re.search('[^a-zA-Z-]', bigrampair[1])) )]
bigramlist[0:10]_____no_output_____#lets see how many bigrams are left after cleaning
len(bigramlist)_____no_output_____#lets make sure we still have the original number of 2500 patents in the
#cleaned token dictionary
len(clean_tagged_sent_dict)_____no_output_____

While some of the above bigrams are meaningful to humans, others may be meaningful from the point of view of the abstract corpus, for the algorithms and the task of analysis of the citation network so we will retain them._____no_output_____ Combining unigrams and bigrams  : We will now combine the unigrams and bigram list to produce a unified list of tokens.

For this we will take help of mwe_tokenizer._____no_output_____

# unibi_tokens_dict dictionary wil lcontain the unified token comprising of unigram and
# collocation joined by '_'
unibi_tokens_dict = {}
# multiphrase tokenizer makes this task easy
mwe_tokenizer = MWETokenizer(bigramlist)
for k, v in clean_tagged_sent_dict.iteritems():
    mwe_tokens = mwe_tokenizer.tokenize(v)
    unibi_tokens_dict[k] = mwe_tokens
unibi_tokens_dict['07910771']_____no_output_____

Now we need to convert these tokens into a form that we can use for

FreqDist function for the next step of removing top 20 frequent word or bigrams based on document frequency.

For this we will replace the collocations joined by '_' by tuple of the words comprising the collocation_____no_output_____

# unibi_conv_tokens_dict dictionary will key as patentId and values as list of
# contain unigrams and tuple of bigrams
unibi_conv_tokens_dict = {}

for k, v in unibi_tokens_dict.iteritems():
    converted_list = []
    for x in v:
        if '_' in x:
            converted_list.append(tuple(x.split('_')))
        else:
            converted_list.append(x)
    unibi_conv_tokens_dict[k] = converted_list
#lets take a look at our resulting dictionary
unibi_conv_tokens_dict['07910771']
#[x for x in converted_list if type(x) == tuple]_____no_output_____

Removing 20 most frequent words

We will now remove 20 most frequent words which will not help in providing document context when performing citation analysis task._____no_output_____

# code Adapted from "Exploring Pre-Processed text and Generating Features" from Tutorial 4
# we will loop through the list of all tokens and make a set of each token list
# chain.from_iterable will then extract all items from those list of sets for each patent
# and then create a list from them.
# this list will be useful for calculating frequency of the various tokens
words_2 = list(chain.from_iterable([set(value) for value in unibi_conv_tokens_dict.values()]))
fd_2 = FreqDist(words_2)

#we will store the 20 most common token in most_common
# we willl remove them later
most_common = fd_2.most_common(20)
most_common_____no_output_____

 Removing words that appear only in one abstract _____no_output_____

# this will create a set of all words who appear only 1 time in the previous list
# indicating that they only appear in 1 abstract
# we willl remove them later
lessFreqWords = set([k for k, v in fd_2.iteritems() if v < 2])
list(lessFreqWords)[0:40]_____no_output_____

taking a look at the bigrams that appear only in one abstract _____no_output_____

# will display the tuples of bigrams from lessFreqWords list
[x for x in lessFreqWords if type(x) is tuple]_____no_output_____#since we have 288 elements in bigram as seen before lets see how many elements we need to remove
len([x for x in lessFreqWords if type(x) is tuple])_____no_output_____

So we will still have a little more than 100 bigrams (288 - 181) after removing the ones which only appear in 1 abstract._____no_output_____Removing the above top 20 frequent and tokens that only appear in one abstract.
We will be using dictionary as it will make processing easier to create sparse vector later_____no_output_____

#removing top 20 frequent tokens and tokens only appearing in one abstract from
#bigram list
for y in most_common:
     while y[0] in bigramlist:
            bigramlist.remove(y[0])
for z in lessFreqWords:
    while z in bigramlist:
        bigramlist.remove(z)
#removing top 20 frequent tokens and tokens only appearing in one abstract from
# cleaned dictionary previously obtained
for k,v in clean_tagged_sent_dict.iteritems():
    for y in most_common:
        while y[0] in v:
            v.remove(y[0])
    for z in lessFreqWords:
        while z in v:
            v.remove(z)
#combining the filtered bigram list and dictionary using mwetokenizer
# since we
finalized_dict = {}
mwe_tokenizer = MWETokenizer(bigramlist)
for k, v in clean_tagged_sent_dict.iteritems():
    mwe_tokens = mwe_tokenizer.tokenize(v)
    finalized_dict[k] = mwe_tokens
finalized_dict['07910771']_____no_output_____

just double checking the removal_____no_output_____

# Just checking if the removal has been done properly

#Should display true if any single abstract appearing value is remaining in dictionary of tokens
for vv in finalized_dict.values():
    for xx in lessFreqWords:
        if(xx in vv):
            print True
#Should display most common element if any frequent value is remaining in dictionary of tokens
for vv in finalized_dict.values():
    for xx in most_common:
        if(xx[0] in vv):
            print xx[0]            _____no_output_____

No output indicates removal is successful_____no_output_____

#checking number of remaining meaningful bigrams
len(bigramlist)_____no_output_____

Its above the required number of 100 meaningful bigrams for the assignment question_____no_output_____Now that we have our resultant dictionary we need to create the vocabulary file.

We will concatenate all the unique words in the dictionary to create the vocabulary and then save it to file._____no_output_____

#set of finalized dictionary will be iterated upon to create set of all unique tokens
# from all patent abstract
vocab_words = set(list(chain.from_iterable([set(value) for value in finalized_dict.values()])))
list(vocab_words)[0:50]   _____no_output_____

Now that we have our vocabulary words ready we will, create index for each word by using enumerate function . We will also convert it to dictionary because later it will be easy to access the words using the index. Also write them to vocab.txt file_____no_output_____

# enumerate will provide the required indexes and then
# we will convert it into dictionary with index as key and words as value
# for easily accessing and looping later
vocab_dict = dict(enumerate(vocab_words))
with open('vocab.txt','w+') as o_fh:
    for k,v in vocab_dict.iteritems():
        o_fh.write('{}:{}\n'.format(k,v))
o_fh.close()_____no_output_____

Now we will have to create a sparse vector. To do that we need to access the index easily for each word in vocabulary as well as get its frequency for abstract in each patent.

By reversing the dictionary of vocabulary previously created this can be done easily. Also we will be using counter library to get the frequency of those words ._____no_output_____

# reversing the vocab dictionary so that the keys are the words instead of index

rev_vocab_dict = {v:k for k, v in vocab_dict.iteritems()}
with open('count_vectors.txt','w+') as o_fh:
    for k,v in finalized_dict.iteritems():
        o_fh.write('{},'.format(k))
        wfreq = Counter(v)
        for index in range(len(v)):
            o_fh.write('{}:{}'.format(rev_vocab_dict[v[index]],wfreq[v[index]]))
            if(index<len(v)-1):
                o_fh.write(',')
        o_fh.write('\n')
o_fh.close()
    _____no_output_____

We can delete the split xml files now that we dont need them_____no_output_____

nosplits = rmxmls(ofilename,2500)_____no_output_____# Viewing some elements of the finalized dictionary
j=0
for k,v in finalized_dict.iteritems():
    if j<10:
        print k,':',v
        j +=1
# finalized_dict07910771 : [u'invention', u'relate', u'produce', u'acrylic', u'acid', u'step', u'reaction', u'glycerol', u'presence', u'molecular', u'oxygen', u'reaction', u'preferably', u'carry', u'gaseous', u'phase', u'presence', u'suitable', u'catalyst']
07910479 : [u'manufacture', u'photodiode', u'array', u'semiconductor', u'substrate', u'main', u'opposite', u'semiconductor', u'substrate', u'layer', u'conductivity', u'proximate', u'main', u'layer', u'conductivity', u'proximate', u'main', u'via', u'substrate', u'extend', u'depth', u'relative', u'main', u'via', u'aspect', u'ratio', u'generally', u'simultaneously', u'via', u'isolation', u'trench', u'substrate', u'space', u'apart', u'via', u'extend', u'depth', u'relative', u'main', u'isolation', u'trench', u'aspect', u'ratio', u'different', u'aspect', u'ratio']
07909645 : [u'housing', u'coaxial_cable', u'connector', u'terminate', u'coaxial_cable', u'housing', u'end', u'configure', u'engage', u'insulating', u'cover', u'coaxial_cable', u'mating', u'end', u'configure', u'engage', u'another', u'coaxial', u'connector', u'base', u'extend', u'end', u'mating', u'end', u'base', u'configure', u'engage', u'electrical', u'conductor', u'coaxial_cable', u'housing', u'also', u'electrical', u'contact', u'electrical', u'contact', u'extension', u'extend', u'outwardly', u'electrical', u'contact', u'wherein', u'extension', u'configure', u'engage', u'outer', u'electrical', u'conductor', u'coaxial_cable']
07909644 : [u'card', u'edge', u'connector', u'assembly', u'connector', u'connector', u'printed', u'circuit', u'board', u'pcb', u'connector', u'respectively', u'insulative', u'housing', u'terminal', u'mount', u'therein', u'insulative', u'housing', u'upper', u'wall', u'low', u'wall', u'mating', u'cavity', u'define', u'therebetween', u'memory', u'module', u'therein', u'pcb', u'define', u'solder', u'region', u'top', u'face', u'thereof', u'terminal', u'respectively', u'solder', u'memory', u'module', u'respectively', u'insert', u'mating', u'cavity', u'along', u'different', u'direction', u'avoid', u'interference', u'memory', u'module', u'assemble', u'process']
07909131 : [u'extract', u'section', u'capable', u'extract', u'specific', u'frequency', u'component', u'input', u'signal', u'extract', u'section', u'extract', u'pinion', u'angle', u'correspond', u'signal', u'indicate', u'state', u'steering', u'frequency', u'component', u'correspond', u'vibration', u'increase', u'generation', u'noise', u'speed', u'reduce', u'mechanism', u'extract', u'section', u'output', u'effective', u'value', u'extracted', u'frequency', u'component', u'power', u'spectrum', u'power', u'spectrum', u'output', u'extract', u'section', u'equal', u'predetermined', u'threshold', u'value', u'output', u'motor', u'signal', u'reduce', u'assist', u'force', u'generate', u'actuator']
07909137 : [u'harness', u'support', u'person', u'ground', u'harness', u'may', u'seat', u'support', u'strap', u'attach', u'seat', u'strap', u'may', u'leg', u'contain', u'loop', u'keep', u'person', u'securely', u'harness', u'harness', u'may', u'also', u'strap', u'secure', u'harness', u'loosely', u'support', u'tree', u'person', u'support', u'harness', u'may', u'secure', u'person', u'tighten', u'waist', u'strap', u'around', u'person', u'waist', u'harness', u'may', u'fall', u'restraint', u'strap', u'enable', u'harness', u'fall', u'arrest']
07909135 : [u'silencer', u'reduce', u'noise', u'noise', u'source', u'housing', u'house', u'noise', u'source', u'internal', u'space', u'housing', u'house', u'noise', u'source', u'internal', u'space', u'internal', u'space', u'housing', u'connector', u'housing', u'internal', u'space', u'internal', u'space', u'space', u'inside', u'internal', u'space', u'outside', u'internal', u'space', u'reduce', u'noise', u'wide', u'space']
07909642 : [u'present', u'invention', u'direct', u'outlet', u'center', u'power', u'inlet', u'box', u'electrically', u'load', u'auxiliary', u'power', u'supply', u'power', u'inlet', u'box', u'outlet', u'center', u'design', u'mount', u'interior', u'wall', u'power', u'inlet', u'box', u'mount', u'exterior', u'wall', u'electrical', u'conductor', u'extend', u'wall', u'outlet', u'center', u'power', u'inlet', u'box', u'outlet', u'center', u'socket', u'adapt', u'plug', u'electrical', u'load', u'auxiliary', u'power', u'may', u'feed', u'electrical', u'load', u'primary', u'power', u'outlet', u'center', u'may', u'illuminate', u'auxiliary', u'power', u'available', u'assist', u'user', u'locate', u'outlet', u'center', u'condition', u'may', u'also', u'indicator', u'lamp', u'illuminate', u'auxiliary', u'power', u'electrical', u'load', u'outlet', u'center']
07909139 : [u'present', u'invention', u'powered', u'lift', u'platform', u'platform', u'guide', u'rail', u'section', u'operative', u'contact', u'platform', u'guide', u'rail', u'section', u'guide', u'rail', u'end', u'guide', u'rail', u'section', u'configure', u'removably', u'attach', u'end', u'guide', u'rail', u'section', u'invention', u'also', u'lift', u'mechanism', u'support', u'platform', u'lift', u'guide', u'operative', u'contact', u'lift', u'mechanism', u'attach', u'upper', u'upper', u'guide', u'rail', u'power', u'supply', u'operate', u'lift', u'mechanism', u'preferred', u'embodiment', u'wheel', u'operatively', u'attach', u'power', u'lift', u'platform', u'also', u'present', u'secure', u'powered', u'lift', u'platform', u'support', u'also', u'present', u'extendable']
07909138 : [u'increase', u'safety', u'perform', u'work', u'ladder', u'describe', u'ladder', u'bracket', u'temporarily', u'attachment', u'ladder', u'structure', u'stabilize', u'secure', u'ladder', u'bracket', u'easy', u'install', u'act', u'limit', u'prevent', u'slippage', u'ladder', u'laterally', u'away', u'structure', u'generally', u'promote', u'great', u'margin', u'safety', u'addition', u'bracket', u'temporarily', u'yet', u'securely', u'attach', u'safety', u'rail', u'ladder']

## 6. Summary
_____no_output_____Now that we have completed the pre-processing, we have obtained the vocabulary as well as the documents in count vector format, ready to be used for  citation modelling  related tasks._____no_output_____


# Notebook from ploomber/projects
Path: guides/spec-api-python/README.ipynb

To run this example locally, [install Ploomber](https://ploomber.readthedocs.io/en/latest/get-started/install.html) and execute: `ploomber examples -n guides/spec-api-python`

To start a free, hosted JupyterLab: [![binder-logo](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/ploomber/binder-env/main?urlpath=git-pull%3Frepo%3Dhttps%253A%252F%252Fgithub.com%252Fploomber%252Fprojects%26urlpath%3Dlab%252Ftree%252Fprojects%252Fguides/spec-api-python%252FREADME.ipynb%26branch%3Dmaster)

Found an issue? [Let us know.](https://github.com/ploomber/projects/issues/new?title=guides/spec-api-python%20issue)

Have questions? [Ask us anything on Slack.](https://ploomber.io/community/)

_____no_output_____# Your first Python pipeline

Introductory tutorial to learn the basics of Ploomber.

**Note:** This is intended for a quick and interactive experience. If you want
to learn about Ploomber's core concepts and design rationale, go to the
[the next tutorial](https://ploomber.readthedocs.io/en/latest/get-started/basic-concepts.html).

## Why Ploomber?

**Notebooks are hard to maintain.** Teams often prototype projects in notebooks, but maintaining them is an error-prone process that slows progress down. Ploomber overcomes the challenges of working with `.ipynb` files allowing teams to develop collaborative, production-ready pipelines using JupyterLab or any text editor.

## Introduction

A pipeline (or **DAG**) is simply a group of tasks with a particular execution order, where subsequent (or **downstream** tasks) use previous (or **upstream**) tasks as inputs.

This example pipeline contains three tasks, the first task, `get.py` gets some data, `clean.py` cleans it, and `plot.py` generates a visualization:_____no_output_____

%%bash
ls *.pyclean.py
get.py
plot.py

**Note:** These tasks are Python scripts, but you can use functions, notebooks,
and SQL scripts. An upcoming guide explains how other types of tasks work.

## Integration with Jupyter

Ploomber integrates with Jupyter. If you open the scripts inside the
`jupyter notebook` app, they will render as notebooks. If you're using `jupyter lab`, you need to right click -> open with -> Notebook as shown below:

![lab-open-with-nb](https://ploomber.io/images/doc/lab-open-with-notebook.png)

**Note**: You can use regular `.ipynb` files for your pipeline; however, using plain `.py` files is recommended since they're easier to manage with git.

Along with the `*.py` files, there is a `pipeline.yaml` file where we declare which files we use as tasks:_____no_output_____```yaml
# Content of pipeline.yaml
tasks:
  - source: get.py
    product:
      nb: output/get.ipynb
      data: output/data.csv

  - source: clean.py
    product:
      nb: output/clean.ipynb
      data: output/clean.csv

  - source: plot.py
    product: output/plot.ipynb

```_____no_output_____Let's plot the pipeline:_____no_output_____

%%bash
ploomber plotPlot saved at: pipeline.png
from IPython.display import Image
Image(filename='pipeline.png')_____no_output_____

You can see that our pipeline has a defined execution order: `get` -> `clean` -> `plot`.

Let's now execute the `status` command, which gives us an overview of the pipeline:_____no_output_____

%%bash
ploomber statusname    Last run      Outdated?    Product       Doc (short)    Location
------  ------------  -----------  ------------  -------------  ------------
get     Has not been  Source code  MetaProduct(                 /Users/Edu/d
        run                        {'data': Fil                 ev/projects-
                                   e('output/da                 ploomber/gui
                                   ta.csv'),                    des/spec-
                                   'nb': File('                 api-python/g
                                   output/get.i                 et.py
                                   pynb')})
clean   Has not been  Source code  MetaProduct(                 /Users/Edu/d
        run           & Upstream   {'data': Fil                 ev/projects-
                                   e('output/cl                 ploomber/gui
                                   ean.csv'),                   des/spec-
                                   'nb': File('                 api-python/c
                                   output/clean                 lean.py
                                   .ipynb')})
plot    Has not been  Source code  File('output                 /Users/Edu/d
        run           & Upstream   /plot.ipynb'                 ev/projects-
                                   )                            ploomber/gui
                                                                des/spec-
                                                                api-python/p
                                                                lot.py

We can see a summary of each task: last execution date, if it's outdated (i.e., source code changed since previous execution), product (output files), documentation (if any), and the source code location.

## How is execution order determined?

Ploomber infers the pipeline structure from your code. For example, to
clean the data, we must get it first; hence, we declare the following in `clean.py`:

~~~python
# this tells Ploomber to execute 'get' task before 'clean'
upstream = ['get']
~~~

Once we finish cleaning the data, we must save it somewhere (In Ploomber, an output is known
as a **product**). Products can be files or SQL relations. Our current example
only generates files.

To specify where to save the output of each task, we use the `product`
key. For example, the `get` task definition looks like this:

~~~yaml
- source: get.py
  product:
    nb: output/get.ipynb
    data: output/data.csv
~~~

Scripts automatically generate a copy of themselves in Jupyter
notebook format (`.ipynb`). That's why we see a notebook in the `product`
dictionary (under the `nb` key). Generating a copy on each execution allows us to create standalone reports for each task, no need to write extra code to save our charts! Notebooks as outputs are an essential concept: `get.py` is part of the pipeline's
source code; in contrast, `output/get.ipynb` is an artifact generated by the source code.

If you don't want to generate output notebooks, you can use Python functions
as tasks. Our upcoming tutorial goes deeper into the different types of tasks.

## Building the pipeline

Let's build the pipeline:_____no_output_____

%%bash
# takes a few seconds to finish
mkdir output
ploomber buildname    Ran?      Elapsed (s)    Percentage
------  ------  -------------  ------------
get     True          3.32293       37.0116
clean   True          2.09533       23.3383
plot    True          3.55981       39.6501

This pipeline saves all the output in the `output/` directory; we have a few
data files:_____no_output_____

%%bash
ls output/*.csvoutput/clean.csv
output/data.csv

And a notebook for each script:_____no_output_____

%%bash
ls output/*.ipynboutput/clean.ipynb
output/get.ipynb
output/plot.ipynb

## Updating the pipeline

Quick experimentation is essential to analyze data. Ploomber allows
you to iterate faster and run more experiments.

Say you found a problematic column and need to add few more lines to your `clean.py` script. Since `get.py` does not depend on `clean.py`, we don't have to rerun it. However, if we modify `clean.py` and want to bring our results up-to-date, we must run `clean.py`, and then `plot.py`, in that order. To save you valuable time, Ploomber keeps track of those dependencies and only reruns outdated tasks.

To see how it works, execute the following to modify the `clean.py` script_____no_output_____

from pathlib import Path

path = Path('clean.py')
clean = path.read_text()
path.write_text(clean + '\nprint("hello")')_____no_output_____

Let's now build again:_____no_output_____

%%bash
# takes a few seconds to finish
ploomber buildname    Ran?      Elapsed (s)    Percentage
------  ------  -------------  ------------
clean   True          2.46494       39.4429
plot    True          3.78445       60.5571
get     False         0              0
# restore contents
path.write_text(clean)_____no_output_____

You'll see that `get.py` didn't run because it was not affected by the change!

Incremental builds are a powerful feature: you can open any of the `.py` files in Jupyter, edit them interactively (as if they were notebooks), then call `ploomber build` to quickly get your results up-to-date.

## Where to go from here

This tutorial shows a bit of what Ploomber can do for you. However, there are many other features to discover: task parallelization, parametrization, execution in the cloud, among others.

Want to dig deeper into Ploomber's core concepts? Check out [the basic concepts tutorial](https://ploomber.readthedocs.io/en/latest/get-started/basic-concepts.html).

Want to take a look at some examples? Check out how to [download templates](https://ploomber.readthedocs.io/en/latest/user-guide/templates.html).

Have questions? [Ask us anything on Slack](https://ploomber.io/community/) or [open an issue](https://github.com/ploomber/ploomber/issues/new?title=Question) on GitHub.

Do you like our project? Show your support with a [star on GitHub](https://github.com/ploomber/ploomber)!_____no_output_____