# Notebook from bioshape-lab/ece594n
Path: lectures/03_c_regression.ipynb

###### 03 Geometric Machine Learning for Shape Analysis

## C) Supervised Learning: Regression

$\color{#003660}{\text{Nina Miolane - Assistant Professor}}$ @ BioShape Lab @ UCSB ECE
_____no_output__________no_output_____# This Unit

- **Unit 1 (Geometry - Math!)**: Differential Geometry for Engineers
- **Unit 2 (Shapes)**: Computational Representations of Biomedical Shapes
- **Unit 3 (Machine Learning)**: **Geometric Machine Learning for Shape Analysis**
  - A) Mean and Covariance
  - B) Supervised Learning: Classification
  - **C) Supervised Learning: Regression**
    - Motivation: Regression on rats skulls
    - Tangent Machine Learning: Tangent linear regression
    - Geometric Machine Learning: Geodesic regression
  - D) Unsupervised Learning: Clustering
  - E) Unsupervised Learning: Dimension Reduction
- **Unit 4 (Deep Learning)**: Geometric Deep Learning for Shape Analysis

$\rightarrow$ We explain the machine learning algorithms and statistics used in these real-world scenarios.

_____no_output_____# Overview of Machine Learning (ML)

Machine Learning is divided into two principal categories of algorithms: supervised and unsupervised learning algorithms. Both learn from data.

$\color{#EF5645}{\text{Definition}}$: **Supervised learning** refers to the task of learning a function that maps an input $x$ to an output $y$ based on example input-output pairs. We say that the model is:
- a classification if the outputs are categorical,
- a regression if the outputs are quantitative.

Supervised Learning_____no_output_____# Regression

$\color{#EF5645}{\text{Given data}}$:
- $(X_1, ..., X_n)$:
  - inputs, e.g. ages of the rats, in days.
- $(y_1, ..., y_n)$:
  - outputs, e.g. rat skull shapes.

Inputs: $X_1, ..., X_n \in \mathbb{R}$. Outputs $y_1, ..., y_n \in \mathbb{R}$.

$\rightarrow \color{#EF5645}{\text{Goal}}$: Build $f$ that predicts the rat skull shape corresponding to a new time $X_\text{new}$.
_____no_output_____## Method

1. Set-up
- $\color{#EF5645}{\text{Decide on the model:}}$ Function $f_\theta$, that outputs $\hat y = f_\theta(X)$: prediction for $y$.
  - $X$: input, e.g. time
  - $y$: true output (ground truth), e.g. true rat skull at this time
  - $\hat y = f_\theta(X)$: model output, e.g. model's predicted skull.
- $\color{#EF5645}{\text{Decide on a loss function:}}$ $L(y, \hat y)$ that evaluates if the model is right or wrong.

$\color{#047C91}{\text{Example}}$:
- The model $f_\theta(X) = \theta_0 + \theta_1 X$ has parameters $\theta=(\theta_0, \theta_1)$.
- The loss function $L(y, \hat y) = d^2(y, \hat y)$ with $d$ is a notion of distance on shape spaces.
  - e.g. $L(y, \hat y)= |y - \hat y |^2 = |y - \theta_0 - \theta_1 X |^2$: least-squares.
_____no_output_____2. $\color{#EF5645}{\text{Split dataset into train / test sets:}}$
   - Train $(X_1, y_1), ..., (X_{n_\text{train}}, y_{n_\text{train}})$: build the algorithm
   - Test $(X_{n_\text{train}+1}, y_{n_\text{train}+1}), ..., (X_n, y_n)$: assess its performances.

3. $\color{#EF5645}{\text{Train:}}$ Build the algorithm
   - Also called: "Fitting the model's parameters $\theta$."
   - Find $\theta^*$ such that the model predicts the train set best:
   $$\theta^* = \text{argmin}_\theta \sum_{i=1}^{n_\text{train}} L(y_i, f_\theta(X_i))$$

4. $\color{#EF5645}{\text{Test:}}$ Assess its performances:
   - Loss on test set: $\sum_{i=n_\text{train}+1}^{n} L(y_i, f_{\theta^*}(X_i))$_____no_output_____# Motivation: Regression of Rat Skulls Shapes_____no_output_____## Rat Skulls

We are interested in modeling the evolution of rat skulls shapes, in order to (later) evaluate if new rats have morphological issues at development.

8 landmarks on a rat's skull._____no_output_____$\color{#EF5645}{\text{Question}}$: Given skulls of a rat at different time points, can we predict what is the shape of the skull of that rat at another time?

$\color{#EF5645}{\text{Our answer}}$: Regression on manifold:
- input: X = time in days: is a real number.
- output: y is a skull shape, represented in Kendall shape spaces, with 8 landmarks.
_____no_output_____# Data

- Vilmann’s rat calvaria (skulls excluding the lower jaw) from X-ray images.
  - 8 landmarks in 2D for each shape.
- 18 individuals, out of which we extract the 1st one.
- 8 time points: at ages of 7, 14, 21, 30, 40, 60, 90, and 150 days._____no_output_____

import numpy as np
import matplotlib.pyplot as plt

import os
os.environ["GEOMSTATS_BACKEND"] = "autograd"_____no_output_____import geomstats.datasets.utils as data_utils

data = np.load("03_rat_skulls.npy")
data_first = data[:8]
print(f"The data of the first rat has shape (n_times, n_landmarks, dim)={data_first.shape}.")The data of the first rat has shape (n_times, n_landmarks, dim)=(8, 8, 2).
fig = plt.figure(figsize=(8, 8))
ax = fig.add_subplot(111)
points = data_first.reshape(-1, 2)
plt_plot, = plt.plot(
    points[:, 0], points[:, 1], 'o', markersize=10)
ax.set_title("Rat skull measurements: 8 landmarks at 8 time points.");_____no_output_____

## Regression Method 1: Tangent Machine Learning_____no_output_____### Recall: "Trick": Tangent Space at the Fréchet Mean

The Fréchet mean gives us a way of transforming our non-linear data into vectors!
1. Compute the Fréchet mean $\bar{x}$ of the data points
2. Consider the tangent space $T_\bar{x}M$of the manifold $M$ at $\bar{x}$
3. Compute the Logarithms of the data points at $\bar{x}$

$\rightarrow$ Get a dataset on a vector space, and apply classical machine learning on it.

$\color{#EF5645}{\text{In practice}}$:
- Either use `FrechetMean` and `metric.log` methods.
- Or the function `ToTangentSpace` does both._____no_output_____1. Set-up

- $\color{#EF5645}{\text{Decide on the model:}}$ Function $f_\theta$, that outputs $\hat y = f_\theta(X)$: prediction for $y$.
- $\color{#EF5645}{\text{Decide on a loss function:}}$ $L(y, \hat y)$ that evaluates if the model is right or wrong.

$\color{#047C91}{\text{Example}}$:
- Use `ToTangentSpace` (do it manually here, with `FrechetMean`, `metric.log`)
- Use any regressor from `scikit-learn`, e.g. `LinearRegression`.
  - `LinearRegression` comes with its loss function._____no_output_____

from pre_shape_autodiff import PreShapeSpace, KendallShapeMetric
from geomstats.learning.frechet_mean import FrechetMean

from sklearn.linear_model import LinearRegression

preshape = PreShapeSpace(k_landmarks=8, m_ambient=2)
kendall_metric = KendallShapeMetric(k_landmarks=8, m_ambient=2)

mean = FrechetMean(metric=kendall_metric, point_type="matrix")
lr = LinearRegression()_____no_output_____

2. $\color{#EF5645}{\text{Split dataset into train / test sets:}}$
   - Train $(X_1, y_1), ..., (X_{n_\text{train}}, y_{n_\text{train}})$: build the algorithm
   - Test $(X_{n_\text{train}+1}, y_{n_\text{train}+1}), ..., (X_n, y_n)$: assess its performances._____no_output_____

from sklearn.model_selection import train_test_split

X = np.linspace(0., 1., 8); X = np.atleast_2d(X).T
y = preshape.projection(data_first)
X_train, X_test, y_train, y_test = train_test_split(X, y)

print(f"The dataset has {len(X)} skulls.")
print(f"The train set has {len(X_train)} skulls.")
print(f"The test set has {len(X_test)} skulls.")The dataset has 8 skulls.
The train set has 6 skulls.
The test set has 2 skulls.

3. $\color{#EF5645}{\text{Train:}}$ Build the algorithm
   - Also called: "Fitting the model's parameters $\theta$."
   - Find $\theta^*$ such that the model predicts the train set best:
   $$\theta^* = \text{argmin}_\theta \sum_{i=1}^{n_\text{train}} L(y_i, f_\theta(X_i))$$_____no_output_____

from geomstats.learning.frechet_mean import FrechetMean

mean.fit(y_train)

tangent_y_train = kendall_metric.log(y_train, base_point=mean.estimate_)
tangent_y_test = kendall_metric.log(y_test, base_point=mean.estimate_)_____no_output_____lr.fit(X_train, tangent_y_train.reshape((6, -1)))_____no_output_____

4. $\color{#EF5645}{\text{Test:}}$ Assess its performances:
   - Loss on test set: $\sum_{i=n_\text{train}+1}^{n} L(y_i, f_{\theta^*}(X_i))$_____no_output_____

print(tangent_y_test.shape)
score = lr.score(X_test, tangent_y_test.reshape((2, -1)))
print(f"We get a score of {score:.2f}.")(2, 8, 2)
We get a score of -0.92.

Best possible score is 1, and the score can be negative because it can be arbitrarily bad._____no_output_____Recall our question.

$\color{#EF5645}{\text{Question}}$: Given skulls of a rat through time, can we predict what is the shape of the skull of that rat at another time?

We can use the fitted model to make new predictions._____no_output_____

tangent_skull = lr.predict(np.array([[4.5]]))

fig = plt.figure(figsize=(6, 6)); ax = fig.add_subplot(111)
points = kendall_metric.exp(tangent_skull.reshape(-1, 2), base_point=mean.estimate_)
plt_plot, = plt.plot(points[:, 0], points[:, 1], 'o', markersize=10)
ax.set_title("Rat skull measurements: 8 landmarks at 8 time points.");_____no_output_____

$\color{#EF5645}{\text{Remark}}$: We can use any regressor available in `scikit-learn`: not necessarily the `LinearRegression` regressor.

$\color{#047C91}{\text{Example}}$: Re-run with `RandomForestRegressor`._____no_output_____

from sklearn.ensemble import RandomForestRegressor

rfr = RandomForestRegressor()

rfr.fit(X_train, tangent_y_train.reshape((6, -1)))

score = rfr.score(X_test, tangent_y_test.reshape((2, -1)))

print(f"R^2 score: {score:.2f}.")R^2 score: -1.90.
tangent_skull =  rfr.predict(np.array([[4.5]]))

fig = plt.figure(figsize=(8, 8))
ax = fig.add_subplot(111)
points = kendall_metric.exp(tangent_skull.reshape(-1, 2), base_point=mean.estimate_)
plt_plot, = plt.plot(
    points[:, 0], points[:, 1], 'o', markersize=10)
ax.set_title("Rat skull predicted at time 4.5 days.");_____no_output_____

## Regression Method 2: Geodesic Regression

$\color{#EF5645}{\text{Previous subsection}}$: Regression Method 1:
- "Project" data on the tangent space at the FrechetMean: `ToTangentSpace`.
- Apply any traditional regression algorithm:
  - e.g. `LinearRegression` or `RandomForestRegressor`.

$\color{#EF5645}{\text{This subsection}}$: Regression Method 2:
- Keep data on the manifold_____no_output_____## Method: Translate the Model

1. Set-up
- $\color{#EF5645}{\text{Decide on the model:}}$ Function $f_\theta$, that outputs $\hat y = f_\theta(X)$: prediction for $y$.
- $\color{#EF5645}{\text{Decide on a loss function:}}$ $L(y, \hat y)$ that evaluates if the model is right or wrong.

The linear regression model with $\theta_0, \theta_1 \in \mathbb{R}^m$:

$$f_\theta(X) = \theta_0 + \theta_1 X \in \mathbb{R}^m$$

Becomes: a geodesic regression model with $\theta_0 \in M$ and $\theta_1 \in T_{\theta_0}M$:

$$f_\theta(X) = \text{Exp}_{\theta_0}\left( \theta_1 X \right) \in M,$$
where Exp is the Riemannian exponential._____no_output_____## Method: Translate the Loss Function

1. Set-up
- $\color{#EF5645}{\text{Decide on the model:}}$ Function $f_\theta$, that outputs $\hat y = f_\theta(X)$: prediction for $y$.
- $\color{#EF5645}{\text{Decide on a loss function:}}$ $L(y, \hat y)$ that evaluates if the model is right or wrong.

The loss function using the square Euclidean distance.
$$L(y, \hat y)= |y - \hat y |^2 = |y - \theta_0 - \theta_1 X |^2$$
now uses the square geodesic distance on $M$:
$$L(y, \hat y)= d_M(y, \hat y )^2 = d_M(y, \text{Exp}_{\theta_0}\left( \theta_1 X \right)  )^2.$$
_____no_output_____
_____no_output_____1. Set-up
- $\color{#EF5645}{\text{Decide on the model:}}$ Function $f_\theta$, that outputs $\hat y = f_\theta(X)$: prediction for $y$.
- $\color{#EF5645}{\text{Decide on a loss function:}}$ $L(y, \hat y)$ that evaluates if the model is right or wrong.
  - `GeodesicRegression` comes with its loss function._____no_output_____

from geomstats.learning.geodesic_regression import GeodesicRegression

gr = GeodesicRegression(
    space=preshape, metric=kendall_metric,
    center_X=False, method="riemannian", initialization="warm_start")

# Warm-start
gr.intercept_ = y_train[0]  # theta0
gr.coef_ = kendall_metric.log(y_train[-1], y_train[0]) # theta1_____no_output_____

Looking inside `GeodesicRegression`:
- https://github.com/geomstats/geomstats/blob/master/geomstats/learning/geodesic_regression.py_____no_output_____3. $\color{#EF5645}{\text{Train:}}$ Build the algorithm
   - Also called: "Fitting the model's parameters $\theta$."
_____no_output_____

gr.fit(np.squeeze(X_train), y_train)_____no_output_____

Looking inside `GeodesicRegression`:
- https://github.com/geomstats/geomstats/blob/master/geomstats/learning/geodesic_regression.py_____no_output_____4. $\color{#EF5645}{\text{Test:}}$ Assess its performances:
   - Loss on test set: $\sum_{i=n_\text{train}+1}^{n} L(y_i, f_{\theta^*}(X_i))$_____no_output_____

print('R^2:', gr.score(np.squeeze(X_test), y_test))R^2: 0.7472292798226752

In this case, computing with the data on the manifold directly gives much better results._____no_output_____Looking inside `GeodesicRegression`:
- https://github.com/geomstats/geomstats/blob/master/geomstats/learning/geodesic_regression.py_____no_output_____Recall our question.

$\color{#EF5645}{\text{Question}}$: Given skulls of a rat through time, can we predict what is the shape of the skull of that rat at another time?

We can use the fitted model to make new predictions._____no_output_____

skull =  gr.predict(np.array([4.5]))
fig = plt.figure(figsize=(6, 6)); ax = fig.add_subplot(111)
points = skull.reshape(-1, 2); plt_plot, = plt.plot(points[:, 0], points[:, 1], 'o', markersize=10)
ax.set_title("Rat skull predicted at time 4.5 days.");_____no_output_____

# This Unit

- **Unit 1 (Geometry - Math!)**: Differential Geometry for Engineers
- **Unit 2 (Shapes)**: Computational Representations of Biomedical Shapes
- **Unit 3 (Machine Learning)**: **Geometric Machine Learning for Shape Analysis**
  - A) Mean and Covariance
  - B) Supervised Learning: Classification
  - **C) Supervised Learning: Regression**
    - Motivation: Regression on aging rats skulls
    - Tangent Machine Learning: Tangent linear regression
    - Geometric Machine Learning: Geodesic regression
  - D) Unsupervised Learning: Clustering
  - E) Unsupervised Learning: Dimension Reduction
- **Unit 4 (Deep Learning)**: Geometric Deep Learning for Shape Analysis

$\rightarrow$ We explain the machine learning algorithms and statistics used in these real-world scenarios.

_____no_output_____


# Notebook from sethbam9/tutorials
Path: quran/start.ipynb

# Tutorial

This notebook gets you started with using
[Text-Fabric](https://annotation.github.io/text-fabric/) for coding in the Quran.

Familiarity with the underlying
[data model](https://annotation.github.io/text-fabric/tf/about/datamodel.html)
is recommended._____no_output_____## Installing Text-Fabric

### Python

You need to have Python on your system. Most systems have it out of the box,
but alas, that is python2 and we need at least python **3.6**.

Install it from [python.org](https://www.python.org) or from
[Anaconda](https://www.anaconda.com/download).

### TF itself

```
pip3 install text-fabric
```

### Jupyter notebook

You need [Jupyter](http://jupyter.org).

If it is not already installed:

```
pip3 install jupyter
```_____no_output_____## Tip
If you start computing with this tutorial, first copy its parent directory to somewhere else,
outside your `syrnt` directory.
If you pull changes from the `syrnt` repository later, your work will not be overwritten.
Where you put your tutorial directory is up till you.
It will work from any directory._____no_output_____

%load_ext autoreload
%autoreload 2_____no_output_____import os
import collections_____no_output_____from tf.app import use_____no_output_____

## Quran data

Text-Fabric will fetch a standard set of features for you from the newest github release binaries.

The data will be stored in the `text-fabric-data` in your home directory._____no_output_____# Load Features
The data of the corpus is organized in features.
They are *columns* of data.
Think of the text as a gigantic spreadsheet, where row 1 corresponds to the
first word, row 2 to the second word, and so on, for all 100,000+ words.

The letters of each word is a column `form` in that spreadsheet.

The corpus contains ca. 30 columns, not only for the words, but also for
textual objects, such as *suras*, *ayas*, and *word groups*.

Instead of putting that information in one big table, the data is organized in separate columns.
We call those columns **features**._____no_output_____For the very last version, use `hot`.

For the latest release, use `latest`.

If you have cloned the repos (TF app and data), use `clone`.

If you do not want/need to upgrade, leave out the checkout specifiers._____no_output_____

A = use("quran:clone", checkout="clone", hoist=globals())
# A = use('quran:hot', checkout="hot", hoist=globals())
# A = use('quran:latest', checkout="latest", hoist=globals())
# A = use('quran', hoist=globals())_____no_output_____

## API

At this point it is helpful to throw a quick glance at the text-fabric API documentation
(see the links under **API Members** above).

The most essential thing for now is that we can use `F` to access the data in the features
we've loaded.
But there is more, such as `N`, which helps us to walk over the text, as we see in a minute._____no_output_____# Counting

In order to get acquainted with the data, we start with the simple task of counting.

## Count all nodes
We use the
[`N.walk()` generator](https://annotation.github.io/text-fabric/tf/core/nodes.html#tf.core.nodes.Nodes.walk)
to walk through the nodes.

We compared corpus to a gigantic spreadsheet, where the rows correspond to the words.
In Text-Fabric, we call the rows `slots`, because they are the textual positions that can be filled with words.

We also mentioned that there are also more textual objects.
They are the verses, chapters and books.
They also correspond to rows in the big spreadsheet.

In Text-Fabric we call all these rows *nodes*, and the `N()` generator
carries us through those nodes in the textual order.

Just one extra thing: the `info` statements generate timed messages.
If you use them instead of `print` you'll get a sense of the amount of time that
the various processing steps typically need._____no_output_____

A.indent(reset=True)
A.info("Counting nodes ...")

i = 0
for n in N.walk():
    i += 1

A.info("{} nodes".format(i))  0.00s Counting nodes ...
  0.04s 218282 nodes

## What are those nodes?
Every node has a type, like word, or aya, or sura.
We know that we have approximately 100,000 words and a few other nodes.
But what exactly are they?

Text-Fabric has two special features, `otype` and `oslots`, that must occur in every Text-Fabric data set.
`otype` tells you for each node its type, and you can ask for the number of `slot`s in the text.

Here we go!_____no_output_____

F.otype.slotType_____no_output_____F.otype.maxSlot_____no_output_____F.otype.maxNode_____no_output_____F.otype.all_____no_output_____C.levels.data_____no_output_____

This is interesting: above you see all the textual objects, with the average size of their objects,
the node where they start, and the node where they end._____no_output_____## Count individual object types
This is an intuitive way to count the number of nodes in each type.
Note in passing, how we use the `indent` in conjunction with `info` to produce neat timed
and indented progress messages._____no_output_____

A.indent(reset=True)
A.info("counting objects ...")

for otype in F.otype.all:
    i = 0
    A.indent(level=1, reset=True)

    for n in F.otype.s(otype):
        i += 1

    A.info("{:>7} {}s".format(i, otype))

A.indent(level=0)
A.info("Done")  0.00s counting objects ...
   |     0.00s       7 manzils
   |     0.00s      15 sajdas
   |     0.00s      30 juzs
   |     0.00s     114 suras
   |     0.00s     240 hizbs
   |     0.00s     556 rukus
   |     0.00s     604 pages
   |     0.00s    6236 ayas
   |     0.00s    4832 lexs
   |     0.01s   77429 groups
   |     0.01s  128219 words
  0.03s Done

# Viewing textual objects

We use the A API (the extra power) to peek into the corpus._____no_output_____Let's inspect some words._____no_output_____

wordShow = (1000, 10000, 100000)
for word in wordShow:
    A.pretty(word)_____no_output_____

# Feature statistics

`F`
gives access to all features.
Every feature has a method
`freqList()`
to generate a frequency list of its values, higher frequencies first.
Here are the parts of speech:_____no_output_____

F.pos.freqList()_____no_output_____

# Lexeme matters

## Top 10 frequent verbs

If we count the frequency of words, we usually mean the frequency of their
corresponding roots or lexemes.

Let's start with roots._____no_output_____

verbs = collections.Counter()
A.indent(reset=True)
A.info("Collecting data")

for w in F.otype.s("word"):
    if F.pos.v(w) != "verb":
        continue
    verbs[F.root.v(w)] += 1

A.info("Done")
print(
    "".join(
        "{}: {}\n".format(verb, cnt)
        for (verb, cnt) in sorted(verbs.items(), key=lambda x: (-x[1], x[0]))[0:10]
    )
)  0.00s Collecting data
  0.07s Done
qwl: 1620
kwn: 1358
Amn: 558
Aty: 535
Elm: 425
jEl: 340
rAy: 315
kfr: 304
jyA: 278
Eml: 276

Now the same with lexemes.
There are several methods for working with lexemes.

### Method 1: counting words_____no_output_____

verbs = collections.Counter()
A.indent(reset=True)
A.info("Collecting data")

for w in F.otype.s("word"):
    if F.pos.v(w) != "verb":
        continue
    verbs[F.lemma.v(w)] += 1

A.info("Done")
print(
    "".join(
        "{}: {}\n".format(verb, cnt)
        for (verb, cnt) in sorted(verbs.items(), key=lambda x: (-x[1], x[0]))[0:10]
    )
)  0.00s Collecting data
  0.07s Done
qaAla: 1618
kaAna: 1358
'aAmana: 537
Ealima: 382
jaEala: 340
kafara: 289
jaA^'a: 278
Eamila: 276
A^taY: 271
ra'aA: 271

## Lexeme distribution

Let's do a bit more fancy lexeme stuff.

### Hapaxes

A hapax can be found by inspecting lexemes and see to how many word nodes they are linked.
If that is number is one, we have a hapax.

We print 10 hapaxes with their gloss._____no_output_____

A.indent(reset=True)

hapax = []
lexIndex = collections.defaultdict(list)

for n in F.otype.s("word"):
    lexIndex[F.lemma.v(n)].append(n)

hapax = dict((lex, occs) for (lex, occs) in lexIndex.items() if len(occs) == 1)

A.info("{} hapaxes found".format(len(hapax)))

for h in sorted(hapax)[0:10]:
    print(f"\t{h}")  0.07s 1994 hapaxes found
	$aAkilat
	$aAni}
	$aAriko
	$aAwiro
	$aTo_#
	$a`Ti}
	$a`mixa`t
	$a`xiSap
	$afatayon
	$agafa

If we want more info on the hapaxes, we get that by means of its *node*.
The lexIndex dictionary stores the occurrences of a lexeme as a list of nodes.

Let's get the part of speech and the Arabic form of those 10 hapaxes._____no_output_____

for h in sorted(hapax)[0:10]:
    node = hapax[h][0]
    print(f"\t{F.pos.v(node):<12} {F.unicode.v(node)}")	noun         شَاكِلَتِ
	noun         شَانِئَ
	verb         شَارِكْ
	verb         شَاوِرْ
	noun         شَطْـَٔ
	noun         شَٰطِئِ
	adjective    شَٰمِخَٰتٍ
	noun         شَٰخِصَةٌ
	noun         شَفَتَيْنِ
	verb         شَغَفَ

### Small occurrence base

The occurrence base of a lexeme are the suras in which it occurs.
Let's look for lexemes that occur in a single sura.

Oh yes, we have already found the hapaxes, we will skip them here._____no_output_____

A.indent(reset=True)
A.info("Finding single sura lexemes")

lexSuraIndex = {}

for (lex, occs) in lexIndex.items():
    lexSuraIndex[lex] = set(L.u(n, otype="sura")[0] for n in occs)

singleSura = [
    (lex, occs)
    for (lex, occs) in lexIndex.items()
    if len(lexSuraIndex.get(lex, [])) == 1
]
singleSuraWithoutHapax = [(lex, occs) for (lex, occs) in singleSura if len(occs) != 1]

A.info("{} single sura lexemes found".format(len(singleSura)))

for data in (singleSura, singleSuraWithoutHapax):
    print("=====================================")
    for (lex, occs) in sorted(data[0:10]):
        print(
            "{:<15} ({}x) first {:>5} last {:>5}".format(
                lex,
                len(occs),
                "{}:{}".format(*T.sectionFromNode(occs[0])),
                "{}:{}".format(*T.sectionFromNode(occs[-1])),
            )
        )  0.00s Finding single sura lexemes
  0.81s 2228 single sura lexemes found
=====================================
>aZolama        (1x) first  2:20 last  2:20
Ha*ar           (2x) first  2:19 last 2:243
Say~ib          (1x) first  2:19 last  2:19
baEuwDap        (1x) first  2:26 last  2:26
magoDuwb        (1x) first   1:7 last   1:7
nuqad~isu       (1x) first  2:30 last  2:30
rabiHat         (1x) first  2:16 last  2:16
vamarap         (1x) first  2:25 last  2:25
yasofiku        (2x) first  2:30 last  2:84
{sotawoqada     (1x) first  2:17 last  2:17
=====================================
$aTor           (5x) first 2:144 last 2:150
Ha*ar           (2x) first  2:19 last 2:243
Hayov2          (2x) first 2:144 last 2:150
Hur~            (2x) first 2:178 last 2:178
Sibogap         (2x) first 2:138 last 2:138
baqarap         (4x) first  2:67 last  2:71
huwd2           (3x) first 2:111 last 2:140
taTaw~aEa       (2x) first 2:158 last 2:184
yasofiku        (2x) first  2:30 last  2:84
yataEal~amu     (2x) first 2:102 last 2:102

### Confined to suras

As a final exercise with lexemes, lets make a list of all suras, and show their total number of lexemes and
the number of lexemes that occur exclusively in that sura._____no_output_____

A.indent(reset=True)
A.info("Making sura-lexeme index")

allSura = collections.defaultdict(set)
allLex = set()

for s in F.otype.s("sura"):
    for w in L.d(s, "word"):
        ln = F.lemma.v(w)
        allSura[s].add(ln)
        allLex.add(ln)

A.info("Found {} lexemes".format(len(allLex)))  0.00s Making sura-lexeme index
  0.10s Found 4833 lexemes
A.indent(reset=True)
A.info("Finding single sura lexemes")

lexSuraIndex = {}

for (lex, occs) in lexIndex.items():
    lexSuraIndex[lex] = set(L.u(n, otype="sura")[0] for n in occs)

singleSuraLex = collections.defaultdict(set)
for (lex, suras) in lexSuraIndex.items():
    if len(suras) == 1:
        singleSuraLex[list(suras)[0]].add(lex)

singleSura = {sura: len(lexs) for (sura, lexs) in singleSuraLex.items()}

A.info("found {} single sura lexemes".format(sum(singleSura.values())))  0.00s Finding single sura lexemes
  0.77s found 2228 single sura lexemes
print(
    "{:<30} {:>4} {:>4} {:>4} {:>5}\n{}".format(
        "sura name",
        "sura",
        "#all",
        "#own",
        "%own",
        "-" * 51,
    )
)
suraList = []

for s in F.otype.s("sura"):
    suraName = Fs("name@en").v(s)
    sura = T.suraName(s)
    a = len(allSura[s])
    o = singleSura.get(s, 0)
    p = 100 * o / a
    suraList.append((suraName, sura, a, o, p))

for x in sorted(suraList, key=lambda e: (-e[4], -e[2], e[1])):
    print("{:<30} {:>4} {:>4} {:>4} {:>4.1f}%".format(*x))sura name                      sura #all #own  %own
---------------------------------------------------
Abundance                       108    9    4 44.4%
Quraysh                         106   16    5 31.2%
The Dawn                        113   17    5 29.4%
The Chargers                    100   32    9 28.1%
Sincerity                       112    9    2 22.2%
The Traducer                    104   28    6 21.4%
The Palm Fibre                  111   21    4 19.0%
The Overwhelming                 88   69   13 18.8%
The Beneficent                   55  142   26 18.3%
The Overthrowing                 81   77   14 18.2%
The Morning Star                 86   44    8 18.2%
The Elephant                    105   22    4 18.2%
The Sun                          91   45    8 17.8%
Defrauding                       83   96   17 17.7%
The Inevitable                   56  206   36 17.5%
The City                         90   63   11 17.5%
The Calamity                    101   24    4 16.7%
Those who drag forth             79  127   21 16.5%
He frowned                       80  103   17 16.5%
The Resurrection                 75  104   17 16.3%
The Morning Hours                93   31    5 16.1%
The Dawn                         89   94   15 16.0%
The Emissaries                   77  108   17 15.7%
Mary                             19  360   52 14.4%
The Reality                      69  157   22 14.0%
The Repentance                    9  638   89 13.9%
The Cave                         18  552   71 12.9%
The Star                         53  188   24 12.8%
The Cow                           2 1137  145 12.8%
Joseph                           12  512   65 12.7%
Mankind                         114   16    2 12.5%
The Pen                          68  171   21 12.3%
The Cloaked One                  74  155   19 12.3%
The Moon                         54  188   22 11.7%
The Enshrouded One               73  129   14 10.9%
The Announcement                 78  122   13 10.7%
The Splitting Open               84   76    8 10.5%
Taa-Haa                          20  483   50 10.4%
The Light                        24  416   43 10.3%
Those drawn up in Ranks          37  360   37 10.3%
Noah                             71  128   13 10.2%
The Table                         5  685   69 10.1%
The letter Saad                  38  338   34 10.1%
Competition                     102   20    2 10.0%
The Night Journey                17  533   53  9.9%
The Clans                        33  454   43  9.5%
The Women                         4  810   75  9.3%
The Pilgrimage                   22  486   44  9.1%
The Fig                          95   34    3  8.8%
Muhammad                         47  239   21  8.8%
The letter Qaaf                  50  207   18  8.7%
The Jinn                         72  139   12  8.6%
The Prophets                     21  423   35  8.3%
The Clot                         96   50    4  8.0%
Sheba                            34  330   26  7.9%
The Cattle                        6  725   57  7.9%
The Family of Imraan              3  761   59  7.8%
The Spoils of War                 8  400   31  7.8%
The Ascending Stairways          70  142   11  7.7%
Man                              76  155   12  7.7%
The Winnowing Winds              51  194   15  7.7%
The Stories                      28  468   36  7.7%
The Mount                        52  182   14  7.7%
The Inner Apartments             49  160   12  7.5%
The Poets                        26  410   30  7.3%
Hud                              11  554   40  7.2%
The Declining Day, Epoch        103   14    1  7.1%
The Bee                          16  552   39  7.1%
The Exile                        59  213   15  7.0%
The Night                        92   57    4  7.0%
The Ant                          27  414   29  7.0%
The Rock                         15  289   20  6.9%
The Heights                       7  819   51  6.2%
The Criterion                    25  372   23  6.2%
The Pleading Woman               58  198   12  6.1%
Ornaments of gold                43  334   20  6.0%
The Victory                      48  253   15  5.9%
The Cleaving                     82   54    3  5.6%
Divorce                          65  148    8  5.4%
Abraham                          14  334   18  5.4%
The Iron                         57  249   13  5.2%
The Thunder                      13  348   18  5.2%
The Believers                    23  392   20  5.1%
The Evidence                     98   59    3  5.1%
The Most High                    87   61    3  4.9%
The Romans                       30  290   14  4.8%
Almsgiving                      107   21    1  4.8%
Yaseen                           36  298   14  4.7%
The Sovereignty                  67  171    8  4.7%
The Consolation                  94   22    1  4.5%
Luqman                           31  248   11  4.4%
The Smoke                        44  183    8  4.4%
The Power, Fate                  97   23    1  4.3%
The Originator                   35  335   14  4.2%
The Prohibition                  66  144    6  4.2%
The Opening                       1   24    1  4.2%
The Constellations               85   77    3  3.9%
Explained in detail              41  311   12  3.9%
The Forgiver                     40  398   15  3.8%
The Earthquake                   99   28    1  3.6%
The Groups                       39  393   14  3.6%
She that is to be examined       60  158    5  3.2%
The Hypocrites                   63  103    3  2.9%
Jonas                            10  486   13  2.7%
Consultation                     42  304    8  2.6%
The Dunes                        46  275    7  2.5%
Crouching                        45  201    5  2.5%
The Ranks                        61  124    3  2.4%
The Spider                       29  336    7  2.1%
The Prostration                  32  193    2  1.0%
Friday                           62  104    1  1.0%
Mutual Disillusion               64  138    1  0.7%
Divine Support                  110   19    0  0.0%
The Disbelievers                109    9    0  0.0%

## For all section types

What we did for suras, we can also do for the other section types.

We generalize the task into a function, that accepts the kind of section as parameter.
Then we can call that function for all our section types._____no_output_____

def lexBase(section):
    # make indices
    lexemesPerSection = {}
    sectionsPerLexeme = {}
    for s in F.otype.s(section):
        for w in L.d(s, otype="word"):
            lex = F.lemma.v(w)
            lexemesPerSection.setdefault(s, set()).add(lex)
            sectionsPerLexeme.setdefault(lex, set()).add(s)

    print(
        "{:<10} {:>4} {:>4} {:>5}\n{}".format(
            section,
            "#all",
            "#own",
            "%own",
            "-" * 26,
        )
    )
    sectionList = []

    for s in F.otype.s(section):
        n = F.number.v(s)
        myLexes = lexemesPerSection[s]
        a = len(myLexes)
        o = len([lex for lex in myLexes if len(sectionsPerLexeme[lex]) == 1])
        p = 100 * o / a
        sectionList.append((n, a, o, p))

    for x in sorted(sectionList, key=lambda e: (-e[3], -e[1], e[0])):
        print("{:<10} {:>4} {:>4} {:>4.1f}%".format(*x))
    print("=" * 26)_____no_output_____for section in (
    "manzil",
    #  'sajda',
    #  'juz',
    #  'ruku',
    #  'hizb',
    #  'page',
):
    lexBase(section)manzil     #all #own  %own
--------------------------
7          2120  685 32.3%
4          1907  415 21.8%
1          1694  302 17.8%
2          1773  316 17.8%
5          1580  235 14.9%
3          1493  222 14.9%
6          1516  215 14.2%
==========================

# Layer API
We travel upwards and downwards, forwards and backwards through the nodes.
The Layer-API (`L`) provides functions: `u()` for going up, and `d()` for going down,
`n()` for going to next nodes and `p()` for going to previous nodes.

These directions are indirect notions: nodes are just numbers, but by means of the
`oslots` feature they are linked to slots. One node *contains* an other node, if the one is linked to a set of slots that contains the set of slots that the other is linked to.
And one if next or previous to an other, if its slots follow of precede the slots of the other one.

`L.u(node)` **Up** is going to nodes that embed `node`.

`L.d(node)` **Down** is the opposite direction, to those that are contained in `node`.

`L.n(node)` **Next** are the next *adjacent* nodes, i.e. nodes whose first slot comes immediately after the last slot of `node`.

`L.p(node)` **Previous** are the previous *adjacent* nodes, i.e. nodes whose last slot comes immediately before the first slot of `node`.

All these functions yield nodes of all possible otypes.
By passing an optional parameter, you can restrict the results to nodes of that type.

The result are ordered according to the order of things in the text.

The functions return always a tuple, even if there is just one node in the result.

## Going up
We go from the first word to the book it contains.
Note the `[0]` at the end. You expect one book, yet `L` returns a tuple.
To get the only element of that tuple, you need to do that `[0]`.

If you are like me, you keep forgetting it, and that will lead to weird error messages later on._____no_output_____

firstSura = L.u(1, otype="sura")[0]
print(firstSura)218169

And let's see all the containing objects of word 3:_____no_output_____

w = 3
for otype in F.otype.all:
    if otype == F.otype.slotType:
        continue
    up = L.u(w, otype=otype)
    upNode = "x" if len(up) == 0 else up[0]
    print("word {} is contained in {} {}".format(w, otype, upNode))word 3 is contained in manzil 216987
word 3 is contained in sajda x
word 3 is contained in juz 212125
word 3 is contained in sura 218169
word 3 is contained in hizb 211885
word 3 is contained in ruku 217598
word 3 is contained in page 216994
word 3 is contained in aya 128220
word 3 is contained in lex 212156
word 3 is contained in group 134457

## Going next
Let's go to the next nodes of the first book._____no_output_____

afterFirstSura = L.n(firstSura)
for n in afterFirstSura:
    print(
        "{:>7}: {:<13} first slot={:<6}, last slot={:<6}".format(
            n,
            F.otype.v(n),
            E.oslots.s(n)[0],
            E.oslots.s(n)[-1],
        )
    )
secondSura = L.n(firstSura, otype="sura")[0]     49: word          first slot=49    , last slot=49
 134485: group         first slot=49    , last slot=49
 128227: aya           first slot=49    , last slot=49
 216995: page          first slot=49    , last slot=112
 217599: ruku          first slot=49    , last slot=149
 218170: sura          first slot=49    , last slot=10291

## Going previous

And let's see what is right before the second book._____no_output_____

for n in L.p(secondSura):
    print(
        "{:>7}: {:<13} first slot={:<6}, last slot={:<6}".format(
            n,
            F.otype.v(n),
            E.oslots.s(n)[0],
            E.oslots.s(n)[-1],
        )
    ) 218169: sura          first slot=1     , last slot=48
 217598: ruku          first slot=1     , last slot=48
 216994: page          first slot=1     , last slot=48
 128226: aya           first slot=34    , last slot=48
 134484: group         first slot=47    , last slot=48
     48: word          first slot=48    , last slot=48

## Going down_____no_output_____We go to the chapters of the second book, and just count them._____no_output_____

ayas = L.d(secondSura, otype="aya")
print(len(ayas))286

## The first aya
We pick the first aya and the first word, and explore what is above and below them._____no_output_____

for n in [1, L.u(1, otype="aya")[0]]:
    A.indent(level=0)
    A.info("Node {}".format(n), tm=False)
    A.indent(level=1)
    A.info("UP", tm=False)
    A.indent(level=2)
    A.info("\n".join(["{:<15} {}".format(u, F.otype.v(u)) for u in L.u(n)]), tm=False)
    A.indent(level=1)
    A.info("DOWN", tm=False)
    A.indent(level=2)
    A.info("\n".join(["{:<15} {}".format(u, F.otype.v(u)) for u in L.d(n)]), tm=False)
A.indent(level=0)
A.info("Done", tm=False)Node 1
   |   UP
   |      |   134456          group
   |      |   128220          aya
   |      |   216994          page
   |      |   217598          ruku
   |      |   218169          sura
   |      |   211885          hizb
   |      |   212125          juz
   |      |   216987          manzil
   |   DOWN
   |      |
Node 128220
   |   UP
   |      |   216994          page
   |      |   217598          ruku
   |      |   218169          sura
   |      |   211885          hizb
   |      |   212125          juz
   |      |   216987          manzil
   |   DOWN
   |      |   134456          group
   |      |   1               word
   |      |   2               word
   |      |   134457          group
   |      |   3               word
   |      |   134458          group
   |      |   4               word
   |      |   5               word
   |      |   134459          group
   |      |   6               word
   |      |   7               word
Done

# Text API

So far, we have mainly seen nodes and their numbers, and the names of node types.
You would almost forget that we are dealing with text.
So let's try to see some text.

In the same way as `F` gives access to feature data,
`T` gives access to the text.
That is also feature data, but you can tell Text-Fabric which features are specifically
carrying the text, and in return Text-Fabric offers you
a Text API: `T`.

## Formats
Arabic text can be represented in a number of ways:

* in transliteration, or in Arabic characters,
* showing the actual text or only the lexemes, or roots.

If you wonder where the information about text formats is stored:
not in the program text-fabric, but in the data set.
It has a feature `otext`, which specifies the formats and which features
must be used to produce them. `otext` is the third special feature in a TF data set,
next to `otype` and `oslots`.
It is an optional feature.
If it is absent, there will be no `T` API.

Here is a list of all available formats in this data set._____no_output_____

sorted(T.formats)_____no_output_____

## Using the formats

We can pretty display in other formats:_____no_output_____

for word in wordShow:
    A.pretty(word, fmt="text-trans-full")_____no_output_____

Now let's use those formats to print out the first aya of the Quran._____no_output_____

a1 = F.otype.s("aya")[0]

for fmt in sorted(T.formats):
    print("{}:\n\t{}".format(fmt, T.text(a1, fmt=fmt, descend=True)))lex-trans-full:
	{som {ll~ah r~aHoma`n r~aHiym
root-trans-full:
	smw Alh rHm rHm
text-orig-full:
	بِسْمِ ٱللَّهِ ٱلرَّحْمَٰنِ ٱلرَّحِيمِ
text-trans-full:
	bisomi {ll~ahi {lr~aHoma`ni {lr~aHiymi

If we do not specify a format, the **default** format is used (`text-orig-full`)._____no_output_____

print(T.text(a1))بِسْمِ ٱللَّهِ ٱلرَّحْمَٰنِ ٱلرَّحِيمِ

## Whole text in all formats in about a second
Part of the pleasure of working with computers is that they can crunch massive amounts of data.
The text of the Quran Bible is a piece of cake.

It takes less than a second to have that cake and eat it.
In nearly a handful formats._____no_output_____

A.indent(reset=True)
A.info("writing plain text of whole Quran in all formats")

text = collections.defaultdict(list)

for a in F.otype.s("aya"):
    words = L.d(a, "word")
    for fmt in sorted(T.formats):
        text[fmt].append(T.text(words, fmt=fmt))

A.info("done {} formats".format(len(text)))

for fmt in sorted(text):
    print("{}\n{}\n".format(fmt, "\n".join(text[fmt][0:5])))  0.00s writing plain text of whole Quran in all formats
  1.04s done 4 formats
lex-trans-full
{som {ll~ah r~aHoma`n r~aHiym
Hamod {ll~ah rab~ Ea`lamiyn
r~aHoma`n r~aHiym
ma`lik yawom diyn
<iy~aA Eabada <iy~aA {sotaEiynu

root-trans-full
smw Alh rHm rHm
Hmd Alh rbb Elm
rHm rHm
mlk ywm dyn
 Ebd  Ewn

text-orig-full
بِسْمِ ٱللَّهِ ٱلرَّحْمَٰنِ ٱلرَّحِيمِ
ٱلْحَمْدُ لِلَّهِ رَبِّ ٱلْعَٰلَمِينَ
ٱلرَّحْمَٰنِ ٱلرَّحِيمِ
مَٰلِكِ يَوْمِ ٱلدِّينِ
إِيَّاكَ نَعْبُدُ وَإِيَّاكَ نَسْتَعِينُ

text-trans-full
bisomi {ll~ahi {lr~aHoma`ni {lr~aHiymi
{loHamodu lil~ahi rab~i {loEa`lamiyna
{lr~aHoma`ni {lr~aHiymi
ma`liki yawomi {ld~iyni
<iy~aAka naEobudu wa<iy~aAka nasotaEiynu

### The full plain text
We write a few formats to file, in your `Downloads` folder._____no_output_____

orig = "text-orig-full"
trans = "text-trans-full"
for fmt in (orig, trans):
    with open(os.path.expanduser(f"~/Downloads/Quran-{fmt}.txt"), "w") as f:
        f.write("\n".join(text[fmt]))_____no_output_____!head -n 20 ~/Downloads/Quran-{orig}.txtبِسْمِ ٱللَّهِ ٱلرَّحْمَٰنِ ٱلرَّحِيمِ
ٱلْحَمْدُ لِلَّهِ رَبِّ ٱلْعَٰلَمِينَ
ٱلرَّحْمَٰنِ ٱلرَّحِيمِ
مَٰلِكِ يَوْمِ ٱلدِّينِ
إِيَّاكَ نَعْبُدُ وَإِيَّاكَ نَسْتَعِينُ
ٱهْدِنَا ٱلصِّرَٰطَ ٱلْمُسْتَقِيمَ
صِرَٰطَ ٱلَّذِينَ أَنْعَمْتَ عَلَيْهِمْ غَيْرِ ٱلْمَغْضُوبِ عَلَيْهِمْ وَلَا ٱلضَّآلِّينَ
الٓمٓ
ذَٰلِكَ ٱلْكِتَٰبُ لَا رَيْبَ فِيهِ هُدًى لِّلْمُتَّقِينَ
ٱلَّذِينَ يُؤْمِنُونَ بِٱلْغَيْبِ وَيُقِيمُونَ ٱلصَّلَوٰةَ وَمِمَّا رَزَقْنَٰهُمْ يُنفِقُونَ
وَٱلَّذِينَ يُؤْمِنُونَ بِمَآ أُنزِلَ إِلَيْكَ وَمَآ أُنزِلَ مِن قَبْلِكَ وَبِٱلْءَاخِرَةِ هُمْ يُوقِنُونَ
أُو۟لَٰٓئِكَ عَلَىٰ هُدًى مِّن رَّبِّهِمْ وَأُو۟لَٰٓئِكَ هُمُ ٱلْمُفْلِحُونَ
إِنَّ ٱلَّذِينَ كَفَرُوا۟ سَوَآءٌ عَلَيْهِمْ ءَأَنذَرْتَهُمْ أَمْ لَمْ تُنذِرْهُمْ لَا يُؤْمِنُونَ
خَتَمَ ٱللَّهُ عَلَىٰ قُلُوبِهِمْ وَعَلَىٰ سَمْعِهِمْ وَعَلَىٰٓ أَبْصَٰرِهِمْ غِشَٰوَةٌ وَلَهُمْ عَذَابٌ عَظِيمٌ
وَمِنَ ٱلنَّاسِ مَن يَقُولُ ءَامَنَّا بِٱللَّهِ وَبِٱلْيَوْمِ ٱلْءَاخِرِ وَمَا هُم بِمُؤْمِنِينَ
يُخَٰدِعُونَ ٱللَّهَ وَٱلَّذِينَ ءَامَنُوا۟ وَمَا يَخْدَعُونَ إِلَّآ أَنفُسَهُمْ وَمَا يَشْعُرُونَ
فِى قُلُوبِهِم مَّرَضٌ فَزَادَهُمُ ٱللَّهُ مَرَضًا وَلَهُمْ عَذَابٌ أَلِيمٌۢ بِمَا كَانُوا۟ يَكْذِبُونَ
وَإِذَا قِيلَ لَهُمْ لَا تُفْسِدُوا۟ فِى ٱلْأَرْضِ قَالُوٓا۟ إِنَّمَا نَحْنُ مُصْلِحُونَ
أَلَآ إِنَّهُمْ هُمُ ٱلْمُفْسِدُونَ وَلَٰكِن لَّا يَشْعُرُونَ
وَإِذَا قِيلَ لَهُمْ ءَامِنُوا۟ كَمَآ ءَامَنَ ٱلنَّاسُ قَالُوٓا۟ أَنُؤْمِنُ كَمَآ ءَامَنَ ٱلسُّفَهَآءُ أَلَآ إِنَّهُمْ هُمُ ٱلسُّفَهَآءُ وَلَٰكِن لَّا يَعْلَمُونَ
!head -n 20 ~/Downloads/Quran-{trans}.txtbisomi {ll~ahi {lr~aHoma`ni {lr~aHiymi
{loHamodu lil~ahi rab~i {loEa`lamiyna
{lr~aHoma`ni {lr~aHiymi
ma`liki yawomi {ld~iyni
<iy~aAka naEobudu wa<iy~aAka nasotaEiynu
{hodinaA {lS~ira`Ta {lomusotaqiyma
Sira`Ta {l~a*iyna >anoEamota Ealayohimo gayori {lomagoDuwbi Ealayohimo walaA {lD~aA^l~iyna
Al^m^
*a`lika {lokita`bu laA rayoba fiyhi hudFY l~ilomut~aqiyna
{l~a*iyna yu&ominuwna bi{logayobi wayuqiymuwna {lS~alaw`pa wamim~aA razaqona`humo yunfiquwna
wa{l~a*iyna yu&ominuwna bimaA^ >unzila <ilayoka wamaA^ >unzila min qabolika wabi{lo'aAxirapi humo yuwqinuwna
>uw@la`^}ika EalaY` hudFY m~in r~ab~ihimo wa>uw@la`^}ika humu {lomufoliHuwna
<in~a {l~a*iyna kafaruwA@ sawaA^'N Ealayohimo 'a>an*arotahumo >amo lamo tun*irohumo laA yu&ominuwna
xatama {ll~ahu EalaY` quluwbihimo waEalaY` samoEihimo waEalaY`^ >aboSa`rihimo gi$a`wapN walahumo Ea*aAbN EaZiymN
wamina {ln~aAsi man yaquwlu 'aAman~aA bi{ll~ahi wabi{loyawomi {lo'aAxiri wamaA hum bimu&ominiyna
yuxa`diEuwna {ll~aha wa{l~a*iyna 'aAmanuwA@ wamaA yaxodaEuwna <il~aA^ >anfusahumo wamaA ya$oEuruwna
fiY quluwbihim m~araDN fazaAdahumu {ll~ahu maraDFA walahumo Ea*aAbN >aliymN[ bimaA kaAnuwA@ yako*ibuwna
waaroDi qaAluw^A@ <in~amaA naHonu muSoliHuwna
>alaA^ <in~ahumo humu {lomufosiduwna wala`kin l~aA ya$oEuruwna
waanu&ominu kamaA^ 'aAmana {ls~ufahaA^'u >alaA^ <in~ahumo humu {ls~ufahaA^'u wala`kin l~aA yaEolamuwna

## Sections

A section is a sura, and an aya.
Knowledge of sections is not baked into Text-Fabric.
The config feature `otext.tf` may specify two or three section levels, and tell
what the corresponding node types and features are.

From that knowledge it can construct mappings from nodes to sections, e.g. from aya
nodes to tuples of the form:

    (sura number, aya number)

Here are examples of getting the section that corresponds to a node and vice versa.

**NB:** `sectionFromNode` always delivers a verse specification, either from the
first slot belonging to that node, or, if `lastSlot`, from the last slot
belonging to that node.
_____no_output_____

for x in (
    ("sura, aya of first word", T.sectionFromNode(1)),
    ("node of 1:1", T.nodeFromSection((1, 1))),
    ("node of 2:1", T.nodeFromSection((2, 1))),
    ("node of sura 1", T.nodeFromSection((1,))),
    ("section of sura node", T.sectionFromNode(211890)),
    ("section of aya node", T.sectionFromNode(210000)),
    ("section of juz node", T.sectionFromNode(216850)),
    ("idem, now last word", T.sectionFromNode(216850, lastSlot=True)),
):
    print("{:<30} {}".format(*x))sura, aya of first word        (1, 1)
node of 1:1                    128220
node of 2:1                    128227
node of sura 1                 218169
section of sura node           (2, 92)
section of aya node            (80, 23)
section of juz node            (85, 2)
idem, now last word            (85, 2)

The other sectional units in the quran, `manzil`, `sajda`, `juz`, `ruku`, `hizb`, `page`
are not associated with special Text-Fabric functions in this data set, although we could have
chosen to use two or three of them instead of sura and aya.

But, TF also offers the possibility to define your own sections, independent from and more flexible than
the sections defined above.

For a bit more on sections, consult the [sections recipe in the cookbook](cookbook/sections.ipynb)._____no_output_____# Translations

This data source contains English (by Arberry) and Dutch (by Leemhuis) translations of the Quran.
They are stored in the features `translation@en` and `translation@nl` for aya nodes.

Let's get the translations of sura 107, together with the arabic original.

The translation features are not loaded by default, we load them first._____no_output_____

TF.load("translation@en translation@nl", add=True)

sura = 107

suraNode = T.suraNode(sura)
print(F.name.v(suraNode))

for ayaNode in L.d(suraNode, otype="aya"):
    print(f"{F.number.v(ayaNode)}")
    print(T.text(ayaNode))
    print(Fs("translation@en").v(ayaNode))
    print(Fs("translation@nl").v(ayaNode))  0.00s loading features ...
  0.01s All additional features loaded - for details use loadLog()
الماعون
1
أَرَءَيْتَ ٱلَّذِى يُكَذِّبُ بِٱلدِّينِ
Hast thou seen him who cries lies to the Doom?
Heb jij hem gezien die de godsdienst loochent?
2
فَذَٰلِكَ ٱلَّذِى يَدُعُّ ٱلْيَتِيمَ
That is he who repulses the orphan
Dat is hij die de wees wegduwt
3
وَلَا يَحُضُّ عَلَىٰ طَعَامِ ٱلْمِسْكِينِ
and urges not the feeding of the needy.
en die er niet op aandringt de behoeftige voedsel te geven.
4
فَوَيْلٌ لِّلْمُصَلِّينَ
So woe to those that pray
En wee hen die de salaat bidden
5
ٱلَّذِينَ هُمْ عَن صَلَاتِهِمْ سَاهُونَ
and are heedless of their prayers,
die hun salaat veronachtzamen,
6
ٱلَّذِينَ هُمْ يُرَآءُونَ
to those who make display
die vertoon willen maken
7
وَيَمْنَعُونَ ٱلْمَاعُونَ
and refuse charity.
en die de hulpverlening weigeren.

# Next steps

* **[display](display.ipynb)** become an expert in creating pretty displays of your text structures
* **[search](search.ipynb)** turbo charge your hand-coding with search templates
* **[exportExcel](exportExcel.ipynb)** make tailor-made spreadsheets out of your results
* **[share](share.ipynb)** draw in other people's data and let them use yours
* **[similarAyas](similarAyas.ipynb)** spot the similarities between lines
* **[rings](rings.ipynb)** ring structures in sura 2

CC-BY Dirk Roorda_____no_output_____


# Notebook from zhujiajunbryan/ProtTrans
Path: finetune.ipynb

_____no_output_____

!pip -q install transformers seqeval SentencePiece biopython
     |                                | 10 kB 31.0 MB/s eta 0:00:01
     |▏                               | 20 kB 39.0 MB/s eta 0:00:01
     |▎                               | 30 kB 27.3 MB/s eta 0:00:01
     |▍                               | 40 kB 20.0 MB/s eta 0:00:01
     |▌                               | 51 kB 19.1 MB/s eta 0:00:01
     |▋                               | 61 kB 15.6 MB/s eta 0:00:01
     |▊                               | 71 kB 14.7 MB/s eta 0:00:01
     |▉                               | 81 kB 16.1 MB/s eta 0:00:01
     |▉                               | 92 kB 13.5 MB/s eta 0:00:01
     |█                               | 102 kB 14.7 MB/s eta 0:00:01
     |█                               | 112 kB 14.7 MB/s eta 0:00:01
     |█▏                              | 122 kB 14.7 MB/s eta 0:00:01
     |█▎                              | 133 kB 14.7 MB/s eta 0:00:01
     |█▍                              | 143 kB 14.7 MB/s eta 0:00:01
     |█▌                              | 153 kB 14.7 MB/s eta 0:00:01
     |█▋                              | 163 kB 14.7 MB/s eta 0:00:01
     |█▋                              | 174 kB 14.7 MB/s eta 0:00:01
     |█▊                              | 184 kB 14.7 MB/s eta 0:00:01
     |█▉                              | 194 kB 14.7 MB/s eta 0:00:01
     |██                              | 204 kB 14.7 MB/s eta 0:00:01
     |██                              | 215 kB 14.7 MB/s eta 0:00:01
     |██▏                             | 225 kB 14.7 MB/s eta 0:00:01
     |██▎                             | 235 kB 14.7 MB/s eta 0:00:01
     |██▍                             | 245 kB 14.7 MB/s eta 0:00:01
     |██▍                             | 256 kB 14.7 MB/s eta 0:00:01
     |██▌                             | 266 kB 14.7 MB/s eta 0:00:01
     |██▋                             | 276 kB 14.7 MB/s eta 0:00:01
     |██▊                             | 286 kB 14.7 MB/s eta 0:00:01
     |██▉                             | 296 kB 14.7 MB/s eta 0:00:01
     |███                             | 307 kB 14.7 MB/s eta 0:00:01
     |███                             | 317 kB 14.7 MB/s eta 0:00:01
     |███▏                            | 327 kB 14.7 MB/s eta 0:00:01
     |███▎                            | 337 kB 14.7 MB/s eta 0:00:01
     |███▎                            | 348 kB 14.7 MB/s eta 0:00:01
     |███▍                            | 358 kB 14.7 MB/s eta 0:00:01
     |███▌                            | 368 kB 14.7 MB/s eta 0:00:01
     |███▋                            | 378 kB 14.7 MB/s eta 0:00:01
     |███▊                            | 389 kB 14.7 MB/s eta 0:00:01
     |███▉                            | 399 kB 14.7 MB/s eta 0:00:01
     |████                            | 409 kB 14.7 MB/s eta 0:00:01
     |████                            | 419 kB 14.7 MB/s eta 0:00:01
     |████                            | 430 kB 14.7 MB/s eta 0:00:01
     |████▏                           | 440 kB 14.7 MB/s eta 0:00:01
     |████▎                           | 450 kB 14.7 MB/s eta 0:00:01
     |████▍                           | 460 kB 14.7 MB/s eta 0:00:01
     |████▌                           | 471 kB 14.7 MB/s eta 0:00:01
     |████▋                           | 481 kB 14.7 MB/s eta 0:00:01
     |████▊                           | 491 kB 14.7 MB/s eta 0:00:01
     |████▉                           | 501 kB 14.7 MB/s eta 0:00:01
     |████▉                           | 512 kB 14.7 MB/s eta 0:00:01
     |█████                           | 522 kB 14.7 MB/s eta 0:00:01
     |█████                           | 532 kB 14.7 MB/s eta 0:00:01
     |█████▏                          | 542 kB 14.7 MB/s eta 0:00:01
     |█████▎                          | 552 kB 14.7 MB/s eta 0:00:01
     |█████▍                          | 563 kB 14.7 MB/s eta 0:00:01
     |█████▌                          | 573 kB 14.7 MB/s eta 0:00:01
     |█████▋                          | 583 kB 14.7 MB/s eta 0:00:01
     |█████▋                          | 593 kB 14.7 MB/s eta 0:00:01
     |█████▊                          | 604 kB 14.7 MB/s eta 0:00:01
     |█████▉                          | 614 kB 14.7 MB/s eta 0:00:01
     |██████                          | 624 kB 14.7 MB/s eta 0:00:01
     |██████                          | 634 kB 14.7 MB/s eta 0:00:01
     |██████▏                         | 645 kB 14.7 MB/s eta 0:00:01
     |██████▎                         | 655 kB 14.7 MB/s eta 0:00:01
     |██████▍                         | 665 kB 14.7 MB/s eta 0:00:01
     |██████▌                         | 675 kB 14.7 MB/s eta 0:00:01
     |██████▌                         | 686 kB 14.7 MB/s eta 0:00:01
     |██████▋                         | 696 kB 14.7 MB/s eta 0:00:01
     |██████▊                         | 706 kB 14.7 MB/s eta 0:00:01
     |██████▉                         | 716 kB 14.7 MB/s eta 0:00:01
     |███████                         | 727 kB 14.7 MB/s eta 0:00:01
     |███████                         | 737 kB 14.7 MB/s eta 0:00:01
     |███████▏                        | 747 kB 14.7 MB/s eta 0:00:01
     |███████▎                        | 757 kB 14.7 MB/s eta 0:00:01
     |███████▎                        | 768 kB 14.7 MB/s eta 0:00:01
     |███████▍                        | 778 kB 14.7 MB/s eta 0:00:01
     |███████▌                        | 788 kB 14.7 MB/s eta 0:00:01
     |███████▋                        | 798 kB 14.7 MB/s eta 0:00:01
     |███████▊                        | 808 kB 14.7 MB/s eta 0:00:01
     |███████▉                        | 819 kB 14.7 MB/s eta 0:00:01
     |████████                        | 829 kB 14.7 MB/s eta 0:00:01
     |████████                        | 839 kB 14.7 MB/s eta 0:00:01
     |████████                        | 849 kB 14.7 MB/s eta 0:00:01
     |████████▏                       | 860 kB 14.7 MB/s eta 0:00:01
     |████████▎                       | 870 kB 14.7 MB/s eta 0:00:01
     |████████▍                       | 880 kB 14.7 MB/s eta 0:00:01
     |████████▌                       | 890 kB 14.7 MB/s eta 0:00:01
     |████████▋                       | 901 kB 14.7 MB/s eta 0:00:01
     |████████▊                       | 911 kB 14.7 MB/s eta 0:00:01
     |████████▉                       | 921 kB 14.7 MB/s eta 0:00:01
     |████████▉                       | 931 kB 14.7 MB/s eta 0:00:01
     |█████████                       | 942 kB 14.7 MB/s eta 0:00:01
     |█████████                       | 952 kB 14.7 MB/s eta 0:00:01
     |█████████▏                      | 962 kB 14.7 MB/s eta 0:00:01
     |█████████▎                      | 972 kB 14.7 MB/s eta 0:00:01
     |█████████▍                      | 983 kB 14.7 MB/s eta 0:00:01
     |█████████▌                      | 993 kB 14.7 MB/s eta 0:00:01
     |█████████▋                      | 1.0 MB 14.7 MB/s eta 0:00:01
     |█████████▊                      | 1.0 MB 14.7 MB/s eta 0:00:01
     |█████████▊                      | 1.0 MB 14.7 MB/s eta 0:00:01
     |█████████▉                      | 1.0 MB 14.7 MB/s eta 0:00:01
     |██████████                      | 1.0 MB 14.7 MB/s eta 0:00:01
     |██████████                      | 1.1 MB 14.7 MB/s eta 0:00:01
     |██████████▏                     | 1.1 MB 14.7 MB/s eta 0:00:01
     |██████████▎                     | 1.1 MB 14.7 MB/s eta 0:00:01
     |██████████▍                     | 1.1 MB 14.7 MB/s eta 0:00:01
     |██████████▌                     | 1.1 MB 14.7 MB/s eta 0:00:01
     |██████████▌                     | 1.1 MB 14.7 MB/s eta 0:00:01
     |██████████▋                     | 1.1 MB 14.7 MB/s eta 0:00:01
     |██████████▊                     | 1.1 MB 14.7 MB/s eta 0:00:01
     |██████████▉                     | 1.1 MB 14.7 MB/s eta 0:00:01
     |███████████                     | 1.1 MB 14.7 MB/s eta 0:00:01
     |███████████                     | 1.2 MB 14.7 MB/s eta 0:00:01
     |███████████▏                    | 1.2 MB 14.7 MB/s eta 0:00:01
     |███████████▎                    | 1.2 MB 14.7 MB/s eta 0:00:01
     |███████████▎                    | 1.2 MB 14.7 MB/s eta 0:00:01
     |███████████▍                    | 1.2 MB 14.7 MB/s eta 0:00:01
     |███████████▌                    | 1.2 MB 14.7 MB/s eta 0:00:01
     |███████████▋                    | 1.2 MB 14.7 MB/s eta 0:00:01
     |███████████▊                    | 1.2 MB 14.7 MB/s eta 0:00:01
     |███████████▉                    | 1.2 MB 14.7 MB/s eta 0:00:01
     |████████████                    | 1.2 MB 14.7 MB/s eta 0:00:01
     |████████████                    | 1.3 MB 14.7 MB/s eta 0:00:01
     |████████████                    | 1.3 MB 14.7 MB/s eta 0:00:01
     |████████████▏                   | 1.3 MB 14.7 MB/s eta 0:00:01
     |████████████▎                   | 1.3 MB 14.7 MB/s eta 0:00:01
     |████████████▍                   | 1.3 MB 14.7 MB/s eta 0:00:01
     |████████████▌                   | 1.3 MB 14.7 MB/s eta 0:00:01
     |████████████▋                   | 1.3 MB 14.7 MB/s eta 0:00:01
     |████████████▊                   | 1.3 MB 14.7 MB/s eta 0:00:01
     |████████████▉                   | 1.3 MB 14.7 MB/s eta 0:00:01
     |█████████████                   | 1.4 MB 14.7 MB/s eta 0:00:01
     |█████████████                   | 1.4 MB 14.7 MB/s eta 0:00:01
     |█████████████                   | 1.4 MB 14.7 MB/s eta 0:00:01
     |█████████████▏                  | 1.4 MB 14.7 MB/s eta 0:00:01
     |█████████████▎                  | 1.4 MB 14.7 MB/s eta 0:00:01
     |█████████████▍                  | 1.4 MB 14.7 MB/s eta 0:00:01
     |█████████████▌                  | 1.4 MB 14.7 MB/s eta 0:00:01
     |█████████████▋                  | 1.4 MB 14.7 MB/s eta 0:00:01
     |█████████████▊                  | 1.4 MB 14.7 MB/s eta 0:00:01
     |█████████████▊                  | 1.4 MB 14.7 MB/s eta 0:00:01
     |█████████████▉                  | 1.5 MB 14.7 MB/s eta 0:00:01
     |██████████████                  | 1.5 MB 14.7 MB/s eta 0:00:01
     |██████████████                  | 1.5 MB 14.7 MB/s eta 0:00:01
     |██████████████▏                 | 1.5 MB 14.7 MB/s eta 0:00:01
     |██████████████▎                 | 1.5 MB 14.7 MB/s eta 0:00:01
     |██████████████▍                 | 1.5 MB 14.7 MB/s eta 0:00:01
     |██████████████▌                 | 1.5 MB 14.7 MB/s eta 0:00:01
     |██████████████▌                 | 1.5 MB 14.7 MB/s eta 0:00:01
     |██████████████▋                 | 1.5 MB 14.7 MB/s eta 0:00:01
     |██████████████▊                 | 1.5 MB 14.7 MB/s eta 0:00:01
     |██████████████▉                 | 1.6 MB 14.7 MB/s eta 0:00:01
     |███████████████                 | 1.6 MB 14.7 MB/s eta 0:00:01
     |███████████████                 | 1.6 MB 14.7 MB/s eta 0:00:01
     |███████████████▏                | 1.6 MB 14.7 MB/s eta 0:00:01
     |███████████████▎                | 1.6 MB 14.7 MB/s eta 0:00:01
     |███████████████▎                | 1.6 MB 14.7 MB/s eta 0:00:01
     |███████████████▍                | 1.6 MB 14.7 MB/s eta 0:00:01
     |███████████████▌                | 1.6 MB 14.7 MB/s eta 0:00:01
     |███████████████▋                | 1.6 MB 14.7 MB/s eta 0:00:01
     |███████████████▊                | 1.6 MB 14.7 MB/s eta 0:00:01
     |███████████████▉                | 1.7 MB 14.7 MB/s eta 0:00:01
     |████████████████                | 1.7 MB 14.7 MB/s eta 0:00:01
     |████████████████                | 1.7 MB 14.7 MB/s eta 0:00:01
     |████████████████▏               | 1.7 MB 14.7 MB/s eta 0:00:01
     |████████████████▏               | 1.7 MB 14.7 MB/s eta 0:00:01
     |████████████████▎               | 1.7 MB 14.7 MB/s eta 0:00:01
     |████████████████▍               | 1.7 MB 14.7 MB/s eta 0:00:01
     |████████████████▌               | 1.7 MB 14.7 MB/s eta 0:00:01
     |████████████████▋               | 1.7 MB 14.7 MB/s eta 0:00:01
     |████████████████▊               | 1.8 MB 14.7 MB/s eta 0:00:01
     |████████████████▉               | 1.8 MB 14.7 MB/s eta 0:00:01
     |█████████████████               | 1.8 MB 14.7 MB/s eta 0:00:01
     |█████████████████               | 1.8 MB 14.7 MB/s eta 0:00:01
     |█████████████████               | 1.8 MB 14.7 MB/s eta 0:00:01
     |█████████████████▏              | 1.8 MB 14.7 MB/s eta 0:00:01
     |█████████████████▎              | 1.8 MB 14.7 MB/s eta 0:00:01
     |█████████████████▍              | 1.8 MB 14.7 MB/s eta 0:00:01
     |█████████████████▌              | 1.8 MB 14.7 MB/s eta 0:00:01
     |█████████████████▋              | 1.8 MB 14.7 MB/s eta 0:00:01
     |█████████████████▊              | 1.9 MB 14.7 MB/s eta 0:00:01
     |█████████████████▊              | 1.9 MB 14.7 MB/s eta 0:00:01
     |█████████████████▉              | 1.9 MB 14.7 MB/s eta 0:00:01
     |██████████████████              | 1.9 MB 14.7 MB/s eta 0:00:01
     |██████████████████              | 1.9 MB 14.7 MB/s eta 0:00:01
     |██████████████████▏             | 1.9 MB 14.7 MB/s eta 0:00:01
     |██████████████████▎             | 1.9 MB 14.7 MB/s eta 0:00:01
     |██████████████████▍             | 1.9 MB 14.7 MB/s eta 0:00:01
     |██████████████████▌             | 1.9 MB 14.7 MB/s eta 0:00:01
     |██████████████████▌             | 1.9 MB 14.7 MB/s eta 0:00:01
     |██████████████████▋             | 2.0 MB 14.7 MB/s eta 0:00:01
     |██████████████████▊             | 2.0 MB 14.7 MB/s eta 0:00:01
     |██████████████████▉             | 2.0 MB 14.7 MB/s eta 0:00:01
     |███████████████████             | 2.0 MB 14.7 MB/s eta 0:00:01
     |███████████████████             | 2.0 MB 14.7 MB/s eta 0:00:01
     |███████████████████▏            | 2.0 MB 14.7 MB/s eta 0:00:01
     |███████████████████▎            | 2.0 MB 14.7 MB/s eta 0:00:01
     |███████████████████▍            | 2.0 MB 14.7 MB/s eta 0:00:01
     |███████████████████▍            | 2.0 MB 14.7 MB/s eta 0:00:01
     |███████████████████▌            | 2.0 MB 14.7 MB/s eta 0:00:01
     |███████████████████▋            | 2.1 MB 14.7 MB/s eta 0:00:01
     |███████████████████▊            | 2.1 MB 14.7 MB/s eta 0:00:01
     |███████████████████▉            | 2.1 MB 14.7 MB/s eta 0:00:01
     |████████████████████            | 2.1 MB 14.7 MB/s eta 0:00:01
     |████████████████████            | 2.1 MB 14.7 MB/s eta 0:00:01
     |████████████████████▏           | 2.1 MB 14.7 MB/s eta 0:00:01
     |████████████████████▏           | 2.1 MB 14.7 MB/s eta 0:00:01
     |████████████████████▎           | 2.1 MB 14.7 MB/s eta 0:00:01
     |████████████████████▍           | 2.1 MB 14.7 MB/s eta 0:00:01
     |████████████████████▌           | 2.2 MB 14.7 MB/s eta 0:00:01
     |████████████████████▋           | 2.2 MB 14.7 MB/s eta 0:00:01
     |████████████████████▊           | 2.2 MB 14.7 MB/s eta 0:00:01
     |████████████████████▉           | 2.2 MB 14.7 MB/s eta 0:00:01
     |█████████████████████           | 2.2 MB 14.7 MB/s eta 0:00:01
     |█████████████████████           | 2.2 MB 14.7 MB/s eta 0:00:01
     |█████████████████████           | 2.2 MB 14.7 MB/s eta 0:00:01
     |█████████████████████▏          | 2.2 MB 14.7 MB/s eta 0:00:01
     |█████████████████████▎          | 2.2 MB 14.7 MB/s eta 0:00:01
     |█████████████████████▍          | 2.2 MB 14.7 MB/s eta 0:00:01
     |█████████████████████▌          | 2.3 MB 14.7 MB/s eta 0:00:01
     |█████████████████████▋          | 2.3 MB 14.7 MB/s eta 0:00:01
     |█████████████████████▊          | 2.3 MB 14.7 MB/s eta 0:00:01
     |█████████████████████▊          | 2.3 MB 14.7 MB/s eta 0:00:01
     |█████████████████████▉          | 2.3 MB 14.7 MB/s eta 0:00:01
     |██████████████████████          | 2.3 MB 14.7 MB/s eta 0:00:01
     |██████████████████████          | 2.3 MB 14.7 MB/s eta 0:00:01
     |██████████████████████▏         | 2.3 MB 14.7 MB/s eta 0:00:01
     |██████████████████████▎         | 2.3 MB 14.7 MB/s eta 0:00:01
     |██████████████████████▍         | 2.3 MB 14.7 MB/s eta 0:00:01
     |██████████████████████▌         | 2.4 MB 14.7 MB/s eta 0:00:01
     |██████████████████████▋         | 2.4 MB 14.7 MB/s eta 0:00:01
     |██████████████████████▋         | 2.4 MB 14.7 MB/s eta 0:00:01
     |██████████████████████▊         | 2.4 MB 14.7 MB/s eta 0:00:01
     |██████████████████████▉         | 2.4 MB 14.7 MB/s eta 0:00:01
     |███████████████████████         | 2.4 MB 14.7 MB/s eta 0:00:01
     |███████████████████████         | 2.4 MB 14.7 MB/s eta 0:00:01
     |███████████████████████▏        | 2.4 MB 14.7 MB/s eta 0:00:01
     |███████████████████████▎        | 2.4 MB 14.7 MB/s eta 0:00:01
     |███████████████████████▍        | 2.4 MB 14.7 MB/s eta 0:00:01
     |███████████████████████▍        | 2.5 MB 14.7 MB/s eta 0:00:01
     |███████████████████████▌        | 2.5 MB 14.7 MB/s eta 0:00:01
     |███████████████████████▋        | 2.5 MB 14.7 MB/s eta 0:00:01
     |███████████████████████▊        | 2.5 MB 14.7 MB/s eta 0:00:01
     |███████████████████████▉        | 2.5 MB 14.7 MB/s eta 0:00:01
     |████████████████████████        | 2.5 MB 14.7 MB/s eta 0:00:01
     |████████████████████████        | 2.5 MB 14.7 MB/s eta 0:00:01
     |████████████████████████▏       | 2.5 MB 14.7 MB/s eta 0:00:01
     |████████████████████████▏       | 2.5 MB 14.7 MB/s eta 0:00:01
     |████████████████████████▎       | 2.5 MB 14.7 MB/s eta 0:00:01
     |████████████████████████▍       | 2.6 MB 14.7 MB/s eta 0:00:01
     |████████████████████████▌       | 2.6 MB 14.7 MB/s eta 0:00:01
     |████████████████████████▋       | 2.6 MB 14.7 MB/s eta 0:00:01
     |████████████████████████▊       | 2.6 MB 14.7 MB/s eta 0:00:01
     |████████████████████████▉       | 2.6 MB 14.7 MB/s eta 0:00:01
     |█████████████████████████       | 2.6 MB 14.7 MB/s eta 0:00:01
     |█████████████████████████       | 2.6 MB 14.7 MB/s eta 0:00:01
     |█████████████████████████       | 2.6 MB 14.7 MB/s eta 0:00:01
     |█████████████████████████▏      | 2.6 MB 14.7 MB/s eta 0:00:01
     |█████████████████████████▎      | 2.7 MB 14.7 MB/s eta 0:00:01
     |█████████████████████████▍      | 2.7 MB 14.7 MB/s eta 0:00:01
     |█████████████████████████▌      | 2.7 MB 14.7 MB/s eta 0:00:01
     |█████████████████████████▋      | 2.7 MB 14.7 MB/s eta 0:00:01
     |█████████████████████████▊      | 2.7 MB 14.7 MB/s eta 0:00:01
     |█████████████████████████▉      | 2.7 MB 14.7 MB/s eta 0:00:01
     |█████████████████████████▉      | 2.7 MB 14.7 MB/s eta 0:00:01
     |██████████████████████████      | 2.7 MB 14.7 MB/s eta 0:00:01
     |██████████████████████████      | 2.7 MB 14.7 MB/s eta 0:00:01
     |██████████████████████████▏     | 2.7 MB 14.7 MB/s eta 0:00:01
     |██████████████████████████▎     | 2.8 MB 14.7 MB/s eta 0:00:01
     |██████████████████████████▍     | 2.8 MB 14.7 MB/s eta 0:00:01
     |██████████████████████████▌     | 2.8 MB 14.7 MB/s eta 0:00:01
     |██████████████████████████▋     | 2.8 MB 14.7 MB/s eta 0:00:01
     |██████████████████████████▋     | 2.8 MB 14.7 MB/s eta 0:00:01
     |██████████████████████████▊     | 2.8 MB 14.7 MB/s eta 0:00:01
     |██████████████████████████▉     | 2.8 MB 14.7 MB/s eta 0:00:01
     |███████████████████████████     | 2.8 MB 14.7 MB/s eta 0:00:01
     |███████████████████████████     | 2.8 MB 14.7 MB/s eta 0:00:01
     |███████████████████████████▏    | 2.8 MB 14.7 MB/s eta 0:00:01
     |███████████████████████████▎    | 2.9 MB 14.7 MB/s eta 0:00:01
     |███████████████████████████▍    | 2.9 MB 14.7 MB/s eta 0:00:01
     |███████████████████████████▍    | 2.9 MB 14.7 MB/s eta 0:00:01
     |███████████████████████████▌    | 2.9 MB 14.7 MB/s eta 0:00:01
     |███████████████████████████▋    | 2.9 MB 14.7 MB/s eta 0:00:01
     |███████████████████████████▊    | 2.9 MB 14.7 MB/s eta 0:00:01
     |███████████████████████████▉    | 2.9 MB 14.7 MB/s eta 0:00:01
     |████████████████████████████    | 2.9 MB 14.7 MB/s eta 0:00:01
     |████████████████████████████    | 2.9 MB 14.7 MB/s eta 0:00:01
     |████████████████████████████▏   | 2.9 MB 14.7 MB/s eta 0:00:01
     |████████████████████████████▏   | 3.0 MB 14.7 MB/s eta 0:00:01
     |████████████████████████████▎   | 3.0 MB 14.7 MB/s eta 0:00:01
     |████████████████████████████▍   | 3.0 MB 14.7 MB/s eta 0:00:01
     |████████████████████████████▌   | 3.0 MB 14.7 MB/s eta 0:00:01
     |████████████████████████████▋   | 3.0 MB 14.7 MB/s eta 0:00:01
     |████████████████████████████▊   | 3.0 MB 14.7 MB/s eta 0:00:01
     |████████████████████████████▉   | 3.0 MB 14.7 MB/s eta 0:00:01
     |█████████████████████████████   | 3.0 MB 14.7 MB/s eta 0:00:01
     |█████████████████████████████   | 3.0 MB 14.7 MB/s eta 0:00:01
     |█████████████████████████████   | 3.1 MB 14.7 MB/s eta 0:00:01
     |█████████████████████████████▏  | 3.1 MB 14.7 MB/s eta 0:00:01
     |█████████████████████████████▎  | 3.1 MB 14.7 MB/s eta 0:00:01
     |█████████████████████████████▍  | 3.1 MB 14.7 MB/s eta 0:00:01
     |█████████████████████████████▌  | 3.1 MB 14.7 MB/s eta 0:00:01
     |█████████████████████████████▋  | 3.1 MB 14.7 MB/s eta 0:00:01
     |█████████████████████████████▊  | 3.1 MB 14.7 MB/s eta 0:00:01
     |█████████████████████████████▉  | 3.1 MB 14.7 MB/s eta 0:00:01
     |█████████████████████████████▉  | 3.1 MB 14.7 MB/s eta 0:00:01
     |██████████████████████████████  | 3.1 MB 14.7 MB/s eta 0:00:01
     |██████████████████████████████  | 3.2 MB 14.7 MB/s eta 0:00:01
     |██████████████████████████████▏ | 3.2 MB 14.7 MB/s eta 0:00:01
     |██████████████████████████████▎ | 3.2 MB 14.7 MB/s eta 0:00:01
     |██████████████████████████████▍ | 3.2 MB 14.7 MB/s eta 0:00:01
     |██████████████████████████████▌ | 3.2 MB 14.7 MB/s eta 0:00:01
     |██████████████████████████████▋ | 3.2 MB 14.7 MB/s eta 0:00:01
     |██████████████████████████████▋ | 3.2 MB 14.7 MB/s eta 0:00:01
     |██████████████████████████████▊ | 3.2 MB 14.7 MB/s eta 0:00:01
     |██████████████████████████████▉ | 3.2 MB 14.7 MB/s eta 0:00:01
     |███████████████████████████████ | 3.2 MB 14.7 MB/s eta 0:00:01
     |███████████████████████████████ | 3.3 MB 14.7 MB/s eta 0:00:01
     |███████████████████████████████▏| 3.3 MB 14.7 MB/s eta 0:00:01
     |███████████████████████████████▎| 3.3 MB 14.7 MB/s eta 0:00:01
     |███████████████████████████████▍| 3.3 MB 14.7 MB/s eta 0:00:01
     |███████████████████████████████▍| 3.3 MB 14.7 MB/s eta 0:00:01
     |███████████████████████████████▌| 3.3 MB 14.7 MB/s eta 0:00:01
     |███████████████████████████████▋| 3.3 MB 14.7 MB/s eta 0:00:01
     |███████████████████████████████▊| 3.3 MB 14.7 MB/s eta 0:00:01
     |███████████████████████████████▉| 3.3 MB 14.7 MB/s eta 0:00:01
     |████████████████████████████████| 3.3 MB 14.7 MB/s eta 0:00:01
     |████████████████████████████████| 3.4 MB 14.7 MB/s
     |████████████████████████████████| 43 kB 2.1 MB/s
     |████████████████████████████████| 1.2 MB 65.9 MB/s
     |████████████████████████████████| 2.3 MB 70.7 MB/s
     |████████████████████████████████| 596 kB 74.5 MB/s
     |████████████████████████████████| 61 kB 576 kB/s
     |████████████████████████████████| 3.3 MB 82.5 MB/s
     |████████████████████████████████| 895 kB 77.3 MB/s
  Building wheel for seqeval (setup.py) ... done
import torch
from transformers import AutoTokenizer, Trainer, TrainingArguments, AutoModelForSequenceClassification
from torch.utils.data import Dataset
import os
import pandas as pd
import requests
from tqdm.auto import tqdm
import numpy as np
from sklearn.metrics import accuracy_score, precision_recall_fscore_support
import re_____no_output_____model_name = 'Rostlab/prot_bert_bfd'_____no_output_____class DeepLocDataset(Dataset):
    """Face Landmarks dataset."""

    def __init__(self, split="train", tokenizer_name='Rostlab/prot_bert_bfd', max_length=1024):
        """
        Args:
            csv_file (string): Path to the csv file with annotations.
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_name, do_lower_case=False)

        if split=="train":
          self.seqs, self.labels = self.load_tr()
        elif split=="valid":
          self.seqs,self.labels = self.ld_ev()
        elif split=="test":
          self.seqs, self.labels = self.load_te()

        self.max_length = max_length

    def load_tr(seq):
      with open("/content/Train_p.fasta", "r") as f:
        data = f.readlines()
      tr_pseqs = []
      for ele in data:
          if ele[0] != '>':
              tr_pseqs.append(ele)

      with open("/content/Train_n.fasta", "r") as f:
          data = f.readlines()
      tr_nseqs = []
      for ele in data:
          if ele[0] != '>':
              tr_nseqs.append(ele)
      tr_seqs = tr_nseqs + tr_pseqs
      tr_label = [0] * len(tr_nseqs) + [1] * len(tr_pseqs)
      from sklearn.model_selection import train_test_split
      tr_seqs,none1,tr_label,none2 = train_test_split(tr_seqs,tr_label,test_size=0.2,random_state=1,shuffle=True)
      return tr_seqs,tr_label

    def ld_ev(seqs):
      with open("/content/Train_p.fasta", "r") as f:
        data = f.readlines()
      tr_pseqs = []
      for ele in data:
          if ele[0] != '>':
              tr_pseqs.append(ele)

      with open("/content/Train_n.fasta", "r") as f:
          data = f.readlines()
      tr_nseqs = []
      for ele in data:
          if ele[0] != '>':
              tr_nseqs.append(ele)
      tr_seqs = tr_nseqs + tr_pseqs
      tr_label = [0] * len(tr_nseqs) + [1] * len(tr_pseqs)
      from sklearn.model_selection import train_test_split
      tr_seqs,none1,tr_label,none2 = train_test_split(tr_seqs,tr_label,test_size=0.2,random_state=1,shuffle=True)
      return none1,none2

    def covert(sequence):
      a = list(sequence)
      st = ''
      for ele in a:
        st = st+ele+' '
      st = st[0:-1]
      return st

    def load_te(seq):
      with open("/content/te_p.fasta", "r") as f:
        data = f.readlines()
      te_pseqs = []
      for ele in data:
          if ele[0] != '>':
              te_pseqs.append(ele)

      with open("/content/te_n.fasta", "r") as f:
          data = f.readlines()
      te_nseqs = []
      for ele in data:
          if ele[0] != '>':
              te_nseqs.append(ele)
      te_seqs = te_nseqs + te_pseqs
      te_label = [0] * len(te_nseqs) + [1] * len(te_pseqs)
      return te_seqs,te_label

    def __len__(self):
        return len(self.labels)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()

        seq = " ".join("".join(self.seqs[idx].split()))
        seq = re.sub(r"[UZOB]", "X", seq)

        seq_ids = self.tokenizer(seq, truncation=True, padding='max_length', max_length=self.max_length)

        sample = {key: torch.tensor(val) for key, val in seq_ids.items()}
        sample['labels'] = torch.tensor(self.labels[idx])

        return sample_____no_output_____train_dataset = DeepLocDataset(split="train", tokenizer_name=model_name, max_length=1024)
val_dataset = DeepLocDataset(split="valid", tokenizer_name=model_name, max_length=1024)
test_dataset = DeepLocDataset(split="test", tokenizer_name=model_name, max_length=1024)_____no_output_____def compute_metrics(pred):
    labels = pred.label_ids
    preds = pred.predictions.argmax(-1)
    precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='binary')
    acc = accuracy_score(labels, preds)
    return {
        'accuracy': acc,
        'f1': f1,
        'precision': precision,
        'recall': recall
    }_____no_output_____def model_init():
  return AutoModelForSequenceClassification.from_pretrained(model_name)_____no_output_____training_args = TrainingArguments(
    output_dir='./results',          # output directory
    num_train_epochs=2,              # total number of training epochs
    per_device_train_batch_size=1,   # batch size per device during training
    per_device_eval_batch_size=32,   # batch size for evaluation
    warmup_steps=5,               # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='./logs',            # directory for storing logs
    logging_steps=200,               # How often to print logs
    do_train=True,                   # Perform training
    do_eval=True,                    # Perform evaluation
    evaluation_strategy="epoch",     # evalute after eachh epoch
    gradient_accumulation_steps=64,  # total number of steps before back propagation
    fp16=False,                       # Use mixed precision
    fp16_opt_level="02",             # mixed precision mode
    run_name="ProBert-BFD-MS",       # experiment name
    seed=2                           # Seed for experiment reproducibility 3x3
)

trainer = Trainer(
    model_init=model_init,                # the instantiated 🤗 Transformers model to be trained
    args=training_args,                   # training arguments, defined above
    train_dataset=train_dataset,          # training dataset
    eval_dataset=val_dataset,             # evaluation dataset
    compute_metrics = compute_metrics,    # evaluation metrics
)

trainer.train()PyTorch: setting up devices
The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).
loading configuration file https://huggingface.co/Rostlab/prot_bert_bfd/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/67f460acc7e7e147ff828e909ffe419d00d66ce679c682bc4ab715c107bcbe41.baf557855a8618d0ddfb6c23bfd135bfc38ccf8c3fb099b8df45eb110ccf05e9
Model config BertConfig {
  "_name_or_path": "Rostlab/prot_bert_bfd",
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.0,
  "classifier_dropout": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.0,
  "hidden_size": 1024,
  "initializer_range": 0.02,
  "intermediate_size": 4096,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 40000,
  "model_type": "bert",
  "num_attention_heads": 16,
  "num_hidden_layers": 30,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "transformers_version": "4.15.0",
  "type_vocab_size": 2,
  "use_cache": true,
  "vocab_size": 30
}

loading weights file https://huggingface.co/Rostlab/prot_bert_bfd/resolve/main/pytorch_model.bin from cache at /root/.cache/huggingface/transformers/0a05878f9e3a0d39834dc6f21b88471696d7453a07bac7246152a6ef307c9af4.c5b9869da882baaf70e8e70cf32d81500803511e3220e24457115a03445fa65f
Some weights of the model checkpoint at Rostlab/prot_bert_bfd were not used when initializing BertForSequenceClassification: ['cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.bias', 'cls.predictions.decoder.bias', 'cls.predictions.bias']
- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at Rostlab/prot_bert_bfd and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
loading configuration file https://huggingface.co/Rostlab/prot_bert_bfd/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/67f460acc7e7e147ff828e909ffe419d00d66ce679c682bc4ab715c107bcbe41.baf557855a8618d0ddfb6c23bfd135bfc38ccf8c3fb099b8df45eb110ccf05e9
Model config BertConfig {
  "_name_or_path": "Rostlab/prot_bert_bfd",
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.0,
  "classifier_dropout": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.0,
  "hidden_size": 1024,
  "initializer_range": 0.02,
  "intermediate_size": 4096,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 40000,
  "model_type": "bert",
  "num_attention_heads": 16,
  "num_hidden_layers": 30,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "transformers_version": "4.15.0",
  "type_vocab_size": 2,
  "use_cache": true,
  "vocab_size": 30
}

loading weights file https://huggingface.co/Rostlab/prot_bert_bfd/resolve/main/pytorch_model.bin from cache at /root/.cache/huggingface/transformers/0a05878f9e3a0d39834dc6f21b88471696d7453a07bac7246152a6ef307c9af4.c5b9869da882baaf70e8e70cf32d81500803511e3220e24457115a03445fa65f
Some weights of the model checkpoint at Rostlab/prot_bert_bfd were not used when initializing BertForSequenceClassification: ['cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.bias', 'cls.predictions.decoder.bias', 'cls.predictions.bias']
- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at Rostlab/prot_bert_bfd and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
***** Running training *****
  Num examples = 665
  Num Epochs = 2
  Instantaneous batch size per device = 1
  Total train batch size (w. parallel, distributed & accumulation) = 64
  Gradient Accumulation steps = 64
  Total optimization steps = 20
predictions, label_ids, metrics = trainer.predict(test_dataset)
print(metrics)***** Running Prediction *****
  Num examples = 212
  Batch size = 32
tokenizer = AutoTokenizer.from_pretrained(model_name, do_lower_case=False,local_files_only=False)
trainer.save_model('./model/')

#tokenizer.save_pretrained('./content/drive/MyDrive/model/')
#trainer.save_model('./content/drive/MyDrive/model/')

from transformers import AutoTokenizer, AutoModel, pipeline
model = AutoModel.from_pretrained("/content/model",local_files_only=True)

fe = pipeline('feature-extraction', model=model, tokenizer=tokenizer,device=0 )
sequences_Example = ["A E T"]
embedding = fe(sequences_Example)
embedding = np.array(embedding)
print(embedding.shape)
model.predict(test_dataset)loading configuration file https://huggingface.co/Rostlab/prot_bert_bfd/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/67f460acc7e7e147ff828e909ffe419d00d66ce679c682bc4ab715c107bcbe41.baf557855a8618d0ddfb6c23bfd135bfc38ccf8c3fb099b8df45eb110ccf05e9
Model config BertConfig {
  "_name_or_path": "Rostlab/prot_bert_bfd",
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.0,
  "classifier_dropout": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.0,
  "hidden_size": 1024,
  "initializer_range": 0.02,
  "intermediate_size": 4096,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 40000,
  "model_type": "bert",
  "num_attention_heads": 16,
  "num_hidden_layers": 30,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "transformers_version": "4.15.0",
  "type_vocab_size": 2,
  "use_cache": true,
  "vocab_size": 30
}

loading file https://huggingface.co/Rostlab/prot_bert_bfd/resolve/main/vocab.txt from cache at /root/.cache/huggingface/transformers/0a4700fc7f247c9f9bce4868c3a27d3490d77df38af38872dfa437aae5c01780.998710b71be69197f8bce77a79c59cf2b4fd59d44504d771ed5b82e305bf3e2a
loading file https://huggingface.co/Rostlab/prot_bert_bfd/resolve/main/tokenizer.json from cache at None
loading file https://huggingface.co/Rostlab/prot_bert_bfd/resolve/main/added_tokens.json from cache at None
loading file https://huggingface.co/Rostlab/prot_bert_bfd/resolve/main/special_tokens_map.json from cache at /root/.cache/huggingface/transformers/a96f9efecc47827e957ec053de7c57570abdc32a586f94b84bcd5cff71457e73.dd8bd9bfd3664b530ea4e645105f557769387b3da9f79bdb55ed556bdd80611d
loading file https://huggingface.co/Rostlab/prot_bert_bfd/resolve/main/tokenizer_config.json from cache at /root/.cache/huggingface/transformers/0c19549fa50aee807268e582b0c46cd6ff8aef9711662bf9a33dceb61059e8fb.ee2f006d53c792efe2ed11adf7b44d96af5eda4e1077f2f1ecd2f713c55f96e4
loading configuration file https://huggingface.co/Rostlab/prot_bert_bfd/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/67f460acc7e7e147ff828e909ffe419d00d66ce679c682bc4ab715c107bcbe41.baf557855a8618d0ddfb6c23bfd135bfc38ccf8c3fb099b8df45eb110ccf05e9
Model config BertConfig {
  "_name_or_path": "Rostlab/prot_bert_bfd",
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.0,
  "classifier_dropout": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.0,
  "hidden_size": 1024,
  "initializer_range": 0.02,
  "intermediate_size": 4096,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 40000,
  "model_type": "bert",
  "num_attention_heads": 16,
  "num_hidden_layers": 30,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "transformers_version": "4.15.0",
  "type_vocab_size": 2,
  "use_cache": true,
  "vocab_size": 30
}

loading configuration file https://huggingface.co/Rostlab/prot_bert_bfd/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/67f460acc7e7e147ff828e909ffe419d00d66ce679c682bc4ab715c107bcbe41.baf557855a8618d0ddfb6c23bfd135bfc38ccf8c3fb099b8df45eb110ccf05e9
Model config BertConfig {
  "_name_or_path": "Rostlab/prot_bert_bfd",
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.0,
  "classifier_dropout": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.0,
  "hidden_size": 1024,
  "initializer_range": 0.02,
  "intermediate_size": 4096,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 40000,
  "model_type": "bert",
  "num_attention_heads": 16,
  "num_hidden_layers": 30,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "transformers_version": "4.15.0",
  "type_vocab_size": 2,
  "use_cache": true,
  "vocab_size": 30
}

Saving model checkpoint to ./model/
Configuration saved in ./model/config.json
Model weights saved in ./model/pytorch_model.bin
***** Running Prediction *****
  Num examples = 212
  Batch size = 32


# Notebook from NhuanTDBK/Kaggle_StackedOverflow
Path: QuickLookAtData.ipynb

import numpy as np
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
import seaborn as snb
import nltk
from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation
import matplotlib.pyplot as plt
import re_____no_output_____train_bio = pd.read_csv("input_light/biology.csv",encoding='utf-8')
train_cooking = pd.read_csv("input_light/cooking.csv",encoding='utf-8')
train_crypto = pd.read_csv("input_light/crypto.csv",encoding='utf-8')
train_dyi = pd.read_csv("input_light/diy.csv",encoding='utf-8')
train_robotic = pd.read_csv("input_light/robotics.csv",encoding='utf-8')
train_travel = pd.read_csv("input_light/travel.csv",encoding='utf-8')
# test_df = pd.read_csv("input_light/test.csv",encoding='utf-8')_____no_output_____# df_list = []
df_list = pd.concat([train_bio,train_cooking,train_crypto,train_dyi,train_robotic,train_travel])_____no_output_____

## Length of dataset_____no_output_____

df_list.head()_____no_output_____df_list.dropna(axis=0,how='any',inplace=True)_____no_output_____df_concat = df_list['title'] + ' ' + df_list['content']_____no_output_____df_list['doc'] = df_concat.map(lambda d: d.replace('-','_'))_____no_output_____print "Total length = %s"%df_list.shape[0]Total length = 86998
print "Length of null data = %s"%df_list[df_list['doc'].isnull()].shape[0]Length of null data = 0
print "Length of null data = %s"%df_list[df_list['tags'].isnull()].shape[0]Length of null data = 0

## Number of unique tags in training datasets_____no_output_____

df_list['tags'][0:10]_____no_output_____# Y = [item.replace("-","_").split(" ") for item in df_list['tags']]
all_tags = df_list['tags'].map(lambda d: d.replace('-','_'))_____no_output_____length_of_tags = pd.DataFrame(zip(*np.unique([len(tag) for tag in Y],return_counts=True)),columns=['length','count'])_____no_output_____%matplotlib inline
# ax = plt.subplot()
# ax.plot(length_of_tags['length'],label='')
ax = length_of_tags.set_index("length").plot(kind='bar',rot=0,title="Length of tags")_____no_output_____

## Analyze type of data_____no_output_____

type_of_tags = []

def get_post_tag(tags):
    return np.array(nltk.pos_tag(tags.split(' ')))[:,1]

for tag in df_list.tags:
    type_of_tags.extend(get_post_tag(tag))

count_tag = pd.DataFrame(zip(*np.unique(type_of_tags,return_counts=True)),columns=['tag','count'])_____no_output_____%matplotlib inline
count_tag.sort_values(['count'],ascending=False).iloc[0:10].set_index(['tag']).plot(kind='bar',title="Type of tag",rot=0)_____no_output_____

## Number of tags in title_____no_output_____

def remove_number(demo):
    a = []
    for f in demo.split(' '):
        if not(f.isdigit()):
            a.append(f)
    return a_____no_output_____# token_pattern='^[0-9]*'
word_title = [remove_number(item) for item in df_list['title']]
title_dict = list(set(sum(word_title,[])))_____no_output_____title_dict = set(title_dict)_____no_output_____number_of_tags_in_title = title_dict.intersection(tags)_____no_output_____print "Ratio of tags in title = %s %%"%(len(number_of_tags_in_title)*100/len(all_tags))Ratio of tags in title = 1 %

## TF-IDF + LDA_____no_output_____

tfidfVec = TfidfVectorizer(max_df=0.95)
countVec = CountVectorizer(max_df=0.95,min_df=2,ngram_range=(1,2),token_pattern='[a-zA-Z]{2,}')_____no_output_____df_tf = countVec.fit_transform(df_list['doc'])_____no_output_____n_topics = 6_____no_output_____lda = LatentDirichletAllocation(n_topics=n_topics, max_iter=5,
                                learning_method='online',
                                learning_offset=50.,
                                random_state=4111,n_jobs=2)
lda.fit(df_tf)_____no_output_____def print_top_words(model, feature_names, n_top_words):
    for topic_idx, topic in enumerate(model.components_):
        print("Topic #%d:" % topic_idx)
        print(" ".join([feature_names[i]
                        for i in topic.argsort()[:-n_top_words - 1:-1]]))
    print()_____no_output_____print("\nTopics in LDA model:")
n_top_words = 30
tf_feature_names = countVec.get_feature_names()
print_top_words(lda, tf_feature_names, n_top_words)_____no_output_____idx = lda.transform(df_tf[0]).argmax()

# words_topic = zip(*(lda.components_[3],countVec.get_feature_names()))
words_topic = lda.components_[idx]
feature_names = countVec.get_feature_names()
n_words = 100

print [feature_names[i] for i in words_topic.argsort()[:-n_words - 1:-1]]_____no_output_____tokenizer = countVec.build_analyzer()
a = tokenizer(df_list['title'].iloc[0])
b = [(c,words_topic[countVec.vocabulary_[c]]) for c in a if countVec.vocabulary_.has_key(c)]_____no_output_____df_list['tags'].iloc[0]_____no_output_____