# Notebook from lmoresi/uw-docker-landing-page
Path: notebooks/__work_in_progress/models_broken/ConvectionSolidus.ipynb

## Convection model with solidus

This notebook implements a thermal convection model with temperature dependent viscosity.

$$
    \nabla \cdot \left( \eta \nabla \dot\varepsilon \right) - \nabla p =
     \mathrm{Ra}  T \mathbf{\hat z}
$$

$$
    \nabla \cdot \mathbf{v} = 0
$$

The thermal evolution is controlled by advection and  diffusion:

$$
    \frac{D T}{D t} = \nabla^2 T
$$

The viscosity is defined by

$$
    \eta = \eta_0 \exp\left( \frac{E}{T+1} \right)
$$

$\mathrm{Ra}$ is the Rayleigh number is defined by

$$
    \mathrm{Ra} = \frac{g \rho \alpha \Delta T h ^3}{\kappa \eta _0} ;
$$

An initial temperature distribution is required. It could be read in from a previous calculation but here we use a perturbed linear temperature gradient.

$$
    T = (1 - y) + 0.1 \cdot \left( \sin(\pi y) + \cos(\pi x) \right)
$$

We compare the temperature at each point with a simple solidus defined as a linear function with depth
and capped below a certain depth. The excess temperature could be used as the start of a very simple
instantaneous and history-independent melting model such as that of Watson & McKenzie (1991).

We can also see how the melting at depth can be mapped to the surface and tracked on particles through time.

Watson, S., McKenzie, D., Melt Generation by Plumes: A Study of Hawaiian Volcanism
J. Petrology (1991) 32 (3): 501-537 doi:10.1093/petrology/32.3.501
_____no_output_____

<code>
# Convection model that uses functions to determine where the temperature
# lies above a simple solidus (linear with depth, then capped)

# Also shows how to map the melt production values on a swarm to the surface (but not how to
# create an integral of the values ...

# TODO: show how we can track the melt history over time 1) for a particle and 2) for a particle at the surface
# recording melt distribution (will also be useful for subduction to track the arc)

%matplotlib inline

import underworld as uw
import math
from underworld import function as fn
import glucifer.pylab as plt_____no_output_____dim = 2_____no_output_____# create mesh objects (make alternates - either Q2dP1 or Q1dP0 which can be used for computation. The discontinuous
# submeshes can also be retrieved)

Q2dQ1Mesh = uw.mesh.FeMesh_Cartesian( elementType='Q2/dPc1',
                                      elementRes=(64,16),
                                      minCoord=(0.,0.),
                                      maxCoord=(4.,1.)  )

Q2Mesh   = Q2dQ1Mesh # the default action is to behave as the master mesh
dQ1Mesh  = Q2dQ1Mesh.subMesh

Q1dP0Mesh = uw.mesh.FeMesh_Cartesian( elementType='Q1/dQ0',
                                         elementRes=(128,32),
                                           minCoord=(0.,0.),
                                           maxCoord=(4.,1.)  )

Q1Mesh =  Q1dP0Mesh
dP0Mesh = Q1dP0Mesh.subMesh

velocityMesh =  Q2Mesh
pressureMesh = dQ1Mesh
_____no_output_____# create meshvariables
velocityField    = uw.mesh.MeshVariable( mesh=velocityMesh,   nodeDofCount=dim )
pressureField    = uw.mesh.MeshVariable( mesh=pressureMesh,   nodeDofCount=1 )
temperatureField = uw.mesh.MeshVariable( mesh=velocityMesh,   nodeDofCount=1 )_____no_output_____# Initialise data.. Note that we are also setting boundary conditions here

velocityField.data[:] = [0.,0.]
pressureField.data[:] = 0.
temperatureField.data[:] = 0.

# Setup temperature initial condition via underlying data viewed as numpy arrays

toptemp = 0.
bottomtemp = 1.
scaleFactor = bottomtemp - toptemp
import math
tempData = temperatureField.data
perturbation_scale = 0.1
for index, coord in enumerate(velocityMesh.data):
    pertCoeff = math.cos( math.pi * coord[0] ) * math.sin( math.pi * coord[1] )
    tempData[index] = min(1,max(0,toptemp + scaleFactor*(1. - coord[1]) + perturbation_scale * pertCoeff ));_____no_output_____figtemp = plt.Figure(figsize=(1024,256))
tempminmax = fn.view.min_max(temperatureField)
figtemp.Surface(tempminmax, velocityMesh, colours=["red", "white", "blue"], colourBar=False)
figtemp.show()_____no_output_____# Get list of special sets.
# These are sets of vertices on the mesh. In this case we want to set them as boundary conditions.
velocityMesh.specialSets.keys()_____no_output_____# Get the actual sets
#
#  HJJJJJJH
#  I      I
#  I      I
#  I      I
#  HJJJJJJH
#
#  Note that H = I & J

# Note that we use operator overloading to combine sets
IWalls = velocityMesh.specialSets["MinI_VertexSet"] + velocityMesh.specialSets["MaxI_VertexSet"]
JWalls = velocityMesh.specialSets["MinJ_VertexSet"] + velocityMesh.specialSets["MaxJ_VertexSet"]_____no_output_____# Now setup the dirichlet boundary condition
# Note that through this object, we are flagging to the system
# that these nodes are to be considered as boundary conditions.
# Also note that we provide a tuple of sets.. One for the Vx, one for Vy.
freeslipBC = uw.conditions.DirichletCondition(    variable=velocityField,
                                                  indexSetsPerDof=(IWalls,JWalls) )

# also set dirichlet for temp field
tempBC = uw.conditions.DirichletCondition(    variable=temperatureField,
                                              indexSetsPerDof=(JWalls,) )_____no_output_____# lets ensure temp boundaries are still what we want as the IC above might have set slightly different values
# on the boundaries

for index in velocityMesh.specialSets["MinJ_VertexSet"]:
    temperatureField.data[index] = 1.
for index in velocityMesh.specialSets["MaxJ_VertexSet"]:
    temperatureField.data[index] = 0_____no_output_____# We create swarms of particles which can advect, and which we can use to track history
gSwarm = uw.swarm.Swarm( mesh=velocityMesh )

# Now we add a data variable which will store an index to determine material
materialVariable = gSwarm.add_variable( dataType="float", count=1 )
spareVariable    = gSwarm.add_variable( dataType="float", count=1 )

# Layouts are used to populate the swarm across the whole domain
# Create the layout object
layout = uw.swarm.layouts.GlobalSpaceFillerLayout( swarm=gSwarm, particlesPerCell=13 )
# Now use it to populate.
gSwarm.populate_using_layout( layout=layout )_____no_output_____import numpy

materialVariable.data[:] = 100.0

xCoord = numpy.sign(numpy.mod(gSwarm.particleCoordinates.data.T[0],0.25) - 0.125).reshape(-1,1)
yCoord = numpy.sign(numpy.mod(gSwarm.particleCoordinates.data.T[1],0.25) - 0.125).reshape(-1,1)

materialVariable.data[:] = xCoord[:] * yCoord[:]

mVmin = materialVariable.data.min()
mVmax = materialVariable.data.max()

materialVariable.data[:] =  (materialVariable.data[:] - mVmin) / (mVmax-mVmin)
spareVariable.data[:]    =  1.0 + (materialVariable.data[:] - mVmin) / (mVmax-mVmin)
_____no_output_____# Rheology

eta0 = 1.0e-3
activationEnergy = 2.0 * math.log(1/eta0)
Ra = 1.e6
fn_viscosity = eta0 *  fn.math.exp( activationEnergy / (fn.misc.max(0.0,fn.misc.min(1.0,temperatureField))+1.) )

densityFn = Ra*temperatureField
# Define our gravity using a python tuple (this will be automatically converted to a function)
gravity = ( 0.0, 1.0 )
# now create a buoyancy force vector.. the gravity tuple is converted to a function
# here via operator overloading
buoyancyFn = gravity*densityFn

print "Delta eta = ", fn_viscosity.evaluate((0.0,1.0)) / fn_viscosity.evaluate((0.0,0.0))
Delta eta =  [[1000.]]
# lets take a look at the eta function

figEta = plt.Figure(figsize=(1024,256))
figEta.Surface(fn_viscosity, velocityMesh, colours=["#FFFFFF","#555555", "#111111", '#000000'], colourBar=False)
figEta.Points(gSwarm, colours=["darkred", "blue"], pointSize=3.0, sizeVariable=materialVariable, colourVariable=materialVariable, opacity=0.5, colourBar=False)
figEta.show()_____no_output_____# Setup a stokes system
# For PIC style integration, we include a swarm for the a PIC integration swarm is generated within.
# For gauss integration, simple do not include the swarm. Nearest neighbour is used where required.
stokes = uw.systems.Stokes(velocityField=velocityField,
                              pressureField=pressureField,
                              conditions=[freeslipBC,],
                              fn_viscosity=fn.exception.SafeMaths(fn_viscosity),
                              fn_bodyforce=buoyancyFn )

solver=uw.systems.Solver(stokes)

solver.options.main.Q22_pc_type='uw'  # also try 'gtkg', 'gkgdiag' and 'uwscale'
solver.options.main.penalty = 1.0
solver.options.A11.ksp_rtol=1e-4
solver.options.scr.ksp_rtol=1e-3
solver.options.scr.use_previous_guess = True
solver.options.scr.ksp_set_min_it_converge = 1
solver.options.scr.ksp_set_max_it = 100
solver.options.mg.levels = 5
solver.options.mg.mg_levels_ksp_type = 'chebyshev'
solver.options.mg_accel.mg_accelerating_smoothing = True
solver.options.mg_accel.mg_accelerating_smoothing_view = False
solver.options.mg_accel.mg_smooths_to_start = 1

# Load this solver onto the equation system (displacing the default)
stokes.solve = solver.solve
_____no_output_____# Create advdiff system
advDiff = uw.systems.AdvectionDiffusion( temperatureField, velocityField, fn_diffusivity=1., conditions=[tempBC,] )

# And the particle updating system
advector = uw.systems.SwarmAdvector( swarm=gSwarm, velocityField=velocityField, order=2 )

# Also create some integral objects which are used to calculate statistics
v2sum_integral  = uw.utils.Integral( mesh=velocityMesh, fn=fn.math.dot(velocityField, velocityField) )
volume_integral = uw.utils.Integral( mesh=velocityMesh, fn=1. )_____no_output_____def nusselt(temp_field, mesh):

    surface_Tgradient = -temp_field.fn_gradient.evaluate(velocityMesh.specialSets["MaxJ_VertexSet"]).T[1]
    basalT = temp_field.evaluate(velocityMesh.specialSets["MinJ_VertexSet"])

    nusselt = surface_Tgradient.mean() / basalT.mean()

    return nusselt_____no_output_____# Stepping. Initialise time and timestep.
realtime = 0.
step = 0
time_vals = []
vrms_vals = []
Nu_vals = []_____no_output_____import time

start = time.clock()
# Perform 1000 or so steps

volume = volume_integral.integrate()
while step<500:
    # Get solution for initial configuration
    stokes.solve()
    # Retrieve the maximum possible timestep for the AD system.
    dt = advDiff.get_max_dt()
    if step == 0:
        dt = 0.
    # Advect using this timestep siz
    advDiff.integrate(dt)
    advector.integrate(dt)
    # Calculate the RMS velocity
    v2sum = v2sum_integral.integrate()
    rms_v = math.sqrt(v2sum[0]/volume[0])
    nu_no = nusselt(temperatureField, velocityMesh)

    # Increment time and store results
    realtime += dt
    step += 1
    vrms_vals.append( rms_v )
    time_vals.append( realtime )
    Nu_vals.append(nu_no)

    if step == 0 or step % 25 == 0:
        print "step = {:04d}; time = {:.6f}, vrms = {:6.2f}, nusselt = {:6.4f}, CPU = {:4.1f}s".format(
               step, realtime, rms_v, nu_no, time.clock()-start)
        # Animation frame

        figVT = plt.Figure()
        figVT = plt.Figure(figsize=(1024,512))
        figVT.Surface(temperatureField, velocityMesh, colours=["#333333","#666666", "#999999", '#FFFFFFF'])
        figVT.Points(gSwarm, colours=["red", "blue"], colourBar=False, pointSize=4.0, colourVariable=materialVariable, opacity=0.5)
        figVT.VectorArrows(velocityField, velocityMesh, lengthScale=0.1/rms_v, arrowHeadSize=0.15 )
        figVT.save_image("TEST/Temperature+Markers.{:04d}".format(step))
        del figVT

        step = 0025; time = 0.000506, vrms = 162.51, nusselt = 1.0103, CPU =  8.0s
print "average time per timestep = ", (time.clock() - start)/float(step), " over ", step, " steps"_____no_output_____import matplotlib.pyplot as pyplot
#import numpy as np

surface_points_X = velocityMesh.data[velocityMesh.specialSets["MinJ_VertexSet"].data].T[0]

figure, (plot1, plot2) = pyplot.subplots(1,2)
figure.set_size_inches(10,6)
plot1.plot(time_vals, vrms_vals, linewidth=2.5)
plot2.plot(time_vals, Nu_vals, linewidth=2.5)

figure.show()_____no_output_____figVT = plt.Figure(figsize=(1024,384))
figVT.Surface(temperatureField, velocityMesh, colours=["#333333","#666666", "#999999", '#FFFFFFF'])
figVT.Points(gSwarm, colours=["darkred", "white"], colourBar=True, pointSize=4.0, colourVariable=materialVariable, opacity=0.25)
figVT.VectorArrows(velocityField, velocityMesh, lengthScale=0.1/rms_v, arrowHeadSize=0.15 )
figVT.show()_____no_output_____# Definition of a linear solidus (with cutoff) using underworld functions

T0 = 0.7
D0 = 1.0
Tex = 0.85
Dex = 0.5

yCoordFn = fn.input()[1]
solidusFn = T0 - ((T0 - Tex) / (D0 - Dex)) * (D0 - yCoordFn)
excessTempFn = fn.misc.max(0.0, temperatureField - solidusFn)

ET2 = excessTempFn.evaluate(gSwarm)
_____no_output_____spareVariable.data[:] = ET2[:]

limTempFn = fn.misc.min(fn.misc.max(temperatureField, 0.0),1.0)

figVT = plt.Figure(figsize=(1024,256))
figVT.Surface(limTempFn, velocityMesh, colours=["#00000","#444444", "#999999", '#FFFFFFF'])
figVT.Points(gSwarm, colours=["white", "red", "yellow"], colourBar=True, pointSize=100.0,
             colourVariable=spareVariable, sizeVariable=spareVariable)
figVT.VectorArrows(velocityField, velocityMesh, lengthScale=0.1/rms_v, arrowHeadSize=0.15 )
figVT.show()_____no_output_____## map the melt production rate (currently excess temperature) to the surface
## in parallel this requires us to communicate the information to processes which
## own the surface. One way to do this for swarm data is to teleport the whole lot to y=1.0

# An improvement would be to only make a swarm with the particles having non-zero melt
# and then flatten this to the surface

melting_particles = numpy.where(spareVariable.data > 0.0)[0]

spareSwarm = uw.swarm.Swarm( mesh=Q1Mesh )
spareSwarm.add_particles_with_coordinates(gSwarm.particleCoordinates.data[melting_particles])
newVar         = spareSwarm.add_variable(dataType="float", count=1)
newVar.data[:] = spareVariable.data[melting_particles]

# Flatten swarm to surface
spareSwarm.particleCoordinates.data[:,1] = 1.0
spareSwarm.update_particle_owners()
_____no_output_____ExcessTField = uw.mesh.MeshVariable( mesh=dP0Mesh, nodeDofCount=1 )
ExcessTField.data[:] = 0.0_____no_output_____for point, value in enumerate(newVar.data):
#    print point, value
    cell = spareSwarm.owningCell.data[point]
    ExcessTField.data[ cell ] += value

print numpy.count_nonzero(ExcessTField.data)_____no_output_____figVT = plt.Figure(figsize=(1024,256))
figVT.Surface(ExcessTField, Q1Mesh, colours=["#00000","red", "orange", 'yellow'])
figVT.Points(gSwarm, colours=["white", "red", "yellow"], colourBar=True, pointSize=100.0,
             colourVariable=spareVariable, sizeVariable=spareVariable)
figVT.show()_____no_output_____surfacePoints = velocityMesh.data[velocityMesh.specialSets["MaxJ_VertexSet"].data]
surfaceExcessT = ExcessTField.evaluate(surfacePoints)

figure, (plot1) = pyplot.subplots(1,1)
figure.set_size_inches(12,3)
plot1.scatter(surfacePoints.T[0], surfaceExcessT, marker="x")
plot1.set_xlim(0,4)

figure.show()

_____no_output_____## What does the stress look like ?

_____no_output_____
</code>


# Notebook from ramon-schildknecht/unstructered_data_analysis
Path: Ramon_Schildknecht.ipynb

# Task_____no_output_____## Requirements
1. Do text preprocessing (e.g., stopword removal, lemmatization, stemming, etc.)
2. TF-IDF text representation
3. Run LDA
4. Identify the optimal number of topics
5. Show top 10 words for each topic.
_____no_output_____## Data
The dataset consists 2885 datasets information in 15 columns:

- Title
- Subtitle
- Owner
- Vote
- Last update
- Tags
- Datatype
- Size
- License
- Views
- Downloads
- Kernels
- Topics
- URL
- Description
_____no_output_____# Approach_____no_output_____One can see that the variables title, subtitle as well as description can be used for topic modeling. Therefore we start with 'Title', go on with 'Description' and finish with 'Subtitle' (if necessary) to get the best possible topics out of the dataset._____no_output_____# Step-by-Step_____no_output_____## Preparation
### Load Packages & Data_____no_output_____

from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation_____no_output_____

# Load the Pandas libraries with alias 'pd'
import pandas as pd
# load data
pd.options.display.max_rows = 10
d = pd.read_csv("dataset.csv")
d
d2 = d # backup for testing_____no_output_____# show dataframe information
# d.shape
d.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2885 entries, 0 to 2884
Data columns (total 15 columns):
Title          2885 non-null object
Subtitle       2413 non-null object
Owner          2885 non-null object
Votes          2885 non-null int64
Last Update    2885 non-null object
Tags           1700 non-null object
Data Type      2885 non-null object
Size           2885 non-null object
License        2885 non-null object
Views          2874 non-null object
Download       2803 non-null object
Kernels        1276 non-null object
Topics         2243 non-null object
URL            2885 non-null object
Description    2874 non-null object
dtypes: int64(1), object(14)
memory usage: 338.2+ KB

# "Title" Analysis_____no_output_____## Text Preprocessing_____no_output_____

# preprocess 'Title'
# Load the regular expression library
import re

# Remove unnessesary characters - simple form of lemmatization
d['title_preprocessed'] = d['Title'].map(lambda x: re.sub('[,\.!?]', '', x))

# Convert the titles to lowercase
d['title_preprocessed'] = d['title_preprocessed'].map(lambda x: x.lower())

# Print out the first rows
d['title_preprocessed'].head() # success_____no_output_____# check interim result

# Import the wordcloud library
from wordcloud import WordCloud

# Join the different processed titles together.
long_string = ','.join(list(d['title_preprocessed'].values))

# Create a WordCloud object
wordcloud = WordCloud(background_color="white", max_words=5000,
                      contour_width=3, contour_color='steelblue',
                      height=400, width = 800)

# Generate a word cloud
wordcloud.generate(long_string)

# Visualize the word cloud
wordcloud.to_image()_____no_output_____# remove stop words
from nltk.corpus import stopwords
stop = stopwords.words('english')
print(stop)
# title_no_stop_words.values.apply(lambda x: [item for item in x if item not in stop]) # first unsuccessful approach
# d['title_preprocessed'] = pd.Series([word for word in d['title_preprocessed'] if word not in stop]) # second unsuccesful approach
d['title_preprocessed'] = d['title_preprocessed'].str.lower().str.split() # old preprocessing step for following solution
d['title_preprocessed'] = d['title_preprocessed'].apply(lambda x: [item for item in x if item not in stop])['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're", "you've", "you'll", "you'd", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', "she's", 'her', 'hers', 'herself', 'it', "it's", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', "that'll", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', "don't", 'should', "should've", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', "aren't", 'couldn', "couldn't", 'didn', "didn't", 'doesn', "doesn't", 'hadn', "hadn't", 'hasn', "hasn't", 'haven', "haven't", 'isn', "isn't", 'ma', 'mightn', "mightn't", 'mustn', "mustn't", 'needn', "needn't", 'shan', "shan't", 'shouldn', "shouldn't", 'wasn', "wasn't", 'weren', "weren't", 'won', "won't", 'wouldn', "wouldn't"]
# check result
print(list(d['Title'][15:16]))
print(list(d['title_preprocessed'][15:16]))
# success['Where it Pays to Attend College']
[['pays', 'attend', 'college']]
# perform stemming
from nltk.stem import PorterStemmer
stemming = PorterStemmer()
temp_list = []
for words in list(d['title_preprocessed']):
    temp_list.append(stemming.stem(" ".join(words)))

    # print(words)
    for word in words:
        stemming.stem(word)
        # print(word)

d['title_preprocessed'] = pd.Series(temp_list)_____no_output_____# check result
print(d['Title'][:10])
print("\n"*2)
print(d['title_preprocessed'][:10])
# success0                Credit Card Fraud Detection
1                   European Soccer Database
2                    TMDB 5000 Movie Dataset
3                  Human Resources Analytics
4                  Global Terrorism Database
5                    Bitcoin Historical Data
6    Kaggle ML and Data Science Survey, 2017
7                               Iris Species
8               World Development Indicators
9     Daily News for Stock Market Prediction
Name: Title, dtype: object

0              credit card fraud detect
1               european soccer databas
2               tmdb 5000 movie dataset
3                human resources analyt
4              global terrorism databas
5               bitcoin historical data
6    kaggle ml data science survey 2017
7                            iris speci
8                 world development ind
9       daily news stock market predict
Name: title_preprocessed, dtype: object

## TF-IDF Transformation_____no_output_____

# load libraries
from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation_____no_output_____# create vectorizer with maximum number of features of 1000
tfidf_vectorizer = TfidfVectorizer(max_features=1000)_____no_output_____

The max_features parameter passed in the TfidfVectorizer will pick out the top 50 features ordered by their TFIDF score._____no_output_____

# tf-idf transformation
tfidf = tfidf_vectorizer.fit_transform(d['title_preprocessed'])
tfidf.data[:50]_____no_output_____# perform LDA
lda = LatentDirichletAllocation(n_components=3, max_iter=5, # LDA, n_components = number of topics to be found
                                learning_method='online',
                                learning_offset=50.,
                                random_state=22) # set seed
                                # doc_topic_prior = alpha = 0.01 per default
                                # topic_word_prior = beta

lda.fit(tfidf)

lda.components______no_output_____

every value above is connected to a word below (e. g. 8.70.... to '10')_____no_output_____

# show some feature names
tfidf_feature_names = tfidf_vectorizer.get_feature_names()
print(tfidf_feature_names[:100])['10', '100', '1000', '101', '11', '13', '13000', '15', '16', '17', '18', '19', '1948', '1949', '1970', '1980', '1990', '1994', '1995', '1996', '1997', '1998', '1999', '1k', '1st', '20', '2000', '2001', '2003', '2004', '2005', '2006', '2007', '2008', '2009', '2010', '2011', '2012', '2013', '2014', '2015', '2016', '2017', '2018', '23', '30', '311', '50', '500', '5000', '911', 'academic', 'access', 'accid', 'accident', 'accidents', 'account', 'act', 'active', 'activity', 'ad', 'ads', 'adult', 'adversarial', 'advertis', 'african', 'air', 'airbnb', 'airline', 'airports', 'al', 'alcohol', 'alexa', 'amazon', 'america', 'american', 'analysi', 'analysis', 'analyt', 'android', 'angeles', 'animal', 'anime', 'annotated', 'annual', 'answer', 'appl', 'apple', 'arabic', 'arch', 'archive', 'area', 'argentina', 'armenian', 'around', 'arrest', 'arrests', 'art', 'article', 'articles']

## Print out top 20 words
One chooses 20 instead of 10 words for better topic name finding._____no_output_____

for topic_idx, topic in enumerate(lda.components_):
    top_words = [tfidf_feature_names[i] for i in topic.argsort()[:-20-1:-1]] # -10 = last 10 values &
    print('Topic:',topic_idx,'--',top_words) # -1 because python starts at 0Topic: 0 -- ['corpu', '2017', 'nyc', 'predict', 'new', 'review', 'movie', 'project', '2014', 'york', 'index', 'sales', 'titan', 'city', 'price', 'churn', 'model', 'brazil', 'product', 'imag']
Topic: 1 -- ['dataset', 'data', 'us', '2016', 'databas', 'crime', '2015', 'world', 'india', 'survey', 'stock', 'market', 'pokemon', 'pric', 'mnist', 'imdb', 'weather', 'indian', 'house', 'attack']
Topic: 2 -- ['data', 'set', 'analysi', 'titanic', 'open', 'tweet', 'word', 'test', 'statist', 'top', 'news', 'trump', 'network', 'human', 'air', 'learning', 'reviews', 'food', 'classif', 'loan']

Topic guessing:

- Topic 0: Crime in India, e. g. https://www.kaggle.com/rajanand/crime-in-india
- Topic 1: Predict Crypto Price Trends, e. g. https://cointelegraph.com/explained/how-to-predict-crypto-price-trends-explained
- Topic 2: Wedding Announcements, e.g. https://www.nytimes.com/2019/05/26/fashion/weddings/this-weeks-wedding-announcements.html_____no_output_____# "Description" Analysis_____no_output_____## Text Preprocessing_____no_output_____

'Title' text preprocessing approach is not working. We must use an own approach for transforming the 'Description' variable. _____no_output_____

# check single item to get problematic patterns
list(d['Description'][2:3])_____no_output_____

Problematic patterns are:
- \r\n: meaning end of line in Windows
- Hyperlinks could be problematic, too

We ignore both patterns for the moment.
_____no_output_____## Use new text preprocessing method_____no_output_____

# create new variable and lower all text
d['description_preprocessed'] = d['Description'].str.lower()_____no_output_____type(d['description_preprocessed'][1])
d['description_preprocessed'][1][:200]_____no_output_____

### Check for missing values and handling them_____no_output_____

d['description_preprocessed'].isnull().values.any()_____no_output_____# get number of rows with NAs in 'Description'
d['description_preprocessed'].isnull().sum().sum()_____no_output_____# show rows with NAs in 'Description'
nan_rows = d[d['description_preprocessed'].isnull()]
nan_rows_____no_output_____# remove all rows with NAs in variable 'Description'
d = d.dropna(subset = ['description_preprocessed'])_____no_output_____# check result
print("The original number of rows were: 2884 (see section 'Load Packages & Data')")
print("The new number of rows are: " + str(len(d['description_preprocessed'])))
# successThe original number of rows were: 2884 (see section 'Load Packages & Data')
The new number of rows are: 2874
data = list(d['description_preprocessed'])
# data = data[:500]_____no_output_____data[:1]_____no_output_____

### Perform stemming_____no_output_____

# perform stemming
from nltk.tokenize import sent_tokenize, word_tokenize
from nltk.stem import PorterStemmer
# nltk.download('punkt')

porter=PorterStemmer()

def stemSentence(sentence):
    token_words=word_tokenize(sentence)
    token_words
    stem_sentence=[]
    for word in token_words:
        stem_sentence.append(porter.stem(word))
        stem_sentence.append(" ")
    return "".join(stem_sentence)

data_temp = []
for sentence in data[:2875]:
    # x = stemSentence(sentence)
    data_temp.append(stemSentence(sentence))
# sentence = data[1]
# x=stemSentence(sentence)
# print(data_temp)
data = data_temp
len(data_temp)_____no_output_____# check first 2 rows
data[:2]
# success_____no_output_____# create tf-idf vectorizer
tfidf_vectorizer = TfidfVectorizer(max_features=1000,
                                   stop_words='english',
                                   encoding = 'utf-8',
                                   decode_error = 'replace')_____no_output_____# tf-idf transformation
tfidf = tfidf_vectorizer.fit_transform(data)
tfidf.data[:50]_____no_output_____# perform LDA
lda = LatentDirichletAllocation(n_components=10, max_iter=5, # LDA, n_components = number of topics to be found
                                learning_method='online', #
                                learning_offset=50.,
                                random_state=0) # set seed
                                # doc_topic_prior = alpha = 0.01 per default
                                # topic_word_prior = beta

lda.fit(tfidf)

lda.components______no_output_____# show some feature names
tfidf_feature_names = tfidf_vectorizer.get_feature_names()
print(tfidf_feature_names[:200])['00', '000', '01', '10', '100', '1000', '11', '12', '13', '14', '15', '16', '17', '18', '19', '1st', '20', '200', '2000', '2005', '2008', '2009', '2010', '2011', '2012', '2013', '2014', '2015', '2016', '2017', '21', '22', '23', '24', '25', '26', '28', '30', '31', '40', '50', '500', '60', 'abl', 'abov', 'access', 'accid', 'accord', 'account', 'accur', 'accuraci', 'achiev', 'acknowledg', 'acquir', 'act', 'action', 'activ', 'actual', 'ad', 'add', 'addit', 'address', 'adjust', 'administr', 'adult', 'affect', 'age', 'agenc', 'aggreg', 'agre', 'agricultur', 'aim', 'air', 'airport', 'al', 'algorithm', 'allow', 'alreadi', 'alway', 'american', 'analys', 'analysi', 'analyz', 'ani', 'anim', 'annot', 'annual', 'anoth', 'answer', 'api', 'app', 'appear', 'appli', 'applic', 'approach', 'approxim', 'april', 'architectur', 'archiv', 'archive', 'area', 'art', 'articl', 'artist', 'arxiv', 'ask', 'assess', 'assign', 'assist', 'associ', 'attack', 'attempt', 'attribut', 'audio', 'august', 'author', 'automat', 'avail', 'averag', 'award', 'background', 'bank', 'bas', 'base', 'basi', 'basic', 'bea', 'becaus', 'becom', 'befor', 'begin', 'behavior', 'benefit', 'best', 'bet', 'better', 'big', 'bike', 'billion', 'binari', 'bird', 'birth', 'bitcoin', 'block', 'blog', 'blood', 'board', 'bodi', 'book', 'brand', 'build', 'bureau', 'busi', 'ca', 'calcul', 'california', 'campaign', 'cancer', 'candid', 'canon', 'captur', 'car', 'card', 'care', 'case', 'categor', 'categori', 'caus', 'cc', 'cell', 'censu', 'center', 'central', 'certain', 'challeng', 'chang', 'charact', 'characterist', 'check', 'chemic', 'chicago', 'child', 'children', 'citat', 'cite', 'citi', 'claim', 'class', 'classif', 'classifi', 'clean', 'close', 'cloud', 'cluster', 'code', 'collabor', 'collect', 'colleg', 'color', 'column']

## Print out top 20 words
One chooses 20 instead of 10 words for better topic name finding._____no_output_____

for topic_idx, topic in enumerate(lda.components_):
    top_words = [tfidf_feature_names[i] for i in topic.argsort()[:-20-1:-1]]
    print('Topic:',topic_idx,'--',top_words) Topic: 0 -- ['attack', 'pokemon', 'tweet', 'data', 'thi', 'shoot', 'dataset', 'cluster', 'time', 'review', 'kill', 'game', 'star', 'wa', 'pakistan', 'twitter', 'user', 'scrape', 'wine', 'number']
Topic: 1 -- ['song', 'music', 'artist', 'audio', 'frequenc', 'data', 'featur', 'thi', 'rank', 'csv', 'file', 'dataset', 'crime', 'salari', 'explor', 'lot', 'contain', 'api', 'question', 'health']
Topic: 2 -- ['doe', 'descript', 'dataset', 'thi', 'data', 'use', 'context', 'http', 'content', 'wa', 'contain', 'acknowledg', 'file', 'includ', 'inspir', 'time', 'ani', 'inform', 'csv', 'column']
Topic: 3 -- ['que', 'da', 'el', 'en', 'la', 'bea', 'mxbean', 'com', 'i0_lphost06', 'serverruntime', 'jdbcdatasourceruntim', 'type', 'jdbcconnectionpoolruntim', 'ca', 'softwar', 'kaggle', 'numer', 'source09', 'csv', 'data']
Topic: 4 -- ['pokemon', 'strongli', 'disagre', 'news', 'data', 'dataset', 'thi', 'integ', 'articl', 'enjoy', 'includ', 'max', 'com', 'stat', 'use', 'wa', 'elect', 'match', 'set', '2016']
Topic: 5 -- ['employ', 'student', 'data', 'column', 'contain', 'messag', 'convers', 'timestamp', 'job', 'whi', 'health', 'month', 'season', 'line', 'labor', 'sale', 'foreign', 'depart', 'risk', 'set']
Topic: 6 -- ['review', 'hotel', 'data', 'csv', 'survey', '2015', 'dataset', 'leagu', 'neg', 'match', 'year', 'contain', 'transact', 'respons', 'thi', 'includ', 'chicago', 'compani', 'employe', 'posit']
Topic: 7 -- ['dri', 'protein', 'dataset', 'book', 'player', 'thi', 'data', 'anim', '12', 'game', 'polic', 'user', 'csv', 'talk', 'rate', 'contain', 'world', 'file', 'citi', 'origin']
Topic: 8 -- ['imag', 'patient', 'dataset', 'label', 'thi', 'uci', 'data', 'class', 'nation', 'use', 'pixel', 'speci', 'tree', 'learn', 'diseas', 'contain', 'machin', 'type', 'repositori', 'energi']
Topic: 9 -- ['custom', 'match', 'line', 'data', 'women', 'shot', 'store', 'valu', 'csv', 'station', 'cancer', 'thi', 'age', 'featur', 'game', 'dataset', 'ha', 'file', 'season', 'rate']

## Evaluate based on perplexity_____no_output_____

pscores = []
# n_topics = range(1, 31) #used for testing single outcomes. 5 steps are sufficient
for n_topic in [2, 3, 4, 5, 10, 15, 20, 30]:
    lda = LatentDirichletAllocation(n_components=n_topic, max_iter=5,random_state=7)

    lda.fit(tfidf)

    perplexity_score = lda.perplexity(tfidf)
    print("Number of topics: " + str(n_topic) + ", perplexity score: " + str(perplexity_score))
    pscores.append(perplexity_score)

# pscores
# perplexity score of 0 is best valueNumber of topics: 2, perplexity score: 848.4649093189715
Number of topics: 3, perplexity score: 966.7378633150506
Number of topics: 4, perplexity score: 1109.9347705177481
Number of topics: 5, perplexity score: 1261.452097453748
Number of topics: 10, perplexity score: 1658.5960443106096
Number of topics: 15, perplexity score: 1873.534299970601
Number of topics: 20, perplexity score: 2414.0302126421866
Number of topics: 30, perplexity score: 3550.810797868119
## plot the perplexity score with n_topics
import matplotlib.pylab as plt
plt.plot([2, 3, 4, 5, 10, 15, 20, 30],pscores,'r+--')
plt.xlabel('# of topics')
plt.ylabel('Perplexity score')
plt.show()_____no_output_____

**Intepretation: up to 20 topics are reasonable, but no more. We use 5 topics as a heuristic and try to determine the topic names.**_____no_output_____

# perform LDA
lda_final = LatentDirichletAllocation(n_components=5, max_iter=5,
                                learning_method='online', #
                                learning_offset=50.,
                                random_state=0) # set seed

lda_final.fit(tfidf)

lda_final.components______no_output_____# show some feature names
tfidf_feature_names = tfidf_vectorizer.get_feature_names()
print(tfidf_feature_names[:200])['00', '000', '01', '10', '100', '1000', '11', '12', '13', '14', '15', '16', '17', '18', '19', '1st', '20', '200', '2000', '2005', '2008', '2009', '2010', '2011', '2012', '2013', '2014', '2015', '2016', '2017', '21', '22', '23', '24', '25', '26', '28', '30', '31', '40', '50', '500', '60', 'abl', 'abov', 'access', 'accid', 'accord', 'account', 'accur', 'accuraci', 'achiev', 'acknowledg', 'acquir', 'act', 'action', 'activ', 'actual', 'ad', 'add', 'addit', 'address', 'adjust', 'administr', 'adult', 'affect', 'age', 'agenc', 'aggreg', 'agre', 'agricultur', 'aim', 'air', 'airport', 'al', 'algorithm', 'allow', 'alreadi', 'alway', 'american', 'analys', 'analysi', 'analyz', 'ani', 'anim', 'annot', 'annual', 'anoth', 'answer', 'api', 'app', 'appear', 'appli', 'applic', 'approach', 'approxim', 'april', 'architectur', 'archiv', 'archive', 'area', 'art', 'articl', 'artist', 'arxiv', 'ask', 'assess', 'assign', 'assist', 'associ', 'attack', 'attempt', 'attribut', 'audio', 'august', 'author', 'automat', 'avail', 'averag', 'award', 'background', 'bank', 'bas', 'base', 'basi', 'basic', 'bea', 'becaus', 'becom', 'befor', 'begin', 'behavior', 'benefit', 'best', 'bet', 'better', 'big', 'bike', 'billion', 'binari', 'bird', 'birth', 'bitcoin', 'block', 'blog', 'blood', 'board', 'bodi', 'book', 'brand', 'build', 'bureau', 'busi', 'ca', 'calcul', 'california', 'campaign', 'cancer', 'candid', 'canon', 'captur', 'car', 'card', 'care', 'case', 'categor', 'categori', 'caus', 'cc', 'cell', 'censu', 'center', 'central', 'certain', 'challeng', 'chang', 'charact', 'characterist', 'check', 'chemic', 'chicago', 'child', 'children', 'citat', 'cite', 'citi', 'claim', 'class', 'classif', 'classifi', 'clean', 'close', 'cloud', 'cluster', 'code', 'collabor', 'collect', 'colleg', 'color', 'column']

## Print out top 20 words
One chooses 20 instead of 10 words for better topic name finding._____no_output_____

for topic_idx, topic in enumerate(lda_final.components_):
    top_words = [tfidf_feature_names[i] for i in topic.argsort()[:-20-1:-1]]
    print('Topic:',topic_idx,'--',top_words) Topic: 0 -- ['movi', 'imdb', 'hotel', 'review', 'custom', 'revenu', 'genr', 'chicago', 'given', 'rate', 'content', 'releas', 'dataset', 'vote', 'offic', 'star', 'product', 'shoot', 'year', 'data']
Topic: 1 -- ['doe', 'descript', 'dataset', 'thi', 'data', 'use', 'http', 'context', 'content', 'imag', 'ani', 'contain', 'file', 'acknowledg', 'inspir', 'wa', 'column', 'com', 'time', 'word']
Topic: 2 -- ['data', 'thi', 'state', 'dataset', 'citi', 'year', 'wa', 'inform', 'countri', 'includ', 'use', 'acknowledg', 'content', 'number', 'report', 'context', 'contain', 'time', 'unit', 'file']
Topic: 3 -- ['bike', 'da', 'el', 'que', 'en', 'la', 'station', 'trip', 'com', 'site', 'data', 'csv', 'ca', 'softwar', 'kaggle', 'bea', 'download', 'dataset', 'http', 'base']
Topic: 4 -- ['dri', 'mxbean', 'bea', 'i0_lphost06', 'serverruntime', 'jdbcdatasourceruntim', 'type', 'jdbcconnectionpoolruntim', 'com', 'protein', 'source09', 'wine', '12', 'string', 'default', 'data', 'numer', 'star', 'elect', 'anim']

#### Interpretation of results

Results could show be the following topics:
- Topic 1 (index 0): IMDB Movie Reviews, e. g. article [IMDB Movie Reviews Dataset](https://www.kaggle.com/iarunava/imdb-movie-reviews-dataset)
- Topic 2 (index 1): HttpContext, e. g. article [Difference Between HttpContext.Current.Items and HttpContext.Current.Session in ASP.Net](https://www.c-sharpcorner.com/UploadFile/dacca2/difference-between-httpcontext-current-items-and-httpcontext/)
- Topic 3 (index 2): Open Data & Metadata, e. g. article [What is Metadata and Why is it as Important as Data Itself?](https://www.opendatasoft.com/blog/2016/08/25/what-is-metadata-and-why-is-it-important-data)
- Topic 4 (index 3): Download Datasets , e. g. article [Dataset Download | Kaggle](https://www.kaggle.com/surajdidwania/dataset-download)
- Topic 5 (index 4): Java MBeans, e. g. article [Navigating MBeans](https://docs.oracle.com/cd/E25054_01/web.1111/e13715/nav_edit.htm)_____no_output_____# Outlook / Next possible tasks_____no_output_____One could perform additional or alternative text preprocessing transformations regarding the variable 'Description' such as

- lemmatization (as alternative to stemming)
- remove URLs
- remove punctuation
- remove pattern '\r\n', e. g. using regex

to check improvement on LDA outcome._____no_output_____


# Notebook from gabrielmahia/Mob_AI_le
Path: chapter6/code/torch_nlp.ipynb

<code>
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

torch.manual_seed(1)

######################################################################

word_to_ix = {"hello": 0, "world": 1}
embeds = nn.Embedding(2, 5)  # 2 words in vocab, 5 dimensional embeddings
lookup_tensor = torch.tensor([word_to_ix["hello"]], dtype=torch.long)
hello_embed = embeds(lookup_tensor)
print(hello_embed)

######################################################################
# An Example: N-Gram Language Modeling
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#
# Recall that in an n-gram language model, given a sequence of words
# :math:`w`, we want to compute
#
# .. math::  P(w_i | w_{i-1}, w_{i-2}, \dots, w_{i-n+1} )
#
# Where :math:`w_i` is the ith word of the sequence.
#
# In this example, we will compute the loss function on some training
# examples and update the parameters with backpropagation.
#

CONTEXT_SIZE = 2
EMBEDDING_DIM = 10
# We will use Shakespeare Sonnet 2
test_sentence = """When forty winters shall besiege thy brow,
And dig deep trenches in thy beauty's field,
Thy youth's proud livery so gazed on now,
Will be a totter'd weed of small worth held:
Then being asked, where all thy beauty lies,
Where all the treasure of thy lusty days;
To say, within thine own deep sunken eyes,
Were an all-eating shame, and thriftless praise.
How much more praise deserv'd thy beauty's use,
If thou couldst answer 'This fair child of mine
Shall sum my count, and make my old excuse,'
Proving his beauty by succession thine!
This were to be new made when thou art old,
And see thy blood warm when thou feel'st it cold.""".split()
# we should tokenize the input, but we will ignore that for now
# build a list of tuples.  Each tuple is ([ word_i-2, word_i-1 ], target word)
trigrams = [([test_sentence[i], test_sentence[i + 1]], test_sentence[i + 2])
            for i in range(len(test_sentence) - 2)]
# print the first 3, just so you can see what they look like
print(trigrams[:3])

vocab = set(test_sentence)
word_to_ix = {word: i for i, word in enumerate(vocab)}

class NGramLanguageModeler(nn.Module):

    def __init__(self, vocab_size, embedding_dim, context_size):
        super(NGramLanguageModeler, self).__init__()
        self.embeddings = nn.Embedding(vocab_size, embedding_dim)
        self.linear1 = nn.Linear(context_size * embedding_dim, 128)
        self.linear2 = nn.Linear(128, vocab_size)

    def forward(self, inputs):
        embeds = self.embeddings(inputs).view((1, -1))
        out = F.relu(self.linear1(embeds))
        out = self.linear2(out)
        log_probs = F.log_softmax(out, dim=1)
        return log_probs

losses = []
loss_function = nn.NLLLoss()
model = NGramLanguageModeler(len(vocab), EMBEDDING_DIM, CONTEXT_SIZE)
optimizer = optim.SGD(model.parameters(), lr=0.001)

for epoch in range(10):
    total_loss = 0
    for context, target in trigrams:

        # Step 1. Prepare the inputs to be passed to the model (i.e, turn the words
        # into integer indices and wrap them in tensors)
        context_idxs = torch.tensor([word_to_ix[w] for w in context], dtype=torch.long)

        # Step 2. Recall that torch *accumulates* gradients. Before passing in a
        # new instance, you need to zero out the gradients from the old
        # instance
        model.zero_grad()

        # Step 3. Run the forward pass, getting log probabilities over next
        # words
        log_probs = model(context_idxs)

        # Step 4. Compute your loss function. (Again, Torch wants the target
        # word wrapped in a tensor)
        loss = loss_function(log_probs, torch.tensor([word_to_ix[target]], dtype=torch.long))

        # Step 5. Do the backward pass and update the gradient
        loss.backward()
        optimizer.step()

        # Get the Python number from a 1-element Tensor by calling tensor.item()
        total_loss += loss.item()
    losses.append(total_loss)
print(losses)  # The loss decreased every iteration over the training data!

######################################################################
# Exercise: Computing Word Embeddings: Continuous Bag-of-Words
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#
# The Continuous Bag-of-Words model (CBOW) is frequently used in NLP deep
# learning. It is a model that tries to predict words given the context of
# a few words before and a few words after the target word. This is
# distinct from language modeling, since CBOW is not sequential and does
# not have to be probabilistic. Typcially, CBOW is used to quickly train
# word embeddings, and these embeddings are used to initialize the
# embeddings of some more complicated model. Usually, this is referred to
# as *pretraining embeddings*. It almost always helps performance a couple
# of percent.
#
# The CBOW model is as follows. Given a target word :math:`w_i` and an
# :math:`N` context window on each side, :math:`w_{i-1}, \dots, w_{i-N}`
# and :math:`w_{i+1}, \dots, w_{i+N}`, referring to all context words
# collectively as :math:`C`, CBOW tries to minimize
#
# .. math::  -\log p(w_i | C) = -\log \text{Softmax}(A(\sum_{w \in C} q_w) + b)
#
# where :math:`q_w` is the embedding for word :math:`w`.
#
# Implement this model in Pytorch by filling in the class below. Some
# tips:
#
# * Think about which parameters you need to define.
# * Make sure you know what shape each operation expects. Use .view() if you need to
#   reshape.
#

CONTEXT_SIZE = 2  # 2 words to the left, 2 to the right
raw_text = """We are about to study the idea of a computational process.
Computational processes are abstract beings that inhabit computers.
As they evolve, processes manipulate other abstract things called data.
The evolution of a process is directed by a pattern of rules
called a program. People create programs to direct processes. In effect,
we conjure the spirits of the computer with our spells.""".split()

# By deriving a set from `raw_text`, we deduplicate the array
vocab = set(raw_text)
vocab_size = len(vocab)

word_to_ix = {word: i for i, word in enumerate(vocab)}
data = []
for i in range(2, len(raw_text) - 2):
    context = [raw_text[i - 2], raw_text[i - 1],
               raw_text[i + 1], raw_text[i + 2]]
    target = raw_text[i]
    data.append((context, target))
print(data[:5])

class CBOW(nn.Module):

    def __init__(self):
        pass

    def forward(self, inputs):
        pass

# create your model and train.  here are some functions to help you make
# the data ready for use by your module

def make_context_vector(context, word_to_ix):
    idxs = [word_to_ix[w] for w in context]
    return torch.tensor(idxs, dtype=torch.long)

make_context_vector(data[0][0], word_to_ix)  # exampletensor([[ 0.6614,  0.2669,  0.0617,  0.6213, -0.4519]],
       grad_fn=<EmbeddingBackward>)
[(['When', 'forty'], 'winters'), (['forty', 'winters'], 'shall'), (['winters', 'shall'], 'besiege')]
[521.3004720211029, 518.5697512626648, 515.8587603569031, 513.1664328575134, 510.49080514907837, 507.8325996398926, 505.1908845901489, 502.564080953598, 499.94889760017395, 497.3451099395752]
[(['We', 'are', 'to', 'study'], 'about'), (['are', 'about', 'study', 'the'], 'to'), (['about', 'to', 'the', 'idea'], 'study'), (['to', 'study', 'idea', 'of'], 'the'), (['study', 'the', 'of', 'a'], 'idea')]

</code>


# Notebook from espg/jupyter-earth
Path: docs/jupyter-resources/introduction/ecosystem_bk.ipynb

# Ecosystem

_What do we mean by an "ecosystem" of software projects?_

[Wikipedia](https://en.wikipedia.org/wiki/Ecosystem) defines an ecosystem as

>  a community of living organisms in conjunction with the nonliving components of their environment, interacting as a system

A few key ideas that are common among living ecosystems and an "ecosystem" of software projects are:
- community
- a common environment
- interaction and interoperability
- reliance and interdependence

![python-stack](../images/python-stack.png)_____no_output_____The term “ecosystem” is often used to describe the modern open-source scientific software. In biology, the term “ecosystem” is defined as a biological community of interacting organisms and their physical environment. Modern open-source scientific software development occurs in a similarly interconnected and interoperable fashion. The act of importing a software package signals a reliance of one part of the system on the functionality and role of another. Projects in this ecosystem lie somewhere along the spectrum from “foundational” projects which serve a wide community and are relied on by increasingly specialized projects, which eventually may be specialized enough to serve a specific scientific domain. Figure XX illustrates this concept of a software stack for the scientific python ecosystem. At the base of the stack is the language, in this case Python. Upon this sits the layer of “core” software packages. These packages provide functionality that is generically useful throughout the scientific ecosystem; for example Numpy for n-dimensional arrays {cite}`Harris2020`, Matplotlib for plotting {cite}`Hunter2007`, Pandas for data frames {cite}`reback2020pandas,mckinney-proc-scipy-2010`, and Jupyter for interactive computing {cite}`Perez2007`. The next layer consists of specialized projects such as Scikit-learn for machine learning {cite}`scikit-learn` or Dask for parallel computation {cite}`DaskDevelopmentTeam2016`; these projects are relied on by others in the ecosystem to solve a general suite of computational tasks. The top layer is that of domain-specific projects, which are tailored to serve a distinct community of researchers and the questions they are pursuing. There are not necessarily hard lines between these layers, as projects mature, they may move between these layers, and some communities might organize projects differently._____no_output_____```{bibliography} ../../bibliography.bib
```_____no_output_____