# Notebook from nealcaren/Text-Mining-with-Python-Notebooks
Path: Notebooks/Uncategorized/Scrape Wikipedia.ipynb

<code>
from requests_html import HTMLSession

import pandas

def get_text(parsed_html):
    article_text = ''
    for paragraph in parsed_html.find('p'):
        new_paragraph = paragraph.text
        article_text  = article_text + '\n' + new_paragraph
    return article_text

def get_headline(parsed_html):
    headline = parsed_html.find('h1')[0].text
    return headline

def scrape_wikipedia(url):
    '''Scrape a Wikipedia page and return the headline and text.'''

    # gets the article
    session = HTMLSession()
    r = session.get(url)

    # parses the html
    parsed_html = r.html

    # grabs the headline
    headline = get_headline(parsed_html)

    # grabs the text
    text = get_text(parsed_html)

    # stores headline and text in a dictionary
    results = {'headline' : headline,
               'text'     : text,
               'url'      : url}

    # send it back
    return results
_____no_output_____scrape_wikipedia('https://en.wikipedia.org/wiki/Sociocultural_evolution')
Sociocultural evolution, sociocultural evolutionism or cultural evolution are theories of cultural and social evolution that describe how cultures and societies change over time. Whereas sociocultural development traces processes that tend to increase the complexity of a society or culture, sociocultural evolution also considers process that can lead to decreases in complexity (degeneration) or that can produce variation or proliferation without any seemingly significant changes in complexity (cladogenesis).[1] Sociocultural evolution is "the process by which structural reorganization is affected through time, eventually producing a form or structure which is qualitatively different from the ancestral form".[2]
Most of the 19th-century and some 20th-century approaches to socioculture aimed to provide models for the evolution of humankind as a whole, arguing that different societies have reached different stages of social development. The most comprehensive attempt to develop a general theory of social evolution centering on the development of sociocultural systems, the work of Talcott Parsons (1902–1979), operated on a scale which included a theory of world history. Another attempt, on a less systematic scale, originated with the world-systems approach from the 1970s.
More recent approaches focus on changes specific to individual societies and reject the idea that cultures differ primarily according to how far each one is on some linear scale of social progress. Most modern archaeologists and cultural anthropologists work within the frameworks of neoevolutionism, sociobiology, and modernization theory.
Many different societies have existed in the course of human history, with estimates as high as a total of over one million separate societies; however, as of 2013[update], the number of current, distinct societies had been estimated as only about two hundred.[3]
Anthropologists and sociologists often assume that human beings have natural social tendencies and that particular human social behaviours have non-genetic causes and dynamics (i.e. people learn them in a social environment and through social interaction). Societies exist in complex social environments (i.e. with natural resources and constraints) and adapt themselves to these environments. It is thus inevitable that all societies change.
Specific theories of social or cultural evolution often attempt to explain differences between coeval societies by positing that different societies have reached different stages of development. Although such theories typically provide models for understanding the relationship between technologies, social structure or the values of a society, they vary as to the extent to which they describe specific mechanisms of variation and change.
Early sociocultural evolution theories – the ideas of Auguste Comte (1798–1857), Herbert Spencer (1820–1903) and Lewis Henry Morgan (1818–1881)  – developed simultaneously with, but independently of, Charles Darwin's works and were popular from the late 19th century to the end of World War I. These 19th-century unilineal evolution theories claimed that societies start out in a primitive state and gradually become more civilized over time; they equated the culture and technology of Western civilization with progress. Some forms of early sociocultural evolution theories (mainly unilineal ones) have led to much-criticised theories like social Darwinism and scientific racism, sometimes used in the past[by whom?] to justify existing policies of colonialism and slavery and to justify new policies such as eugenics.
Most 19th-century and some 20th-century approaches aimed to provide models for the evolution of humankind as a single entity. However, most 20th-century approaches, such as multilineal evolution, focused on changes specific to individual societies. Moreover, they rejected directional change (i.e. orthogenetic, teleological or progressive change). Most archaeologists work within the framework of multilineal evolution. Other contemporary approaches to social change include neoevolutionism, sociobiology, dual inheritance theory, modernisation theory and postindustrial theory.
In his seminal 1976 book The Selfish Gene, Richard Dawkins wrote that "there are some examples of cultural evolution in birds and monkeys, but ... it is our own species that really shows what cultural evolution can do".[4]
Enlightenment and later thinkers often speculated that societies progressed through stages: in other words, they saw history as stadial. While expecting humankind to show increasing development, theorists looked for what determined the course of human history. Georg Wilhelm Friedrich Hegel (1770–1831), for example, saw social development as an inevitable process.[citation needed] It was assumed that societies start out primitive, perhaps in a state of nature, and could progress toward something resembling industrial Europe.
While earlier authors such as Michel de Montaigne (1533–1592) had discussed how societies change through time, the Scottish Enlightenment of the 18th century proved key in the development of the idea of sociocultural evolution.[citation needed] In relation to Scotland's union with England in 1707, several[quantify] Scottish thinkers pondered the relationship between progress and the affluence brought about by increased trade with England. They understood the changes Scotland was undergoing as involving transition from an agricultural to a mercantile society. In "conjectural histories", authors such as Adam Ferguson (1723–1816), John Millar (1735–1801) and Adam Smith (1723–1790) argued that societies all pass through a series of four stages: hunting and gathering, pastoralism and nomadism, agriculture, and finally a stage of commerce.
Philosophical concepts of progress, such as that of Hegel, developed as well during this period. In France, authors such as Claude Adrien Helvétius (1715–1771) and other philosophes were influenced by the Scottish tradition. Later thinkers such as Comte de Saint-Simon (1760–1825) developed these ideas.[citation needed] Auguste Comte (1798–1857) in particular presented a coherent view of social progress and a new discipline to study it: sociology.
These developments took place in a context of wider processes. The first process was colonialism. Although imperial powers settled most differences of opinion with their colonial subjects through force, increased awareness of non-Western peoples raised new questions for European scholars about the nature of society and of culture. Similarly, effective colonial administration required some degree of understanding of other cultures. Emerging theories of sociocultural evolution allowed Europeans to organise their new knowledge in a way that reflected and justified their increasing political and economic domination of others: such systems saw colonised people as less evolved, and colonising people as more evolved. Modern civilization (understood as the Western civilization), appeared the result of steady progress from a state of barbarism, and such a notion was common to many thinkers of the Enlightenment, including Voltaire (1694–1778).
The second process was the Industrial Revolution and the rise of capitalism, which together allowed and promoted continual revolutions in the means of production. Emerging theories of sociocultural evolution reflected a belief that the changes in Europe brought by the Industrial Revolution and capitalism were improvements. Industrialisation, combined with the intense political change brought about by the French Revolution of 1789 and the U.S. Constitution, which paved the way for the dominance of democracy, forced European thinkers to reconsider some of their assumptions about how society was organised.
Eventually, in the 19th century three major classical theories of social and historical change emerged:
These theories had a common factor: they all agreed that the history of humanity is pursuing a certain fixed path, most likely that of social progress. Thus, each past event is not only chronologically, but causally tied to present and future events. The theories postulated that by recreating the sequence of those events, sociology could discover the "laws" of history.[5]
While sociocultural evolutionists agree that an evolution-like process leads to social progress, classical social evolutionists have developed many different theories, known as theories of unilineal evolution. Sociocultural evolutionism became the prevailing theory of early sociocultural anthropology and social commentary, and is associated with scholars like Auguste Comte, Edward Burnett Tylor, Lewis Henry Morgan, Benjamin Kidd, L. T. Hobhouse and Herbert Spencer. Sociocultural evolutionism attempted to formalise social thinking along scientific lines, with the added influence from the biological theory of evolution. If organisms could develop over time according to discernible, deterministic laws, then it seemed reasonable that societies could as well. Human society was compared to a biological organism, and social science equivalents of concepts like variation, natural selection, and inheritance were introduced as factors resulting in the progress of societies. The idea of progress led to that of a fixed "stages" through which human societies progress, usually numbering three – savagery, barbarism, and civilization – but sometimes many more. As early as the late 18th century, the Marquis de Condorcet (1743–1794) listed ten stages, or "epochs", each advancing the rights of man and perfecting the human race. At that time, anthropology was rising as a new scientific discipline, separating from the traditional views of "primitive" cultures that was usually based on religious views.[citation needed]
Classical social evolutionism is most closely associated with the 19th-century writings of Auguste Comte and of Herbert Spencer (coiner of the phrase "survival of the fittest").[6] In many ways, Spencer's theory of "cosmic evolution" has much more in common with the works of Jean-Baptiste Lamarck and Auguste Comte than with contemporary works of Charles Darwin. Spencer also developed and published his theories several years earlier than Darwin. In regard to social institutions, however, there is a good case that Spencer's writings might be classified as social evolutionism. Although he wrote that societies over time progressed – and that progress was accomplished through competition – he stressed that the individual rather than the collectivity is the unit of analysis that evolves; that, in other words, evolution takes place through natural selection and that it affects social as well as biological phenomenon. Nonetheless, the publication of Darwin's works[which?] proved a boon to the proponents of sociocultural evolution, who saw the ideas of biological evolution as an attractive explanation for many questions about the development of society.[7]
Both Spencer and Comte view society as a kind of organism subject to the process of growth—from simplicity to complexity, from chaos to order, from generalisation to specialisation, from flexibility to organisation. They agree that the process of societal growth can be divided into certain stages, have[clarification needed] their beginning and eventual end, and that this growth is in fact social progress: each newer, more-evolved society is "better". Thus progressivism became one of the basic ideas underlying the theory of sociocultural evolutionism.[6]
Auguste Comte, known as "the father of sociology", formulated the law of three stages: human development progresses from the theological stage, in which nature was mythically conceived and man sought the explanation of natural phenomena from supernatural beings; through a metaphysical stage in which nature was conceived of as a result of obscure forces and man sought the explanation of natural phenomena from them; until the final positive stage in which all abstract and obscure forces are discarded, and natural phenomena are explained by their constant relationship.[8] This progress is forced through the development of human mind, and through increasing application of thought, reasoning and logic to the understanding of the world.[9] Comte saw the science-valuing society as the highest, most developed type of human organization.[8]
Herbert Spencer, who argued against government intervention as he believed that society should evolve toward more individual freedom,[10] differentiated between two phases of development as regards societies' internal regulation:[8] the "military" and "industrial" societies.[8] The earlier (and more primitive) military society has the goal of conquest and defense, is centralised, economically self-sufficient, collectivistic, puts the good of a group over the good of an individual, uses compulsion, force and repression, and rewards loyalty, obedience and discipline.[8] The industrial society, in contrast, has a goal of production and trade, is decentralised, interconnected with other societies via economic relations, works through voluntary cooperation and individual self-restraint, treats the good of individual as of the highest value, regulates the social life via voluntary relations; and values initiative, independence and innovation.[8][11] The transition process from the military to industrial society is the outcome of steady evolutionary processes within the society.[8]
Regardless of how scholars of Spencer interpret his relation to Darwin, Spencer became an incredibly popular figure in the 1870s, particularly in the United States. Authors such as Edward L. Youmans, William Graham Sumner, John Fiske, John W. Burgess, Lester Frank Ward, Lewis H. Morgan (1818–1881) and other thinkers of the gilded age all developed theories of social evolutionism as a result of their exposure to Spencer as well as to Darwin.
In his 1877 classic Ancient Societies, Lewis H. Morgan, an anthropologist whose ideas have had much impact on sociology, differentiated between three eras: savagery, barbarism and civilization, which are divided by technological inventions, like fire, bow, pottery in the savage era, domestication of animals, agriculture, metalworking in the barbarian era and alphabet and writing in the civilization era.[12] Thus Morgan drew a link between social progress and technological progress. Morgan viewed technological progress as a force behind social progress, and held that any social change—in social institutions, organizations or ideologies—has its beginnings in technological change.[12][13] Morgan's theories were popularized by Friedrich Engels, who based his famous work The Origin of the Family, Private Property and the State on them.[12] For Engels and other Marxists this theory was important, as it supported their conviction that materialistic factors—economic and technological—are decisive in shaping the fate of humanity.[12]
Edward Burnett Tylor (1832–1917), a pioneer of anthropology, focused on the evolution of culture worldwide, noting that culture is an important part of every society and that it is also subject to a process of evolution. He believed that societies were at different stages of cultural development and that the purpose of anthropology was to reconstruct the evolution of culture, from primitive beginnings to the modern state.
Anthropologists Sir E.B. Tylor in England and Lewis Henry Morgan in the United States worked with data from indigenous people, who (they claimed) represented earlier stages of cultural evolution that gave insight into the process and progression of evolution of culture. Morgan would later[when?] have a significant influence on Karl Marx and on Friedrich Engels, who developed a theory of sociocultural evolution in which the internal contradictions in society generated a series of escalating stages that ended in a socialist society (see Marxism). Tylor and Morgan elaborated the theory of unilinear evolution, specifying criteria for categorising cultures according to their standing within a fixed system of growth of humanity as a whole and examining the modes and mechanisms of this growth. Theirs was often a concern with culture in general, not with individual cultures.
Their analysis of cross-cultural data was based on three assumptions:
Theorists usually measured progression (that is, the difference between one stage and the next) in terms of increasing social complexity (including class differentiation and a complex division of labour), or an increase in intellectual, theological, and aesthetic sophistication. These 19th-century ethnologists used these principles primarily to explain differences in religious beliefs and kinship terminologies among various societies.
Lester Frank Ward (1841–1913), sometimes referred to[by whom?] as the "father" of American sociology, rejected many of Spencer's theories regarding the evolution of societies. Ward, who was also a botanist and a paleontologist, believed that the law of evolution functioned much differently in human societies than it did in the plant and animal kingdoms, and theorized that the "law of nature" had been superseded by the "law of the mind".[14] He stressed that humans, driven by emotions, create goals for themselves and strive to realize them (most effectively with the modern scientific method) whereas there is no such intelligence and awareness guiding the non-human world.[15] Plants and animals adapt to nature; man shapes nature. While Spencer believed that competition and "survival of the fittest" benefited human society and sociocultural evolution, Ward regarded competition as a destructive force, pointing out that all human institutions, traditions and laws were tools invented by the mind of man and that that mind designed them, like all tools, to "meet and checkmate" the unrestrained competition of natural forces.[14] Ward agreed with Spencer that authoritarian governments repress the talents of the individual, but he believed that modern democratic societies, which minimized the role of religion and maximized that of science, could effectively support the individual in his or her attempt to fully utilize their talents and achieve happiness. He believed that the evolutionary processes have four stages:
While Ward regarded modern societies as superior to "primitive" societies (one need only look to the impact of medical science on health and lifespan[citation needed]) he rejected theories of white supremacy; he supported the Out-of-Africa theory of human evolution and believed that all races and social classes were equal in talent.[16] However, Ward did not think that evolutionary progress was inevitable and he feared the degeneration of societies and cultures, which he saw as very evident in the historical record.[17] Ward also did not favor the radical reshaping of society as proposed by the supporters of the eugenics movement or by the followers of Karl Marx; like Comte, Ward believed that sociology was the most complex of the sciences and that true sociogenesis was impossible without considerable research and experimentation.[16]
Émile Durkheim, another of the "fathers" of sociology, developed a dichotomal view of social progress.[18] His key concept was social solidarity, as he defined social evolution in terms of progressing from mechanical solidarity to organic solidarity.[18] In mechanical solidarity, people are self-sufficient, there is little integration and thus there is the need for the use of force and repression to keep society together.[18] In organic solidarity, people are much more integrated and interdependent and specialisation and cooperation are extensive.[18] Progress from mechanical to organic solidarity is based firstly on population growth and increasing population density, secondly on increasing "morality density" (development of more complex social interactions) and thirdly on increasing specialisation in the workplace.[18] To Durkheim, the most important factor in social progress is the division of labour.[18] This[clarification needed] was later used in the mid-1900s by the economist Ester Boserup (1910–1999) to attempt to discount some aspects of Malthusian theory.
Ferdinand Tönnies (1855–1936) describes evolution as the development from informal society, where people have many liberties and there are few laws and obligations, to modern, formal rational society, dominated by traditions and laws, where people are restricted from acting as they wish.[19] He also notes that there is a tendency to standardisation and unification, when all smaller societies are absorbed into a single, large, modern society.[19] Thus Tönnies can be said to describe part of the process known today as globalization. Tönnies was also one of the first sociologists to claim that the evolution of society is not necessarily going in the right direction, that social progress is not perfect, and it can even be called a regression as the newer, more evolved societies are obtained only after paying a high cost, resulting in decreasing satisfaction of the individuals making up that society.[19] Tönnies' work became the foundation of neoevolutionism.[19]
Although Max Weber is not usually counted[by whom?] as a sociocultural evolutionist, his theory of tripartite classification of authority can be viewed[by whom?] as an evolutionary theory as well. Weber distinguishes three ideal types of political leadership, domination and authority:
Weber also notes that legal domination is the most advanced, and that societies evolve from having mostly traditional and charismatic authorities to mostly rational and legal ones.
The early 20th-century inaugurated a period of systematic critical examination, and rejection of the sweeping generalisations of the unilineal theories of sociocultural evolution. Cultural anthropologists such as Franz Boas (1858–1942), along with his students, including Ruth Benedict and Margaret Mead, are regarded[by whom?] as the leaders of anthropology's rejection of classical social evolutionism.
They used sophisticated ethnography and more rigorous empirical methods to argue that Spencer, Tylor, and Morgan's theories were speculative and systematically misrepresented ethnographic data. Theories regarding "stages" of evolution were especially criticised as illusions. Additionally, they rejected the distinction between "primitive" and "civilized" (or "modern"), pointing out that so-called primitive contemporary societies have just as much history, and were just as evolved, as so-called civilized societies. They therefore argued that any attempt to use this theory to reconstruct the histories of non-literate (i.e. leaving no historical documents) peoples is entirely speculative and unscientific.
They observed that the postulated progression, which typically ended with a stage of civilization identical to that of modern Europe, is ethnocentric. They also pointed out that the theory assumes that societies are clearly bounded and distinct, when in fact cultural traits and forms often cross social boundaries and diffuse among many different societies (and are thus an important mechanism of change). Boas in his culture-history approach focused on anthropological fieldwork in an attempt to identify factual processes instead of what he criticized as speculative stages of growth. His approach greatly influenced American anthropology in the first half of the 20th century, and marked a retreat from high-level generalization and from "systems building".
Later critics observed that the assumption of firmly bounded societies was proposed precisely at the time when European powers were colonising non-Western societies, and was thus self-serving. Many anthropologists and social theorists now consider unilineal cultural and social evolution a Western myth seldom based on solid empirical grounds. Critical theorists argue that notions of social evolution are simply justifications for power by the élites of society. Finally, the devastating World Wars that occurred between 1914 and 1945 crippled Europe's self-confidence. After millions of deaths, genocide, and the destruction of Europe's industrial infrastructure, the idea of progress seemed dubious at best.
Thus modern sociocultural evolutionism rejects most of classical social evolutionism due to various theoretical problems:
Because social evolution was posited as a scientific theory, it was often used to support unjust and often racist social practices — particularly colonialism, slavery, and the unequal economic conditions present within industrialized Europe. Social Darwinism is especially criticised, as it purportedly led to some philosophies used by the Nazis.
Weber's major works in economic sociology and the sociology of religion dealt with the rationalization, secularisation, and so called "disenchantment" which he associated with the rise of capitalism and modernity.[20] In sociology, rationalization is the process whereby an increasing number of social actions become based on considerations of teleological efficiency or calculation rather than on motivations derived from morality, emotion, custom, or tradition. Rather than referring to what is genuinely "rational" or "logical", rationalization refers to a relentless quest for goals that might actually function to the detriment of a society. Rationalization is an ambivalent aspect of modernity, manifested especially in Western society – as a behaviour of the capitalist market, of rational administration in the state and bureaucracy, of the extension of modern science, and of the expansion of modern technology.[citation needed]
Weber's thought regarding the rationalizing and secularizing tendencies of modern Western society (sometimes described as the "Weber Thesis") would blend with Marxism to facilitate critical theory, particularly in the work of thinkers such as Jürgen Habermas (born 1929). Critical theorists, as antipositivists, are critical of the idea of a hierarchy of sciences or societies, particularly with respect to the sociological positivism originally set forth by Comte. Jürgen Habermas has critiqued the concept of pure instrumental rationality as meaning that scientific-thinking becomes something akin to ideology itself. For theorists such as Zygmunt Bauman (1925–2017), rationalization as a manifestation of modernity may be most closely and regrettably associated with the events of the Holocaust.
When the critique of classical social evolutionism became widely accepted, modern anthropological and sociological approaches changed respectively. Modern theories are careful to avoid unsourced, ethnocentric speculation, comparisons, or value judgments; more or less regarding individual societies as existing within their own historical contexts. These conditions provided the context for new theories such as cultural relativism and multilineal evolution.
In the 1920s and 1930s, Gordon Childe revolutionized the study of cultural evolutionism. He conducted a comprehensive pre-history account that provided scholars with evidence for African and Asian cultural transmission into Europe. He combated scientific racism by finding the tools and artifacts of the indigenous people from Africa and Asia and showed how they influenced the technology of European culture. Evidence from his excavations countered the idea of Aryan supremacy and superiority. Childe explained cultural evolution by his theory of divergence with modifications of convergence. He postulated that different cultures form separate methods that meet different needs, but when two cultures were in contact they developed similar adaptations, solving similar problems. Rejecting Spencer’s theory of parallel cultural evolution, Childe found that interactions between cultures contributed to the convergence of similar aspects most often attributed to one culture. Childe placed emphasis on human culture as a social construct rather than products of environmental or technological contexts. Childe coined the terms "Neolithic Revolution", and "Urban Revolution" which are still used today in the branch of pre-historic anthropology.
In 1941 anthropologist Robert Redfield wrote about a shift from ‘folk society’ to ‘urban society’. By the 1940s cultural anthropologists such as Leslie White and Julian Steward sought to revive an evolutionary model on a more scientific basis, and succeeded in establishing an approach known as neoevolutionism. White rejected the opposition between "primitive" and "modern" societies but did argue that societies could be distinguished based on the amount of energy they harnessed, and that increased energy allowed for greater social differentiation (White’s law). Steward on the other hand rejected the 19th-century notion of progress, and instead called attention to the Darwinian notion of "adaptation", arguing that all societies had to adapt to their environment in some way.
The anthropologists Marshall Sahlins and Elman Service prepared an edited volume, Evolution and Culture, in which they attempted to synthesise White’s and Steward’s approaches.[21] Other anthropologists, building on or responding to work by White and Steward, developed theories of cultural ecology and ecological anthropology. The most prominent examples are Peter Vayda and Roy Rappaport. By the late 1950s, students of Steward such as Eric Wolf and Sidney Mintz turned away from cultural ecology to Marxism, World Systems Theory, Dependency theory and Marvin Harris’s Cultural materialism.
Today most anthropologists reject 19th-century notions of progress and the three assumptions of unilineal evolution. Following Steward, they take seriously the relationship between a culture and its environment to explain different aspects of a culture. But most modern cultural anthropologists have adopted a general systems approach, examining cultures as emergent systems and arguing that one must consider the whole social environment, which includes political and economic relations among cultures. As a result of simplistic notions of "progressive evolution", more modern, complex cultural evolution theories (such as Dual Inheritance Theory, discussed below) receive little attention in the social sciences, having given way in some cases to a series of more humanist approaches. Some reject the entirety of evolutionary thinking and look instead at historical contingencies, contacts with other cultures, and the operation of cultural symbol systems. In the area of development studies, authors such as Amartya Sen have developed an understanding of ‘development’ and ‘human flourishing’ that also question more simplistic notions of progress, while retaining much of their original inspiration.
Neoevolutionism was the first in a series of modern multilineal evolution theories. It emerged in the 1930s and extensively developed in the period following the Second World War and was incorporated into both anthropology and sociology in the 1960s. It bases its theories on empirical evidence from areas of archaeology, palaeontology, and historiography and tries to eliminate any references to systems of values, be it moral or cultural, instead trying to remain objective and simply descriptive.[22]
While 19th-century evolutionism explained how culture develops by giving general principles of its evolutionary process, it was dismissed by the Historical Particularists as unscientific in the early 20th century. It was the neo-evolutionary thinkers who brought back evolutionary thought and developed it to be acceptable to contemporary anthropology.
Neo-evolutionism discards many ideas of classical social evolutionism, namely that of social progress, so dominant in previous sociology evolution-related theories.[22] Then neo-evolutionism discards the determinism argument and introduces probability, arguing that accidents and free will greatly affect the process of social evolution.[22] It also supports counterfactual history—asking "what if" and considering different possible paths that social evolution may take or might have taken, and thus allows for the fact that various cultures may develop in different ways, some skipping entire stages others have passed through.[22] Neo-evolutionism stresses the importance of empirical evidence. While 19th-century evolutionism used value judgments and assumptions for interpreting data, neo-evolutionism relies on measurable information for analysing the process of sociocultural evolution.
Leslie White, author of The Evolution of Culture: The Development of Civilization to the Fall of Rome (1959), attempted to create a theory explaining the entire history of humanity.[22] The most important factor in his theory is technology.[22] Social systems are determined by technological systems, wrote White in his book,[23] echoing the earlier theory of Lewis Henry Morgan. He proposes a society’s energy consumption as a measure of its advancement.[22] He differentiates between five stages of human development.[22] In the first, people use the energy of their own muscles.[22] In the second, they use the energy of domesticated animals.[22] In the third, they use the energy of plants (so White refers to agricultural revolution here).[22] In the fourth, they learn to use the energy of natural resources: coal, oil, gas.[22] In the fifth, they harness nuclear energy.[22] White introduced a formula, P=E·T, where E is a measure of energy consumed, and T is the measure of efficiency of technical factors utilising the energy.[22] This theory is similar to Russian astronomer Nikolai Kardashev’s later theory of the Kardashev scale.
Julian Steward, author of Theory of Culture Change: The Methodology of Multilinear Evolution (1955, reprinted 1979), created the theory of "multilinear" evolution which examined the way in which societies adapted to their environment. This approach was more nuanced than White’s theory of "unilinear evolution." Steward rejected the 19th-century notion of progress, and instead called attention to the Darwinian notion of "adaptation", arguing that all societies had to adapt to their environment in some way. He argued that different adaptations could be studied through the examination of the specific resources a society exploited, the technology the society relied on to exploit these resources, and the organization of human labour. He further argued that different environments and technologies would require different kinds of adaptations, and that as the resource base or technology changed, so too would a culture. In other words, cultures do not change according to some inner logic, but rather in terms of a changing relationship with a changing environment. Cultures therefore would not pass through the same stages in the same order as they changed—rather, they would change in varying ways and directions. He called his theory "multilineal evolution". He questioned the possibility of creating a social theory encompassing the entire evolution of humanity; however, he argued that anthropologists are not limited to describing specific existing cultures. He believed that it is possible to create theories analysing typical common culture, representative of specific eras or regions. As the decisive factors determining the development of given culture he pointed to technology and economics, but noted that there are secondary factors, like political system, ideologies and religion. All those factors push the evolution of a given society in several directions at the same time; hence the application of the term "multilinear" to his theory of evolution.
Marshall Sahlins, co-editor with Elman Service of Evolution and Culture (1960), divided the evolution of societies into ‘general’ and ‘specific’.[24] General evolution is the tendency of cultural and social systems to increase in complexity, organization and adaptiveness to environment.[24] However, as the various cultures are not isolated, there is interaction and a diffusion of their qualities (like technological inventions).[24] This leads cultures to develop in different ways (specific evolution), as various elements are introduced to them in different combinations and at different stages of evolution.[24]
In his Power and Prestige (1966) and Human Societies: An Introduction to Macrosociology (1974), Gerhard Lenski expands on the works of Leslie White and Lewis Henry Morgan,[24] developing the ecological-evolutionary theory. He views technological progress as the most basic factor in the evolution of societies and cultures.[24] Unlike White, who defined technology as the ability to create and utilise energy, Lenski focuses on information—its amount and uses.[24] The more information and knowledge (especially allowing the shaping of natural environment) a given society has, the more advanced it is.[24] He distinguishes four stages of human development, based on advances in the history of communication.[24] In the first stage, information is passed by genes.[24] In the second, when humans gain sentience, they can learn and pass information through by experience.[24] In the third, humans start using signs and develop logic.[24] In the fourth, they can create symbols and develop language and writing.[24] Advancements in the technology of communication translate into advancements in the economic system and political system, distribution of goods, social inequality and other spheres of social life. He also differentiates societies based on their level of technology, communication and economy: (1) hunters and gatherers, (2) agricultural, (3) industrial, and (4) special (like fishing societies).[24]
Talcott Parsons, author of Societies: Evolutionary and Comparative Perspectives (1966) and The System of Modern Societies (1971) divided evolution into four subprocesses: (1) division, which creates functional subsystems from the main system; (2) adaptation, where those systems evolve into more efficient versions; (3) inclusion of elements previously excluded from the given systems; and (4) generalization of values, increasing the legitimization of the ever more complex system.[25] He shows those processes on 4 stages of evolution: (I) primitive or foraging, (II) archaic agricultural, (III) classical or "historic" in his terminology, using formalized and universalizing theories about reality and (IV) modern empirical cultures. However, these divisions in Parsons’ theory are the more formal ways in which the evolutionary process is conceptualized, and should not be mistaken for Parsons’ actual theory. Parsons develops a theory where he tries to reveal the complexity of the processes which take form between two points of necessity, the first being the cultural "necessity," which is given through the values-system of each evolving community; the other is the environmental necessities, which most directly is reflected in the material realities of the basic production system and in the relative capacity of each industrial-economical level at each window of time. Generally, Parsons highlights that the dynamics and directions of these processes is shaped by the cultural imperative embodied in the cultural heritage, and more secondarily, an outcome of sheer "economic" conditions.
Michel Foucault’s recent, and very much misunderstood, concepts such as Biopower, Biopolitics and Power-knowledge has been cited as breaking free from the traditional conception of man as cultural animal. Foucault regards both the terms “cultural animal” and "human nature"as misleading abstractions, leading to a non critical exemption of man and anything can be justified when regarding social processes or natural phenomena (social phenomena).[26] Foucault argues these complex processes are interrelated, and difficult to study for a reason so those 'truths' cannot be topled or disrupted. For Foucault, the many modern concepts and practices that attempt to uncover “the truth” about human beings (either psychologically, sexually, religion or spiritually) actually create the very types of people they purport to discover. Requiring trained "specialists" and knowledge codes and know how, rigorous pursuit is "put off" or delayed which makes any kind of study not only a ‘taboo’ subject but deliberately ignored. He cites the concept of ‘truth’[27] within many human cultures and the ever flowing dynamics between truth, power, and knowledge as a resultant complex dynamics (Foucault uses the term regimes of truth) and how they flow with ease like water which make the concept of ‘truth’ impervious to any further rational investigation. Some of the West’s most powerful social institutions are powerful for a reason, not because they exhibit powerful structures which inhibit investigation or it is illegal to investigate there historical foundation. It is the very notion of "legitimacy" Foucault cites as examples of "truth" which function as a "Foundationalism" claims to historical accuracy. Foucault argues, systems such as Medicine, Prisons,[28][29] and Religion, as well as groundbreaking works on more abstract theoretical issues of power are suspended or buried into oblivion.[30] He cites as further examples the ‘Scientific study’ of Population biology and Population genetics[31] as both examples of this kind of “Biopower” over the vast majority of the human population giving the new founded political population their ‘politics’ or polity. With the advent of biology and genetics teamed together as new scientific innovations notions of study of knowledge regarding truth belong to the realm of experts who will never divulge their secrets openly, while the bulk of the population do not know their own biology or genetics this is done for them by the experts. This functions as a truth ignorance mechanism: “where the “subjugated knowledge’s,” as those that have been both written out of history and submerged in it in a masked form produces what we now know as truth. He calls them “Knowledge’s from below” and a “historical knowledge of struggles”.Genealogy, Foucault suggests, is a way of getting at these knowledge’s and struggles; “they are about the insurrection of knowledge’s.”Foucault tries to show with the added dimension of “Milieu”(derived from Newtonian mechanics) how this Milieu from the 17th century with the development of the Biological and Physical sciences managed to be interwoven into the political, social and biological relationship of men with the arrival of the concept Work placed upon the industrial population. Foucault uses the term Umwelt, borrowed from Jakob von Uexküll, meaning environment within. Technology, production, cartography the production of Nation states and Government making the efficiency of the Body politic, Law, Heredity and Consanguine[30] not only sound genuine and beyond historical origin and foundation it can be turned into ‘exact truth’ where the individual and the societal body are not only subjugated and nullified but dependent upon it. Foucault is not denying that genetic or biological study is inaccurate or is simply not telling the truth what he means is that notions of this newly discovered sciences were extended to include the vast majority (or whole populations) of populations as an exercise in “regimes change”.Foucault argues that the conceptual meaning from the Middle ages and Canon law period, the Geocentric model, later superseded by the Heliocentrism model placing the position of the law of right in the Middle ages (Exclusive right or its correct legal term Sui generis) was the Divine right of kings and Absolute monarchy where the previous incarnation of truth and rule of political sovereignty was considered absolute and unquestioned by political philosophy (monarchs, popes and emperors). However, Foucault noticed that this Pharaonic version of political power was transversed and it was with 18th-century emergence of Capitalism and Liberal democracy that these terms began to be “democratized”. The modern Pharaonic version represented by the President, the monarch, the Pope and the Prime minister all became propagandized versions or examples of symbol agents all aimed at towards a newly discovered phenomenon, the population.[32] As symbolic symbol agents of power making the mass population having to sacrifice itself all in the name of the newly formed voting franchise we now call Democracy. However, this was all turned on its head (when the Medieval rulers were thrown out and replaced by a more exact apparatus now called the state) when the human sciences suddenly discovered: “The set of mechanisms through which the basic biological features of the human species became an object of a political strategy and took on board the fundamental facts that humans were now a biological species."[33]
Sociobiology departs perhaps the furthest from classical social evolutionism.[34] It was introduced by Edward Wilson in his 1975 book Sociobiology: The New Synthesis and followed his adaptation of evolutionary theory to the field of social sciences. Wilson pioneered the attempt to explain the evolutionary mechanics behind social behaviours such as altruism, aggression, and nurturance.[34] In doing so, Wilson sparked one of the greatest scientific controversies of the 20th century.[34]
The current theory of evolution, the modern evolutionary synthesis (or neo-darwinism), explains that evolution of species occurs through a combination of Darwin’s mechanism of natural selection and Gregor Mendel’s theory of genetics as the basis for biological inheritance and mathematical population genetics.[34] Essentially, the modern synthesis introduced the connection between two important discoveries; the units of evolution (genes) with the main mechanism of evolution (selection).[34]
Due to its close reliance on biology, sociobiology is often considered a branch of the biology, although it uses techniques from a plethora of sciences, including ethology, evolution, zoology, archaeology, population genetics, and many others. Within the study of human societies, sociobiology is closely related to the fields of human behavioral ecology and evolutionary psychology.
Sociobiology has remained highly controversial as it contends genes explain specific human behaviours, although sociobiologists describe this role as a very complex and often unpredictable interaction between nature and nurture. The most notable critics of the view that genes play a direct role in human behaviour have been biologists Richard Lewontin Steven Rose and Stephen Jay Gould.
Since the rise of evolutionary psychology, another school of thought, Dual Inheritance Theory, has emerged in the past 25 years that applies the mathematical standards of Population genetics to modeling the adaptive and selective principles of culture. This school of thought was pioneered by Robert Boyd at UCLA and Peter Richerson at UC Davis and expanded by William Wimsatt, among others. Boyd and Richerson’s book, Culture and the Evolutionary Process (1985), was a highly mathematical description of cultural change, later published in a more accessible form in Not by Genes Alone (2004). In Boyd and Richerson’s view, cultural evolution, operating on socially learned information, exists on a separate but co-evolutionary track from genetic evolution, and while the two are related, cultural evolution is more dynamic, rapid, and influential on human society than genetic evolution. Dual Inheritance Theory has the benefit of providing unifying territory for a "nature and nurture" paradigm and accounts for more accurate phenomenon in evolutionary theory applied to culture, such as randomness effects (drift), concentration dependency, "fidelity" of evolving information systems, and lateral transmission through communication.[35]
Theories of modernization have been developed and popularized in 1950s and 1960s and are closely related to the dependency theory and development theory.[36] They combine the previous theories of sociocultural evolution with practical experiences and empirical research, especially those from the era of decolonization. The theory states that:
Developing from classical social evolutionism theories, the theory of modernization stresses the modernization factor: many societies are simply trying (or need) to emulate the most successful societies and cultures.[36] It also states that it is possible to do so, thus supporting the concepts of social engineering and that the developed countries can and should help those less developed, directly or indirectly.[36]
Among the scientists who contributed much to this theory are Walt Rostow, who in his The Stages of Economic Growth: A Non-Communist Manifesto (1960) concentrates on the economic system side of the modernization, trying to show factors needed for a country to reach the path to modernization in his Rostovian take-off model.[36] David Apter concentrated on the political system and history of democracy, researching the connection between democracy, good governance and efficiency and modernization.[36] David McClelland (The Achieving Society, 1967) approached this subject from the psychological perspective, with his motivations theory, arguing that modernization cannot happen until given society values innovation, success and free enterprise.[36] Alex Inkeles (Becoming Modern, 1974) similarly creates a model of modern personality, which needs to be independent, active, interested in public policies and cultural matters, open to new experiences, rational and able to create long-term plans for the future.[36] Some works of Jürgen Habermas are also connected with this subfield.
The theory of modernization has been subject to some criticism similar to that levied against classical social evolutionism, especially for being too ethnocentric, one-sided and focused on the Western world and its culture.
Cultural evolution follows punctuated equilibrium which Gould and Eldredge developed for biological evolution. Bloomfield[37][38] has written that human societies follow punctuated equilibrium which would mean first, a stable society, and then a transition resulting in a subsequent stable society with greater complexity. This model would claim mankind has had a stable animal society, a transition to a stable tribal society, another transition to a stable peasant society and is currently in a transitional industrial society.
The status of a human society rests on the productivity of food production. Deevey[39] reported on the growth of the number of humans. Deevey also reported on the productivity of food production, noting that productivity changes very little for stable societies, but increases during transitions. When productivity and especially food productivity can no longer be increased, Bloomfield has proposed that man will have achieved a stable automated society.[40]
The Cold War period was marked by rivalry between two superpowers, both of which considered themselves to be the most highly evolved cultures on the planet. The USSR painted itself as a socialist society which emerged from class struggle, destined to reach the state of communism, while sociologists in the United States (such as Talcott Parsons) argued that the freedom and prosperity of the United States were a proof of a higher level of sociocultural evolution of its culture and society. At the same time, decolonization created newly independent countries who sought to become more developed—a model of progress and industrialization which was itself a form of sociocultural evolution.
There is, however, a tradition in European social theory from Rousseau to Max Weber arguing that this progression coincides with a loss of human freedom and dignity. At the height of the Cold War, this tradition merged with an interest in ecology to influence an activist culture in the 1960s. This movement produced a variety of political and philosophical programs which emphasized the importance of bringing society and the environment into harmony.
Many[who?] argue that the next stage of sociocultural evolution consists of a merger with technology, especially information processing technology. Several cumulative major transitions of evolution have transformed life through key innovations in information storage and replication, including RNA, DNA, multicellularity, and also language and culture as inter-human information processing systems.[42][43] in this sense it can be argued that the carbon-based biosphere has generated a cognitive system (humans) capable of creating technology that will result in a comparable evolutionary transition. "Digital information has reached a similar magnitude to information in the biosphere. It increases exponentially, exhibits high-fidelity replication, evolves through differential fitness, is expressed through artificial intelligence (AI), and has facility for virtually limitless recombination. Like previous evolutionary transitions, the potential symbiosis between biological and digital information will reach a critical point where these codes could compete via natural selection. Alternatively, this fusion could create a higher-level superorganism employing a low-conflict division of labor in performing informational tasks...humans already embrace fusions of biology and technology. We spend most of our waking time communicating through digitally mediated channels, ...most transactions on the stock market are executed by automated trading algorithms, and our electric grids are in the hands of artificial intelligence. With one in three marriages in America beginning online, digital algorithms are also taking a role in human pair bonding and reproduction".[41]
Current political theories of the new tribalists consciously mimic ecology and the life-ways of indigenous peoples, augmenting them with modern sciences. Ecoregional Democracy attempts to confine the "shifting groups", or tribes, within "more or less clear boundaries" that a society inherits from the surrounding ecology, to the borders of a naturally occurring ecoregion. Progress can proceed by competition between but not within tribes, and it is limited by ecological borders or by Natural Capitalism incentives which attempt to mimic the pressure of natural selection on a human society by forcing it to adapt consciously to scarce energy or materials. Gaians argue that societies evolve deterministically to play a role in the ecology of their biosphere, or else die off as failures due to competition from more efficient societies exploiting nature's leverage.
Thus, some have appealed to theories of sociocultural evolution to assert that optimizing the ecology and the social harmony of closely knit groups is more desirable or necessary than the progression to "civilization." A 2002 poll of experts on Neoarctic and Neotropic indigenous peoples (reported in Harper's magazine)[citation needed] revealed that all of them would have preferred to be a typical New World person in the year 1491, prior to any European contact, rather than a typical European of that time. This approach has been criticised by pointing out that there are a number of historical examples of indigenous peoples doing severe environmental damage (such as the deforestation of Easter Island and the extinction of mammoths in North America) and that proponents of the goal have been trapped by the European stereotype of the noble savage.

</code>

### Scraping the Sun_____no_output_____

<code>
def get_text(parsed_html):
    article_text = ''
    for paragraph in parsed_html.find('p')[1:-13]:
        new_paragraph = paragraph.text
        article_text  = article_text + '\n' + new_paragraph
    return article_text

def get_headline(parsed_html):
    headline = parsed_html.find('h1')[0].text
    return headline

def get_author(parsed_html):
    author = parsed_html.find('.article__author-name')[0].text
    return author

def scrape_thesun(url):
    '''Scrape a The Sun article and return the headline and text.'''

    # gets the article
    session = HTMLSession()
    r = session.get(url)

    # parses the html
    parsed_html = r.html

    # grabs the headline
    headline = get_headline(parsed_html)

    author = get_author(parsed_html)

    # grabs the text
    text = get_text(parsed_html)

    # stores headline and text in a dictionary
    results = {'headline' : headline,
               'text'     : text,
               'url'      : url,
              'author' : author}

    # send it back
    return results

_____no_output_____scrape_thesun(url)_____no_output_____
</code>

RNA
population genetics
biology
evolution
ecology


# Notebook from areinhold-dev/hmm-tagger
Path: HMM Tagger.ipynb

# Project: Part of Speech Tagging with Hidden Markov Models
---
### Introduction

Part of speech tagging is the process of determining the syntactic category of a word from the words in its surrounding context. It is often used to help disambiguate natural language phrases because it can be done quickly with high accuracy. Tagging can be used for many NLP tasks like determining correct pronunciation during speech synthesis (for example, _dis_-count as a noun vs dis-_count_ as a verb), for information retrieval, and for word sense disambiguation.

In this notebook, you'll use the [Pomegranate](http://pomegranate.readthedocs.io/) library to build a hidden Markov model for part of speech tagging using a "universal" tagset. Hidden Markov models have been able to achieve [>96% tag accuracy with larger tagsets on realistic text corpora](http://www.coli.uni-saarland.de/~thorsten/publications/Brants-ANLP00.pdf). Hidden Markov models have also been used for speech recognition and speech generation, machine translation, gene recognition for bioinformatics, and human gesture recognition for computer vision, and more.

![](_post-hmm.png)

The notebook already contains some code to get you started. You only need to add some new functionality in the areas indicated to complete the project; you will not need to modify the included code beyond what is requested. Sections that begin with **'IMPLEMENTATION'** in the header indicate that you must provide code in the block that follows. Instructions will be provided for each section, and the specifics of the implementation are marked in the code block with a 'TODO' statement. Please be sure to read the instructions carefully!_____no_output_____

**Note:** Once you have completed all of the code implementations, you need to finalize your work by exporting the iPython Notebook as an HTML document. Before exporting the notebook to html, all of the code cells need to have been run so that reviewers can see the final implementation and output. You must then **export the notebook** by running the last cell in the notebook, or by using the menu above and navigating to **File -> Download as -> HTML (.html)** Your submissions should include both the `html` and `ipynb` files.

_____no_output_____

**Note:** Code and Markdown cells can be executed using the `Shift + Enter` keyboard shortcut. Markdown cells can be edited by double-clicking the cell to enter edit mode.

_____no_output_____### The Road Ahead
You must complete Steps 1-3 below to pass the project. The section on Step 4 includes references & resources you can use to further explore HMM taggers.

- [Step 1](#Step-1:-Read-and-preprocess-the-dataset): Review the provided interface to load and access the text corpus
- [Step 2](#Step-2:-Build-a-Most-Frequent-Class-tagger): Build a Most Frequent Class tagger to use as a baseline
- [Step 3](#Step-3:-Build-an-HMM-tagger): Build an HMM Part of Speech tagger and compare to the MFC baseline
- [Step 4](#Step-4:-[Optional]-Improving-model-performance): (Optional) Improve the HMM tagger_____no_output_____

**Note:** Make sure you have selected a **Python 3** kernel in Workspaces or the hmm-tagger conda environment if you are running the Jupyter server on your own machine.

_____no_output_____

%cd "\\europe.bmw.corp\winfs\A-org\AQ-1\AQ-11_intern\40_Big_Data_Analytics\60_Projekte_AQ-16\91_SurveySpace\Jupyter_Notebooks\Playground\Alex_Udacity\git\hmm-tagger"_____no_output_____# Jupyter "magic methods" -- only need to be run once per kernel restart
%load_ext autoreload
%aimport helpers, tests
%autoreload 1_____no_output_____# import python modules -- this cell needs to be run again if you make changes to any of the files
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

from IPython.core.display import HTML
from itertools import chain
from collections import Counter, defaultdict
from helpers import show_model, Dataset
from pomegranate import State, HiddenMarkovModel, DiscreteDistribution_____no_output_____

## Step 1: Read and preprocess the dataset
---
We'll start by reading in a text corpus and splitting it into a training and testing dataset. The data set is a copy of the [Brown corpus](https://en.wikipedia.org/wiki/Brown_Corpus) (originally from the [NLTK](https://www.nltk.org/) library) that has already been pre-processed to only include the [universal tagset](https://arxiv.org/pdf/1104.2086.pdf). You should expect to get slightly higher accuracy using this simplified tagset than the same model would achieve on a larger tagset like the full [Penn treebank tagset](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html), but the process you'll follow would be the same.

The `Dataset` class provided in helpers.py will read and parse the corpus. You can generate your own datasets compatible with the reader by writing them to the following format. The dataset is stored in plaintext as a collection of words and corresponding tags. Each sentence starts with a unique identifier on the first line, followed by one tab-separated word/tag pair on each following line. Sentences are separated by a single blank line.

Example from the Brown corpus.
```
b100-38532
Perhaps	ADV
it	PRON
was	VERB
right	ADJ
;	.
;	.

b100-35577
...
```_____no_output_____

data = Dataset("tags-universal.txt", "brown-universal.txt", train_test_split=0.8)

print("There are {} sentences in the corpus.".format(len(data)))
print("There are {} sentences in the training set.".format(len(data.training_set)))
print("There are {} sentences in the testing set.".format(len(data.testing_set)))

assert len(data) == len(data.training_set) + len(data.testing_set), \
       "The number of sentences in the training set + testing set should sum to the number of sentences in the corpus"There are 57340 sentences in the corpus.
There are 45872 sentences in the training set.
There are 11468 sentences in the testing set.

### The Dataset Interface

You can access (mostly) immutable references to the dataset through a simple interface provided through the `Dataset` class, which represents an iterable collection of sentences along with easy access to partitions of the data for training & testing. Review the reference below, then run and review the next few cells to make sure you understand the interface before moving on to the next step.

```
Dataset-only Attributes:
    training_set - reference to a Subset object containing the samples for training
    testing_set - reference to a Subset object containing the samples for testing

Dataset & Subset Attributes:
    sentences - a dictionary with an entry {sentence_key: Sentence()} for each sentence in the corpus
    keys - an immutable ordered (not sorted) collection of the sentence_keys for the corpus
    vocab - an immutable collection of the unique words in the corpus
    tagset - an immutable collection of the unique tags in the corpus
    X - returns an array of words grouped by sentences ((w11, w12, w13, ...), (w21, w22, w23, ...), ...)
    Y - returns an array of tags grouped by sentences ((t11, t12, t13, ...), (t21, t22, t23, ...), ...)
    N - returns the number of distinct samples (individual words or tags) in the dataset

Methods:
    stream() - returns an flat iterable over all (word, tag) pairs across all sentences in the corpus
    __iter__() - returns an iterable over the data as (sentence_key, Sentence()) pairs
    __len__() - returns the nubmer of sentences in the dataset
```

For example, consider a Subset, `subset`, of the sentences `{"s0": Sentence(("See", "Spot", "run"), ("VERB", "NOUN", "VERB")), "s1": Sentence(("Spot", "ran"), ("NOUN", "VERB"))}`. The subset will have these attributes:

```
subset.keys == {"s1", "s0"}  # unordered
subset.vocab == {"See", "run", "ran", "Spot"}  # unordered
subset.tagset == {"VERB", "NOUN"}  # unordered
subset.X == (("Spot", "ran"), ("See", "Spot", "run"))  # order matches .keys
subset.Y == (("NOUN", "VERB"), ("VERB", "NOUN", "VERB"))  # order matches .keys
subset.N == 5  # there are a total of seven observations over all sentences
len(subset) == 2  # because there are two sentences
```

**Note:** The `Dataset` class is _convenient_, but it is **not** efficient. It is not suitable for huge datasets because it stores multiple redundant copies of the same data.

_____no_output_____#### Sentences

`Dataset.sentences` is a dictionary of all sentences in the training corpus, each keyed to a unique sentence identifier. Each `Sentence` is itself an object with two attributes: a tuple of the words in the sentence named `words` and a tuple of the tag corresponding to each word named `tags`._____no_output_____

key = 'b100-38532'
print("Sentence: {}".format(key))
print("words:\n\t{!s}".format(data.sentences[key].words))
print("tags:\n\t{!s}".format(data.sentences[key].tags))Sentence: b100-38532
words:
	('Perhaps', 'it', 'was', 'right', ';', ';')
tags:
	('ADV', 'PRON', 'VERB', 'ADJ', '.', '.')

**Note:** The underlying iterable sequence is **unordered** over the sentences in the corpus; it is not guaranteed to return the sentences in a consistent order between calls. Use `Dataset.stream()`, `Dataset.keys`, `Dataset.X`, or `Dataset.Y` attributes if you need ordered access to the data.

#### Counting Unique Elements

You can access the list of unique words (the dataset vocabulary) via `Dataset.vocab` and the unique list of tags via `Dataset.tagset`._____no_output_____

print("There are a total of {} samples of {} unique words in the corpus."
      .format(data.N, len(data.vocab)))
print("There are {} samples of {} unique words in the training set."
      .format(data.training_set.N, len(data.training_set.vocab)))
print("There are {} samples of {} unique words in the testing set."
      .format(data.testing_set.N, len(data.testing_set.vocab)))
print("There are {} words in the test set that are missing in the training set."
      .format(len(data.testing_set.vocab - data.training_set.vocab)))

assert data.N == data.training_set.N + data.testing_set.N, \
       "The number of training + test samples should sum to the total number of samples"There are a total of 1161192 samples of 56057 unique words in the corpus.
There are 928458 samples of 50536 unique words in the training set.
There are 232734 samples of 25112 unique words in the testing set.
There are 5521 words in the test set that are missing in the training set.

#### Accessing word and tag Sequences
The `Dataset.X` and `Dataset.Y` attributes provide access to ordered collections of matching word and tag sequences for each sentence in the dataset._____no_output_____

# accessing words with Dataset.X and tags with Dataset.Y
for i in range(2):
    print("Sentence {}:".format(i + 1), data.X[i])
    print()
    print("Labels {}:".format(i + 1), data.Y[i])
    print()Sentence 1: ('Mr.', 'Podger', 'had', 'thanked', 'him', 'gravely', ',', 'and', 'now', 'he', 'made', 'use', 'of', 'the', 'advice', '.')

Labels 1: ('NOUN', 'NOUN', 'VERB', 'VERB', 'PRON', 'ADV', '.', 'CONJ', 'ADV', 'PRON', 'VERB', 'NOUN', 'ADP', 'DET', 'NOUN', '.')

Sentence 2: ('But', 'there', 'seemed', 'to', 'be', 'some', 'difference', 'of', 'opinion', 'as', 'to', 'how', 'far', 'the', 'board', 'should', 'go', ',', 'and', 'whose', 'advice', 'it', 'should', 'follow', '.')

Labels 2: ('CONJ', 'PRT', 'VERB', 'PRT', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'ADP', 'ADV', 'ADV', 'DET', 'NOUN', 'VERB', 'VERB', '.', 'CONJ', 'DET', 'NOUN', 'PRON', 'VERB', 'VERB', '.')

#### Accessing (word, tag) Samples
The `Dataset.stream()` method returns an iterator that chains together every pair of (word, tag) entries across all sentences in the entire corpus._____no_output_____

# use Dataset.stream() (word, tag) samples for the entire corpus
print("\nStream (word, tag) pairs:\n")
for i, pair in enumerate(data.stream()):
    print("\t", pair)
    if i > 5: break
Stream (word, tag) pairs:

	 ('Mr.', 'NOUN')
	 ('Podger', 'NOUN')
	 ('had', 'VERB')
	 ('thanked', 'VERB')
	 ('him', 'PRON')
	 ('gravely', 'ADV')
	 (',', '.')

For both our baseline tagger and the HMM model we'll build, we need to estimate the frequency of tags & words from the frequency counts of observations in the training corpus. In the next several cells you will complete functions to compute the counts of several sets of counts. _____no_output_____## Step 2: Build a Most Frequent Class tagger
---

Perhaps the simplest tagger (and a good baseline for tagger performance) is to simply choose the tag most frequently assigned to each word. This "most frequent class" tagger inspects each observed word in the sequence and assigns it the label that was most often assigned to that word in the corpus._____no_output_____### IMPLEMENTATION: Pair Counts

Complete the function below that computes the joint frequency counts for two input sequences._____no_output_____

def pair_counts(sequences_A, sequences_B):
    """Return a dictionary keyed to each unique value in the first sequence list
    that counts the number of occurrences of the corresponding value from the
    second sequences list.

    For example, if sequences_A is tags and sequences_B is the corresponding
    words, then if 1244 sequences contain the word "time" tagged as a NOUN, then
    you should return a dictionary such that pair_counts[NOUN][time] == 1244
    """
    # TODO: Finish this function!
    for i, (tag,word) in enumerate(zip(sequences_A,sequences_B)):
        dict_raw[tag].append(word)
    for k in dict_raw.keys():
        dict_counted[k] = Counter(dict_raw[k])

    return dict_counted

# Calculate C(t_i, w_i)

# get list of all tags in the datastream:
words_tags = data.training_set.stream()
tags = [i[1] for i in words_tags]

# get list of all words in the datastream:
words_tags = data.training_set.stream()
words = [i[0] for i in words_tags]

# create two dicts with "defaultdict" as this creates new keys if they are not currently present as this eases stuff a lot.
# the first dict will be filled with a list similar to the .stream() function. the second dict will be aggreated/counted on the word/tag pair level (how often is a word in a tag group)
dict_raw = defaultdict(list)
dict_counted = defaultdict(list)

# use the pair_counts function with the list of all tags and with the list of all words.
emission_counts = pair_counts(tags, words)

assert len(emission_counts) == 12, \
       "Uh oh. There should be 12 tags in your dictionary."
assert max(emission_counts["NOUN"], key=emission_counts["NOUN"].get) == 'time', \
       "Hmmm...'time' is expected to be the most common NOUN."
HTML('
Your emission counts look good!
')_____no_output_____emission_counts['NOUN']['time']_____no_output_____

### IMPLEMENTATION: Most Frequent Class Tagger

Use the `pair_counts()` function and the training dataset to find the most frequent class label for each word in the training data, and populate the `mfc_table` below. The table keys should be words, and the values should be the appropriate tag string.

The `MFCTagger` class is provided to mock the interface of Pomegranite HMM models so that they can be used interchangeably._____no_output_____

# Create a lookup table mfc_table where mfc_table[word] contains the tag label most frequently assigned to that word
from collections import namedtuple

FakeState = namedtuple("FakeState", "name")

class MFCTagger:
    # NOTE: You should not need to modify this class or any of its methods
    missing = FakeState(name="<MISSING>")

    def __init__(self, table):
        self.table = defaultdict(lambda: MFCTagger.missing)
        self.table.update({word: FakeState(name=tag) for word, tag in table.items()})

    def viterbi(self, seq):
        """This method simplifies predictions by matching the Pomegranate viterbi() interface"""
        return 0., list(enumerate(["<start>"] + [self.table[w] for w in seq] + ["<end>"]))

# TODO: calculate the frequency of each tag being assigned to each word (hint: similar, but not
# the same as the emission probabilities) and use it to fill the mfc_table

# get list of all tags in the datastream:
words_tags = data.training_set.stream()
tags = [i[1] for i in words_tags]

# get list of all words in the datastream:
words_tags = data.training_set.stream()
words = [i[0] for i in words_tags]

# create same first dict as with the pair_counts:
dict_raw_2 = defaultdict(list)
for i, (tag,word) in enumerate(zip(tags,words)):
    dict_raw_2[word].append(tag)

# create second dict that contains the tag with the highest number per word which is later on used for the mfc_table:
dict_most_tag = defaultdict(list)
for k in dict_raw_2.keys():
    dict_most_tag[k] = (Counter(dict_raw_2[k])).most_common(1)[0][0]

word_counts = pair_counts(tags,words)

mfc_table = dict_most_tag

# DO NOT MODIFY BELOW THIS LINE
mfc_model = MFCTagger(mfc_table) # Create a Most Frequent Class tagger instance

assert len(mfc_table) == len(data.training_set.vocab), ""
assert all(k in data.training_set.vocab for k in mfc_table.keys()), ""
assert sum(int(k not in mfc_table) for k in data.testing_set.vocab) == 5521, ""
HTML('
Your MFC tagger has all the correct words!
')_____no_output_____

### Making Predictions with a Model
The helper functions provided below interface with Pomegranate network models & the mocked MFCTagger to take advantage of the [missing value](http://pomegranate.readthedocs.io/en/latest/nan.html) functionality in Pomegranate through a simple sequence decoding function. Run these functions, then run the next cell to see some of the predictions made by the MFC tagger._____no_output_____

def replace_unknown(sequence):
    """Return a copy of the input sequence where each unknown word is replaced
    by the literal string value 'nan'. Pomegranate will ignore these values
    during computation.
    """
    return [w if w in data.training_set.vocab else 'nan' for w in sequence]

def simplify_decoding(X, model):
    """X should be a 1-D sequence of observations for the model to predict"""
    _, state_path = model.viterbi(replace_unknown(X))
    return [state[1].name for state in state_path[1:-1]]  # do not show the start/end state predictions_____no_output_____

### Example Decoding Sequences with MFC Tagger_____no_output_____

for key in data.testing_set.keys[:3]:
    print("Sentence Key: {}\n".format(key))
    print("Predicted labels:\n-----------------")
    print(simplify_decoding(data.sentences[key].words, mfc_model))
    print()
    print("Actual labels:\n--------------")
    print(data.sentences[key].tags)
    print("\n")Sentence Key: b100-28144

Predicted labels:
-----------------
['CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.']

Actual labels:
--------------
('CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.')

Sentence Key: b100-23146

Predicted labels:
-----------------
['PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.']

Actual labels:
--------------
('PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.')

Sentence Key: b100-35462

Predicted labels:
-----------------
['DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', '<MISSING>', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADV', 'NOUN', '.']

Actual labels:
--------------
('DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADJ', 'NOUN', '.')

### Evaluating Model Accuracy

The function below will evaluate the accuracy of the MFC tagger on the collection of all sentences from a text corpus. _____no_output_____

def accuracy(X, Y, model):
    """Calculate the prediction accuracy by using the model to decode each sequence
    in the input X and comparing the prediction with the true labels in Y.

    The X should be an array whose first dimension is the number of sentences to test,
    and each element of the array should be an iterable of the words in the sequence.
    The arrays X and Y should have the exact same shape.

    X = [("See", "Spot", "run"), ("Run", "Spot", "run", "fast"), ...]
    Y = [(), (), ...]
    """
    correct = total_predictions = 0
    for observations, actual_tags in zip(X, Y):

        # The model.viterbi call in simplify_decoding will return None if the HMM
        # raises an error (for example, if a test sentence contains a word that
        # is out of vocabulary for the training set). Any exception counts the
        # full sentence as an error (which makes this a conservative estimate).
        try:
            most_likely_tags = simplify_decoding(observations, model)
            correct += sum(p == t for p, t in zip(most_likely_tags, actual_tags))
        except:
            pass
        total_predictions += len(observations)
    return correct / total_predictions_____no_output_____

#### Evaluate the accuracy of the MFC tagger
Run the next cell to evaluate the accuracy of the tagger on the training and test corpus._____no_output_____

mfc_training_acc = accuracy(data.training_set.X, data.training_set.Y, mfc_model)
print("training accuracy mfc_model: {:.2f}%".format(100 * mfc_training_acc))

mfc_testing_acc = accuracy(data.testing_set.X, data.testing_set.Y, mfc_model)
print("testing accuracy mfc_model: {:.2f}%".format(100 * mfc_testing_acc))

assert mfc_training_acc >= 0.955, "Uh oh. Your MFC accuracy on the training set doesn't look right."
assert mfc_testing_acc >= 0.925, "Uh oh. Your MFC accuracy on the testing set doesn't look right."
HTML('
Your MFC tagger accuracy looks correct!
')training accuracy mfc_model: 95.72%
testing accuracy mfc_model: 93.01%

## Step 3: Build an HMM tagger
---
The HMM tagger has one hidden state for each possible tag, and parameterized by two distributions: the emission probabilties giving the conditional probability of observing a given **word** from each hidden state, and the transition probabilities giving the conditional probability of moving between **tags** during the sequence.

We will also estimate the starting probability distribution (the probability of each **tag** being the first tag in a sequence), and the terminal probability distribution (the probability of each **tag** being the last tag in a sequence).

The maximum likelihood estimate of these distributions can be calculated from the frequency counts as described in the following sections where you'll implement functions to count the frequencies, and finally build the model. The HMM model will make predictions according to the formula:

$$t_i^n = \underset{t_i^n}{\mathrm{argmax}} \prod_{i=1}^n P(w_i|t_i) P(t_i|t_{i-1})$$

Refer to Speech & Language Processing [Chapter 10](https://web.stanford.edu/~jurafsky/slp3/10.pdf) for more information._____no_output_____### IMPLEMENTATION: Unigram Counts

Complete the function below to estimate the co-occurrence frequency of each symbol over all of the input sequences. The unigram probabilities in our HMM model are estimated from the formula below, where N is the total number of samples in the input. (You only need to compute the counts for now.)

$$P(tag_1) = \frac{C(tag_1)}{N}$$_____no_output_____

def unigram_counts(sequences):
    """Return a dictionary keyed to each unique value in the input sequence list that
    counts the number of occurrences of the value in the sequences list. The sequences
    collection should be a 2-dimensional array.

    For example, if the tag NOUN appears 275558 times over all the input sequences,
    then you should return a dictionary such that your_unigram_counts[NOUN] == 275558.
    """
    # TODO: Finish this function!
    return dict(Counter(sequences))

# TODO: call unigram_counts with a list of tag sequences from the training set
# get list of all tags in the datastream:
tags = [i[1] for i in data.training_set.stream()]
tag_unigrams = unigram_counts(tags)

assert set(tag_unigrams.keys()) == data.training_set.tagset, \
       "Uh oh. It looks like your tag counts doesn't include all the tags!"
assert min(tag_unigrams, key=tag_unigrams.get) == 'X', \
       "Hmmm...'X' is expected to be the least common class"
assert max(tag_unigrams, key=tag_unigrams.get) == 'NOUN', \
       "Hmmm...'NOUN' is expected to be the most common class"
HTML('
Your tag unigrams look good!
')_____no_output_____tag_unigrams_____no_output_____

### IMPLEMENTATION: Bigram Counts

Complete the function below to estimate the co-occurrence frequency of each pair of symbols in each of the input sequences. These counts are used in the HMM model to estimate the bigram probability of two tags from the frequency counts according to the formula: $$P(tag_2|tag_1) = \frac{C(tag_2|tag_1)}{C(tag_2)}$$
_____no_output_____

def bigram_counts(sequences):
    """Return a dictionary keyed to each unique PAIR of values in the input sequences
    list that counts the number of occurrences of pair in the sequences list. The input
    should be a 2-dimensional array.

    For example, if the pair of tags (NOUN, VERB) appear 61582 times, then you should
    return a dictionary such that your_bigram_counts[(NOUN, VERB)] == 61582
    """

    # TODO: Finish this function!
    # create bigrams:
    katze = [tags[i:i+2] for i in range(len(sequences))]

    # keep only bigrams with a len of 2
    rebhuhn = [x for x in katze if len(x) == 2]

    # count number of occurences:
    bonsai = Counter(tuple(item) for item in rebhuhn)

    return bonsai

# TODO: call bigram_counts with a list of tag sequences from the training set
tags = [i[1] for i in data.training_set.stream()]
tag_bigrams = bigram_counts(tags)

assert len(tag_bigrams) == 144, \
       "Uh oh. There should be 144 pairs of bigrams (12 tags x 12 tags)"
assert min(tag_bigrams, key=tag_bigrams.get) in [('X', 'NUM'), ('PRON', 'X')], \
       "Hmmm...The least common bigram should be one of ('X', 'NUM') or ('PRON', 'X')."
assert max(tag_bigrams, key=tag_bigrams.get) in [('DET', 'NOUN')], \
       "Hmmm...('DET', 'NOUN') is expected to be the most common bigram."
HTML('
Your tag bigrams look good!
')_____no_output_____tag_bigrams_____no_output_____

### IMPLEMENTATION: Sequence Starting Counts
Complete the code below to estimate the bigram probabilities of a sequence starting with each tag._____no_output_____

def starting_counts(sequences):
    """Return a dictionary keyed to each unique value in the input sequences list
    that counts the number of occurrences where that value is at the beginning of
    a sequence.

    For example, if 8093 sequences start with NOUN, then you should return a
    dictionary such that your_starting_counts[NOUN] == 8093
    """
    # TODO: Finish this function!
    # keep only first element of sequence
    wauz = [x[0] for x in sequences]

    # Count it.
    mezcal = Counter(wauz)
    return mezcal

# TODO: Calculate the count of each tag starting a sequence
start_end = data.training_set.Y
tag_starts = starting_counts(start_end)

assert len(tag_starts) == 12, "Uh oh. There should be 12 tags in your dictionary."
assert min(tag_starts, key=tag_starts.get) == 'X', "Hmmm...'X' is expected to be the least common starting bigram."
assert max(tag_starts, key=tag_starts.get) == 'DET', "Hmmm...'DET' is expected to be the most common starting bigram."
HTML('
Your starting tag counts look good!
')_____no_output_____tag_starts_____no_output_____

### IMPLEMENTATION: Sequence Ending Counts
Complete the function below to estimate the bigram probabilities of a sequence ending with each tag._____no_output_____

def ending_counts(sequences):
    """Return a dictionary keyed to each unique value in the input sequences list
    that counts the number of occurrences where that value is at the end of
    a sequence.

    For example, if 18 sequences end with DET, then you should return a
    dictionary such that your_starting_counts[DET] == 18
    """
    # TODO: Finish this function!
    # keep only last element of sequence
    huhn = [x[-1] for x in sequences]

    # Count it.
    tequilita = Counter(huhn)
    return tequilita

# TODO: Calculate the count of each tag ending a sequence
start_end = data.training_set.Y
tag_ends = ending_counts(start_end)

assert len(tag_ends) == 12, "Uh oh. There should be 12 tags in your dictionary."
assert min(tag_ends, key=tag_ends.get) in ['X', 'CONJ'], "Hmmm...'X' or 'CONJ' should be the least common ending bigram."
assert max(tag_ends, key=tag_ends.get) == '.', "Hmmm...'.' is expected to be the most common ending bigram."
HTML('
Your ending tag counts look good!
')_____no_output_____tag_ends_____no_output_____

### IMPLEMENTATION: Basic HMM Tagger
Use the tag unigrams and bigrams calculated above to construct a hidden Markov tagger.

- Add one state per tag
    - The emission distribution at each state should be estimated with the formula: $P(w|t) = \frac{C(t, w)}{C(t)}$
- Add an edge from the starting state `basic_model.start` to each tag
    - The transition probability should be estimated with the formula: $P(t|start) = \frac{C(start, t)}{C(start)}$
- Add an edge from each tag to the end state `basic_model.end`
    - The transition probability should be estimated with the formula: $P(end|t) = \frac{C(t, end)}{C(t)}$
- Add an edge between _every_ pair of tags
    - The transition probability should be estimated with the formula: $P(t_2|t_1) = \frac{C(t_1, t_2)}{C(t_1)}$_____no_output_____

basic_model = HiddenMarkovModel(name="base-hmm-tagger")

# TODO: create states with emission probability distributions P(word | tag) and add to the model
# (Hint: you may need to loop & create/add new states)

# get list of all tags in the datastream:
words_tags = data.training_set.stream()
tags = [i[1] for i in words_tags]

# get list of all words in the datastream:
words_tags = data.training_set.stream()
words = [i[0] for i in words_tags]

# using pair_count from Step 2 to get the number of word emissions per tag:
tag_word_no = pair_counts(tags, words)

# using unigram_counts from Step 3:
tags_total = unigram_counts(tags)

####################### S T A T E S ###################################
# create an empty dict of states:
states = []

# For every tag / words_with_number do the following:
for tag, words_with_count in tag_word_no.items():
    # count the total number of word occurences in the tag key
    total = float(sum(words_with_count.values()))
    # Calculate the distribution of the single word in the current tag:
    word_rel = {word: count/total for word, count in words_with_count.items()}
    # Convert dict to DiscreteDistribion format for pomegranate:
    tag_emissions = DiscreteDistribution(word_rel)
    # create a State for pomegranate that will late be added to the model with add_states()
    tag_state = State(tag_emissions, name = tag)
    # fill the "states" dict with the "sub-dict" of each tag.
    states.append(tag_state)

# add the states that were just created with "State()"
basic_model.add_states()

# TODO: add edges between states for the observed transition frequencies P(tag_i | tag_i-1)
# (Hint: you may need to loop & add transitions

####################### S T A R T I N G   T R A N S I T I O N S #################################
# create an empty dict to store the starting probability of each tag:
starting_tag_prob={}

# for each tag: divide its count as a starting tag by the total number of that tag (and store in the dict):
for tag in tags:
    starting_tag_prob[tag]=tag_starts[tag]/tags_total[tag]

# for each of the tag-states add the probability for the starting transition according to pomegranade-docu:
for tag_state in states :
    basic_model.add_transition(basic_model.start,tag_state,starting_tag_prob[tag_state.name])

####################### E N D I N G   T R A N S I T I O N S #################################
# create an empty dict to store the ending probability of each tag:
ending_tag_prob={}

# for each tag: divide its count as an ending tag by the total number of that tag (and store in the dict):
for tag in tags:
    ending_tag_prob[tag]=tag_ends[tag]/tags_total[tag]

# for each of the tag-states add the probability for the ending transition according to pomegranade-docu:
for tag_state in states :
    basic_model.add_transition(tag_state,basic_model.end,ending_tag_prob[tag_state.name])

###################### I N - B E T W E E N  T R A N S I T I O N S #####################################
# create an empty dict to store the probability of transition between two tags:
transition_prob = {}

# for each keys in the tag_bigrams dict do the following:
for key in tag_bigrams.keys():
    # divide the number of pair occurances (e.g. 'ADV', 'NOUN') by the number of total occurences of the first tag of the pair ('ADV') (which means "by all transitions that start with 'ADV'"")
    transition_prob[key]=tag_bigrams.get(key)/tags_total[key[0]]

# for each tag-states add the probability for each transition possibility to the "next_tag_state"
for tag_state in states :
    for next_tag_state in states :
        basic_model.add_transition(tag_state,next_tag_state,transition_prob[(tag_state.name,next_tag_state.name)])

# NOTE: YOU SHOULD NOT NEED TO MODIFY ANYTHING BELOW THIS LINE
# finalize the model
basic_model.bake()

assert all(tag in set(s.name for s in basic_model.states) for tag in data.training_set.tagset), \
       "Every state in your network should use the name of the associated tag, which must be one of the training set tags."
assert basic_model.edge_count() == 168, \
       ("Your network should have an edge from the start node to each state, one edge between every " +
        "pair of tags (states), and an edge from each state to the end node.")
HTML('
Your HMM network topology looks good!
')_____no_output_____hmm_training_acc = accuracy(data.training_set.X, data.training_set.Y, basic_model)
print("training accuracy basic hmm model: {:.2f}%".format(100 * hmm_training_acc))

hmm_testing_acc = accuracy(data.testing_set.X, data.testing_set.Y, basic_model)
print("testing accuracy basic hmm model: {:.2f}%".format(100 * hmm_testing_acc))

assert hmm_training_acc > 0.97, "Uh oh. Your HMM accuracy on the training set doesn't look right."
assert hmm_testing_acc > 0.955, "Uh oh. Your HMM accuracy on the testing set doesn't look right."
HTML('
Your HMM tagger accuracy looks correct! Congratulations, you\'ve finished the project.
')training accuracy basic hmm model: 97.48%
testing accuracy basic hmm model: 95.86%

### Example Decoding Sequences with the HMM Tagger_____no_output_____

for key in data.testing_set.keys[:3]:
    print("Sentence Key: {}\n".format(key))
    print("Predicted labels:\n-----------------")
    print(simplify_decoding(data.sentences[key].words, basic_model))
    print()
    print("Actual labels:\n--------------")
    print(data.sentences[key].tags)
    print("\n")Sentence Key: b100-28144

Predicted labels:
-----------------
['CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.']

Actual labels:
--------------
('CONJ', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'NOUN', 'NUM', '.', 'CONJ', 'NOUN', 'NUM', '.', '.', 'NOUN', '.', '.')

Sentence Key: b100-23146

Predicted labels:
-----------------
['PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.']

Actual labels:
--------------
('PRON', 'VERB', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', 'NOUN', 'VERB', 'VERB', '.', 'ADP', 'VERB', 'DET', 'NOUN', 'ADP', 'NOUN', 'ADP', 'DET', 'NOUN', '.')

Sentence Key: b100-35462

Predicted labels:
-----------------
['DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADJ', 'NOUN', '.']

Actual labels:
--------------
('DET', 'ADJ', 'NOUN', 'VERB', 'VERB', 'VERB', 'ADP', 'DET', 'ADJ', 'ADJ', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', '.', 'ADP', 'ADJ', 'NOUN', '.', 'CONJ', 'ADP', 'DET', 'NOUN', 'ADP', 'ADJ', 'ADJ', '.', 'ADJ', '.', 'CONJ', 'ADJ', 'NOUN', 'ADP', 'ADJ', 'NOUN', '.')

## Finishing the project
---

**Note:** **SAVE YOUR NOTEBOOK**, then run the next cell to generate an HTML copy. You will zip & submit both this file and the HTML copy for review.

_____no_output_____

!!jupyter nbconvert *.ipynb_____no_output_____

## Step 4: [Optional] Improving model performance
---
There are additional enhancements that can be incorporated into your tagger that improve performance on larger tagsets where the data sparsity problem is more significant. The data sparsity problem arises because the same amount of data split over more tags means there will be fewer samples in each tag, and there will be more missing data  tags that have zero occurrences in the data. The techniques in this section are optional.

- [Laplace Smoothing](https://en.wikipedia.org/wiki/Additive_smoothing) (pseudocounts)
    Laplace smoothing is a technique where you add a small, non-zero value to all observed counts to offset for unobserved values.

- Backoff Smoothing
    Another smoothing technique is to interpolate between n-grams for missing data. This method is more effective than Laplace smoothing at combatting the data sparsity problem. Refer to chapters 4, 9, and 10 of the [Speech & Language Processing](https://web.stanford.edu/~jurafsky/slp3/) book for more information.

- Extending to Trigrams
    HMM taggers have achieved better than 96% accuracy on this dataset with the full Penn treebank tagset using an architecture described in [this](http://www.coli.uni-saarland.de/~thorsten/publications/Brants-ANLP00.pdf) paper. Altering your HMM to achieve the same performance would require implementing deleted interpolation (described in the paper), incorporating trigram probabilities in your frequency tables, and re-implementing the Viterbi algorithm to consider three consecutive states instead of two.

### Obtain the Brown Corpus with a Larger Tagset
Run the code below to download a copy of the brown corpus with the full NLTK tagset. You will need to research the available tagset information in the NLTK docs and determine the best way to extract the subset of NLTK tags you want to explore. If you write the following the format specified in Step 1, then you can reload the data using all of the code above for comparison.

Refer to [Chapter 5](http://www.nltk.org/book/ch05.html) of the NLTK book for more information on the available tagsets._____no_output_____

import nltk
from nltk import pos_tag, word_tokenize
from nltk.corpus import brown

nltk.download('brown')
training_corpus = nltk.corpus.brown
training_corpus.tagged_sents()[0]_____no_output_____