# Notebook from Hazqeel09/multilingual-sentiment-analysis-MEC
Path: Scraping Tweets.ipynb

from urllib.request import urlopen
from bs4 import BeautifulSoup

url = "https://mobile.twitter.com/khairulaming?lang=en"
html = urlopen(url)
soup = BeautifulSoup(html, "html.parser")
print(soup.prettify())

# Gets the tweet
tweets = soup.find_all(attrs={"class":'tweet-text'})<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//WAPFORUM//DTD XHTML Mobile 1.1//EN" "http://www.openmobilealliance.org/tech/DTD/xhtml-mobile11.dtd">

  <meta content="True" name="HandheldFriendly"/>
  <meta content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=0" name="viewport"/>
  <link href="https://twitter.com/khairulaming" rel="canonical"/>
  <meta content="twitter://user?screen_name=khairulaming" name="twitter-redirect-url"/>
  <meta content='{"pwreset-iphone":true,"android":true,"email":true}' name="twitter-redirect-srcs"/>
  <link href="https://ma.twimg.com/twitter-mobile/a8491f04da4e54be98cff3a86c58f23a770c26bd/images/favicon.ico" rel="icon" type="image/x-icon">
   <title>
    Khairulaming (@khairulaming) on Twitter
   </title>
   <link href="https://ma.twimg.com/twitter-mobile/a8491f04da4e54be98cff3a86c58f23a770c26bd/assets/a.css" inline="false" media="screen" rel="stylesheet" type="text/css"/>

   <meta content="V0yIS0Ec_o3Ii9KThrCoMCkwTYMMJ_JYx_RSaGhFYvw" name="google-site-verification"/>
   <meta content='{"m2_mmw_scribe_get_url":true}' name="deciders"/>
  </link>

           Log in

           Sign up

         Khairulaming

          @

          khairulaming

         Kuala Lumpur

          I make short and simple cooking videos. Instagram :

           @khairulaming

           instagram.com/_u/khairulaming

         543

         Tweets

          47

          Following

          457,591

          Followers

      <form action="/i/guest/follow/khairulaming" method="post">

        <input name="authenticity_token" type="hidden" value="bb89c2106c5733117305b109a3498e945148e2b1"/>

        <input name="commit" type="submit" value="Follow"/>

      </form>
      <form action="/khairulaming/actions" method="get">

        <input name="commit" type="submit" value="•••"/>

      </form>

       View more photos

        Tweets

        <form action="/khairulaming/follow_sms" method="get">

          <input alt="Follow by SMS" src="https://ma.twimg.com/twitter-mobile/a8491f04da4e54be98cff3a86c58f23a770c26bd/images/sprites/notifications_default_sm.gif" type="image"/>

        </form>

          Khairulaming

           @

          khairulaming

         10h

          Me : bunuh hijau bunuh hijau..aku nampak hijau yg bunuh

Everyone : voted hijau

Me : eh bukannn hijau tu aku wehhh kenapa aku blur ni
.
.
.
Aming was ejected

          View details

          ·

         Khairulaming retweeted

          DJI

           @

          DJIGlobal

         13h

          Capture your moments with DJI Pocket 2. With bigger sensor and new intelligent features like 8x zoom, ActiveTrack 3.0, as well as 3-Axis Stabilized Camera, DJI Pocket 2 is the unconventional camera that will transform your content.

Discover more:

           bit.ly/31mcOXX

           pic.twitter.com/2H3qYHpS65

          View photo

          ·

          Khairulaming

           @

          khairulaming

         18h

          untung fathi boleh sambung tidur

           twitter.com/syaheerahmad/s…

          View details

          ·

          Khairulaming

           @

          khairulaming

         Oct 19

          kenapa sorry bang jago melekat dalam kepala pulak ni

          View details

          ·

          Khairulaming

           @

          khairulaming

         Oct 19

          bila gaji masuk              bila gaji habis

           twitter.com/azfarheri/stat…

           pic.twitter.com/AJ6PG8AA0H

          View photo

          ·

          Khairulaming

           @

          khairulaming

         Oct 19

         Replying to

          @hanifjamals

          ??

           twitter.com/khairulaming/s…

          View conversation

          ·

          Khairulaming

           @

          khairulaming

         Oct 18

          kak limah dh buat asmr ke skrang

           twitter.com/isaacosmantv/s…

          View details

          ·

          Khairulaming

           @

          khairulaming

         Oct 18

          avatar yang tersakiti

           pic.twitter.com/3RkMWSiPFd

          View photo

          ·

         Khairulaming retweeted

          Steve Canal

           @

          SteveCanal

         Oct 14

          Distractions will look like opportunities when you’re not focused on where you’re going.

          View details

          ·

          Khairulaming

           @

          khairulaming

         Oct 17

         Replying to

          @TheShadspeare

          sbeloh 7e tu ko? kawe make sokmo situ. ambo ore tamey uda jugok hehehe

          View conversation

          ·

          Khairulaming

           @

          khairulaming

         Oct 17

         Replying to

          @atifipann

          ejah tumih nge nyo perlebih skit deh

          View conversation

          ·

          Khairulaming

           @

          khairulaming

         Oct 17

          rindu weh nok kelik klate makey nasi kerabu

          View details

          ·

          Khairulaming

           @

          khairulaming

         Oct 17

         Replying to

          @AmaninaAdaris

          @AfiqahMyusof

          happy birthday

           @AfiqahMyusof

          ! semoga di panjangkan umur dan dimurahkan rezekiii. selamat memasak kat dapur dgn periuk baru okeh! 🥳

          View conversation

          ·

          Khairulaming

           @

          khairulaming

         Oct 16

         Replying to

          @kemeyhakim

          kau diam

          View conversation

          ·

          Khairulaming

           @

          khairulaming

         Oct 16

          Among Us ni mengajar aku utk menipu tapi buat buat innocent hihi

          View details

          ·

          Khairulaming

           @

          khairulaming

         Oct 16

          update : dah 10 ribu orang join thank you guyssss 😭

           twitter.com/khairulaming/s…

           pic.twitter.com/9EWUMAUq9B

          View photo

          ·

          Khairulaming

           @

          khairulaming

         Oct 16

         Replying to

          @fazley13

          thank you abam impostorrr hihi

          View conversation

          ·

          Khairulaming

           @

          khairulaming

         Oct 16

          aku ada buat satu telegram channel yg fokus pada food & travel. kat sini aku akan share macam2 info so nanti senang utk korang save and refer in the future. kalau sudi nak mintak tolong korang rt and join channel ni 🙏🏻. Thanks guys! :

           t.me/khairulamingbr…

           pic.twitter.com/ZPqUwnVwv4

          View photo

          ·

          Khairulaming

           @

          khairulaming

         Oct 16

          kalau si amin ni pakai spec dh sbijik muka aku

           twitter.com/hxsm/status/13…

          View details

          ·

          Khairulaming

           @

          khairulaming

         Oct 15

         Replying to

          @hanifjamals

          padahal ni gambar hari rabu

          View conversation

          ·

       Load older Tweets

       Enter a topic, @name, or fullname

     <form action="/search" class="search-input" method="get">

          <input id="q" name="q" type="text" value=""/>

         <input name="s" type="hidden" value="typd">
          <input alt="Search" src="https://ma.twimg.com/twitter-mobile/a8491f04da4e54be98cff3a86c58f23a770c26bd/images/sprites/magnifying_glass.gif" type="image">
          </input>
         </input>

     </form>

        Settings

        Help

      Back to top

     ·

      Turn images off

tweets = soup.find_all(attrs={"class":'tweet-text'})
for tweet in tweets:
    print(tweet.text.strip())Me : bunuh hijau bunuh hijau..aku nampak hijau yg bunuh

Everyone : voted hijau

Me : eh bukannn hijau tu aku wehhh kenapa aku blur ni
.
.
.
Aming was ejected
Capture your moments with DJI Pocket 2. With bigger sensor and new intelligent features like 8x zoom, ActiveTrack 3.0, as well as 3-Axis Stabilized Camera, DJI Pocket 2 is the unconventional camera that will transform your content.

Discover more: bit.ly/31mcOXX pic.twitter.com/2H3qYHpS65
untung fathi boleh sambung tidur twitter.com/syaheerahmad/s…
kenapa sorry bang jago melekat dalam kepala pulak ni
bila gaji masuk              bila gaji habis twitter.com/azfarheri/stat… pic.twitter.com/AJ6PG8AA0H
?? twitter.com/khairulaming/s…
kak limah dh buat asmr ke skrang twitter.com/isaacosmantv/s…
avatar yang tersakiti pic.twitter.com/3RkMWSiPFd
Distractions will look like opportunities when you’re not focused on where you’re going.
sbeloh 7e tu ko? kawe make sokmo situ. ambo ore tamey uda jugok hehehe
ejah tumih nge nyo perlebih skit deh
rindu weh nok kelik klate makey nasi kerabu
happy birthday @AfiqahMyusof ! semoga di panjangkan umur dan dimurahkan rezekiii. selamat memasak kat dapur dgn periuk baru okeh! 🥳
kau diam
Among Us ni mengajar aku utk menipu tapi buat buat innocent hihi
update : dah 10 ribu orang join thank you guyssss 😭 twitter.com/khairulaming/s… pic.twitter.com/9EWUMAUq9B
thank you abam impostorrr hihi
aku ada buat satu telegram channel yg fokus pada food & travel. kat sini aku akan share macam2 info so nanti senang utk korang save and refer in the future. kalau sudi nak mintak tolong korang rt and join channel ni 🙏🏻. Thanks guys! : t.me/khairulamingbr… pic.twitter.com/ZPqUwnVwv4
kalau si amin ni pakai spec dh sbijik muka aku twitter.com/hxsm/status/13…
padahal ni gambar hari rabu
print(len(tweets))20
url = "https://mobile.twitter.com/khairulaming?lang=en".format(0)
print(url)https://mobile.twitter.com/khairulaming?lang=en
url = Request("https://mobile.twitter.com/khairulaming?lang=en".format(1))
page = urlopen(url)<http.client.HTTPResponse object at 0x000001EA1B9C1B50>
from urllib.request import urlopen, Request
from bs4 import BeautifulSoup
url = "https://mobile.twitter.com/khairulaming?lang=en"

for page in range(0,5):
    res = Request(url.format(page))
    page = urlopen(res)
    soup = BeautifulSoup(page,'html.parser')
    tweets = soup.find_all(attrs={"class":'tweet-text'})
    for tweet in tweets:
        print(tweet.text.strip())
tunjuk resit ni kat barista pic.twitter.com/ZtuXdzxIci
pic.twitter.com/loeNq9hBaH
apa lagi menu starbucks custom yang sedap eh. hari tu ada try earl grey tea latte..guna almond milk (letak tea terus dlm steamed milk and bukan dlm water) pastu tambah 3 pump vanilla syrup. yang ni pun sodappppp
16.85
baru try tadi sedappppp. refreshing betuiii rasa dia 😋 twitter.com/nabellaanuar/s… pic.twitter.com/y3zfRTxVRf
Me : bunuh hijau bunuh hijau..aku nampak hijau yg bunuh

Everyone : voted hijau

Me : eh bukannn hijau tu aku wehhh kenapa aku blur ni
.
.
.
Aming was ejected
Capture your moments with DJI Pocket 2. With bigger sensor and new intelligent features like 8x zoom, ActiveTrack 3.0, as well as 3-Axis Stabilized Camera, DJI Pocket 2 is the unconventional camera that will transform your content.

Discover more: bit.ly/31mcOXX pic.twitter.com/2H3qYHpS65
untung fathi boleh sambung tidur twitter.com/syaheerahmad/s…
kenapa sorry bang jago melekat dalam kepala pulak ni
bila gaji masuk              bila gaji habis twitter.com/azfarheri/stat… pic.twitter.com/AJ6PG8AA0H
?? twitter.com/khairulaming/s…
kak limah dh buat asmr ke skrang twitter.com/isaacosmantv/s…
avatar yang tersakiti pic.twitter.com/3RkMWSiPFd
Distractions will look like opportunities when you’re not focused on where you’re going.
sbeloh 7e tu ko? kawe make sokmo situ. ambo ore tamey uda jugok hehehe
ejah tumih nge nyo perlebih skit deh
rindu weh nok kelik klate makey nasi kerabu
happy birthday @AfiqahMyusof ! semoga di panjangkan umur dan dimurahkan rezekiii. selamat memasak kat dapur dgn periuk baru okeh! 🥳
kau diam
Among Us ni mengajar aku utk menipu tapi buat buat innocent hihi
tunjuk resit ni kat barista pic.twitter.com/ZtuXdzxIci
pic.twitter.com/loeNq9hBaH
apa lagi menu starbucks custom yang sedap eh. hari tu ada try earl grey tea latte..guna almond milk (letak tea terus dlm steamed milk and bukan dlm water) pastu tambah 3 pump vanilla syrup. yang ni pun sodappppp
16.85
baru try tadi sedappppp. refreshing betuiii rasa dia 😋 twitter.com/nabellaanuar/s… pic.twitter.com/y3zfRTxVRf
Me : bunuh hijau bunuh hijau..aku nampak hijau yg bunuh

Everyone : voted hijau

Me : eh bukannn hijau tu aku wehhh kenapa aku blur ni
.
.
.
Aming was ejected
Capture your moments with DJI Pocket 2. With bigger sensor and new intelligent features like 8x zoom, ActiveTrack 3.0, as well as 3-Axis Stabilized Camera, DJI Pocket 2 is the unconventional camera that will transform your content.

Discover more: bit.ly/31mcOXX pic.twitter.com/2H3qYHpS65
untung fathi boleh sambung tidur twitter.com/syaheerahmad/s…
kenapa sorry bang jago melekat dalam kepala pulak ni
bila gaji masuk              bila gaji habis twitter.com/azfarheri/stat… pic.twitter.com/AJ6PG8AA0H
?? twitter.com/khairulaming/s…
kak limah dh buat asmr ke skrang twitter.com/isaacosmantv/s…
avatar yang tersakiti pic.twitter.com/3RkMWSiPFd
Distractions will look like opportunities when you’re not focused on where you’re going.
sbeloh 7e tu ko? kawe make sokmo situ. ambo ore tamey uda jugok hehehe
ejah tumih nge nyo perlebih skit deh
rindu weh nok kelik klate makey nasi kerabu
happy birthday @AfiqahMyusof ! semoga di panjangkan umur dan dimurahkan rezekiii. selamat memasak kat dapur dgn periuk baru okeh! 🥳
kau diam
Among Us ni mengajar aku utk menipu tapi buat buat innocent hihi
tunjuk resit ni kat barista pic.twitter.com/ZtuXdzxIci
pic.twitter.com/loeNq9hBaH
apa lagi menu starbucks custom yang sedap eh. hari tu ada try earl grey tea latte..guna almond milk (letak tea terus dlm steamed milk and bukan dlm water) pastu tambah 3 pump vanilla syrup. yang ni pun sodappppp
16.85
baru try tadi sedappppp. refreshing betuiii rasa dia 😋 twitter.com/nabellaanuar/s… pic.twitter.com/y3zfRTxVRf
Me : bunuh hijau bunuh hijau..aku nampak hijau yg bunuh

Everyone : voted hijau

Me : eh bukannn hijau tu aku wehhh kenapa aku blur ni
.
.
.
Aming was ejected
Capture your moments with DJI Pocket 2. With bigger sensor and new intelligent features like 8x zoom, ActiveTrack 3.0, as well as 3-Axis Stabilized Camera, DJI Pocket 2 is the unconventional camera that will transform your content.

Discover more: bit.ly/31mcOXX pic.twitter.com/2H3qYHpS65
untung fathi boleh sambung tidur twitter.com/syaheerahmad/s…
kenapa sorry bang jago melekat dalam kepala pulak ni
bila gaji masuk              bila gaji habis twitter.com/azfarheri/stat… pic.twitter.com/AJ6PG8AA0H
?? twitter.com/khairulaming/s…
kak limah dh buat asmr ke skrang twitter.com/isaacosmantv/s…
avatar yang tersakiti pic.twitter.com/3RkMWSiPFd
Distractions will look like opportunities when you’re not focused on where you’re going.
sbeloh 7e tu ko? kawe make sokmo situ. ambo ore tamey uda jugok hehehe
ejah tumih nge nyo perlebih skit deh
rindu weh nok kelik klate makey nasi kerabu
happy birthday @AfiqahMyusof ! semoga di panjangkan umur dan dimurahkan rezekiii. selamat memasak kat dapur dgn periuk baru okeh! 🥳
kau diam
Among Us ni mengajar aku utk menipu tapi buat buat innocent hihi
tunjuk resit ni kat barista pic.twitter.com/ZtuXdzxIci
pic.twitter.com/loeNq9hBaH
apa lagi menu starbucks custom yang sedap eh. hari tu ada try earl grey tea latte..guna almond milk (letak tea terus dlm steamed milk and bukan dlm water) pastu tambah 3 pump vanilla syrup. yang ni pun sodappppp
16.85
baru try tadi sedappppp. refreshing betuiii rasa dia 😋 twitter.com/nabellaanuar/s… pic.twitter.com/y3zfRTxVRf
Me : bunuh hijau bunuh hijau..aku nampak hijau yg bunuh

Everyone : voted hijau

Me : eh bukannn hijau tu aku wehhh kenapa aku blur ni
.
.
.
Aming was ejected
Capture your moments with DJI Pocket 2. With bigger sensor and new intelligent features like 8x zoom, ActiveTrack 3.0, as well as 3-Axis Stabilized Camera, DJI Pocket 2 is the unconventional camera that will transform your content.

Discover more: bit.ly/31mcOXX pic.twitter.com/2H3qYHpS65
untung fathi boleh sambung tidur twitter.com/syaheerahmad/s…
kenapa sorry bang jago melekat dalam kepala pulak ni
bila gaji masuk              bila gaji habis twitter.com/azfarheri/stat… pic.twitter.com/AJ6PG8AA0H
?? twitter.com/khairulaming/s…
kak limah dh buat asmr ke skrang twitter.com/isaacosmantv/s…
avatar yang tersakiti pic.twitter.com/3RkMWSiPFd
Distractions will look like opportunities when you’re not focused on where you’re going.
sbeloh 7e tu ko? kawe make sokmo situ. ambo ore tamey uda jugok hehehe
ejah tumih nge nyo perlebih skit deh
rindu weh nok kelik klate makey nasi kerabu
happy birthday @AfiqahMyusof ! semoga di panjangkan umur dan dimurahkan rezekiii. selamat memasak kat dapur dgn periuk baru okeh! 🥳
kau diam
Among Us ni mengajar aku utk menipu tapi buat buat innocent hihi
tunjuk resit ni kat barista pic.twitter.com/ZtuXdzxIci
pic.twitter.com/loeNq9hBaH
apa lagi menu starbucks custom yang sedap eh. hari tu ada try earl grey tea latte..guna almond milk (letak tea terus dlm steamed milk and bukan dlm water) pastu tambah 3 pump vanilla syrup. yang ni pun sodappppp
16.85
baru try tadi sedappppp. refreshing betuiii rasa dia 😋 twitter.com/nabellaanuar/s… pic.twitter.com/y3zfRTxVRf
Me : bunuh hijau bunuh hijau..aku nampak hijau yg bunuh

Everyone : voted hijau

Me : eh bukannn hijau tu aku wehhh kenapa aku blur ni
.
.
.
Aming was ejected
Capture your moments with DJI Pocket 2. With bigger sensor and new intelligent features like 8x zoom, ActiveTrack 3.0, as well as 3-Axis Stabilized Camera, DJI Pocket 2 is the unconventional camera that will transform your content.

Discover more: bit.ly/31mcOXX pic.twitter.com/2H3qYHpS65
untung fathi boleh sambung tidur twitter.com/syaheerahmad/s…
kenapa sorry bang jago melekat dalam kepala pulak ni
bila gaji masuk              bila gaji habis twitter.com/azfarheri/stat… pic.twitter.com/AJ6PG8AA0H
?? twitter.com/khairulaming/s…
kak limah dh buat asmr ke skrang twitter.com/isaacosmantv/s…
avatar yang tersakiti pic.twitter.com/3RkMWSiPFd
Distractions will look like opportunities when you’re not focused on where you’re going.
sbeloh 7e tu ko? kawe make sokmo situ. ambo ore tamey uda jugok hehehe
ejah tumih nge nyo perlebih skit deh
rindu weh nok kelik klate makey nasi kerabu
happy birthday @AfiqahMyusof ! semoga di panjangkan umur dan dimurahkan rezekiii. selamat memasak kat dapur dgn periuk baru okeh! 🥳
kau diam
Among Us ni mengajar aku utk menipu tapi buat buat innocent hihi
url = "https://twitter.com/search?q=road"
html = urlopen(url)
soup = BeautifulSoup(html, "html.parser")
print(soup.prettify())

# Gets the tweet
tweets = soup.find_all(attrs={"data-testid":'tweet'})
print(tweets)
for tweet in tweets:
    print(tweet.text.strip())<!DOCTYPE html>

 <meta charset="utf-8"/>
 <meta content="width=device-width,initial-scale=1,maximum-scale=1,user-scalable=0,viewport-fit=cover" name="viewport"/>
 <link href="//abs.twimg.com" rel="preconnect"/>
 <link href="//api.twitter.com" rel="preconnect"/>
 <link href="//pbs.twimg.com" rel="preconnect"/>
 <link href="//t.co" rel="preconnect"/>
 <link href="//video.twimg.com" rel="preconnect"/>
 <link href="//abs.twimg.com" rel="dns-prefetch"/>
 <link href="//api.twitter.com" rel="dns-prefetch"/>
 <link href="//pbs.twimg.com" rel="dns-prefetch"/>
 <link href="//t.co" rel="dns-prefetch"/>
 <link href="//video.twimg.com" rel="dns-prefetch"/>
 <link as="script" crossorigin="anonymous" href="https://abs.twimg.com/responsive-web/client-web-legacy/polyfills.18e394f5.js" nonce="YjFiNzkyNGEtYThlMS00Yjg0LTg5ZjItMDY1MzAxMDYyM2Y1" rel="preload"/>
 <link as="script" crossorigin="anonymous" href="https://abs.twimg.com/responsive-web/client-web-legacy/vendors~main.45e01195.js" nonce="YjFiNzkyNGEtYThlMS00Yjg0LTg5ZjItMDY1MzAxMDYyM2Y1" rel="preload"/>
 <link as="script" crossorigin="anonymous" href="https://abs.twimg.com/responsive-web/client-web-legacy/i18n/en.b2193155.js" nonce="YjFiNzkyNGEtYThlMS00Yjg0LTg5ZjItMDY1MzAxMDYyM2Y1" rel="preload"/>
 <link as="script" crossorigin="anonymous" href="https://abs.twimg.com/responsive-web/client-web-legacy/main.ed211455.js" nonce="YjFiNzkyNGEtYThlMS00Yjg0LTg5ZjItMDY1MzAxMDYyM2Y1" rel="preload"/>
 <meta content="2231777543" property="fb:app_id"/>
 <meta content="Twitter" property="og:site_name"/>
 <meta content="V0yIS0Ec_o3Ii9KThrCoMCkwTYMMJ_JYx_RSaGhFYvw" name="google-site-verification"/>
 <link crossorigin="use-credentials" href="/manifest.json" rel="manifest"/>
 <link href="https://twitter.com/search?q=road" hreflang="x-default" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=ar" hreflang="ar" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=bg" hreflang="bg" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=bn" hreflang="bn" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=ca" hreflang="ca" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=cs" hreflang="cs" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=da" hreflang="da" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=de" hreflang="de" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=el" hreflang="el" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=en" hreflang="en" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=en-GB" hreflang="en-GB" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=es" hreflang="es" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=eu" hreflang="eu" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=fa" hreflang="fa" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=fi" hreflang="fi" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=tl" hreflang="tl" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=fr" hreflang="fr" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=ga" hreflang="ga" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=gl" hreflang="gl" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=gu" hreflang="gu" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=he" hreflang="he" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=hi" hreflang="hi" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=hr" hreflang="hr" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=hu" hreflang="hu" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=id" hreflang="id" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=it" hreflang="it" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=ja" hreflang="ja" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=kn" hreflang="kn" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=ko" hreflang="ko" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=mr" hreflang="mr" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=ms" hreflang="ms" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=nb" hreflang="nb" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=nl" hreflang="nl" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=pl" hreflang="pl" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=pt" hreflang="pt" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=ro" hreflang="ro" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=ru" hreflang="ru" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=sk" hreflang="sk" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=sr" hreflang="sr" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=sv" hreflang="sv" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=ta" hreflang="ta" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=th" hreflang="th" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=tr" hreflang="tr" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=uk" hreflang="uk" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=ur" hreflang="ur" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=vi" hreflang="vi" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=zh" hreflang="zh" rel="alternate"/>
 <link href="https://twitter.com/search?q=road&lang=zh-Hant" hreflang="zh-Hant" rel="alternate"/>
 <link href="https://twitter.com/search?q=road" rel="canonical"/>
 <link href="/opensearch.xml" rel="search" title="Twitter" type="application/opensearchdescription+xml"/>
 <link color="#1da1f2" href="https://abs.twimg.com/responsive-web/client-web-legacy/icon-svg.9e211f65.svg" rel="mask-icon" sizes="any"/>
 <link href="//abs.twimg.com/favicons/twitter.ico" rel="shortcut icon" type="image/x-icon"/>
 <link href="https://abs.twimg.com/responsive-web/client-web-legacy/icon-ios.8ea219d5.png" rel="apple-touch-icon" sizes="192x192">
  <meta content="yes" name="mobile-web-app-capable"/>
  <meta content="Twitter" name="apple-mobile-web-app-title"/>
  <meta content="white" name="apple-mobile-web-app-status-bar-style"/>
  <meta content="#ffffff" name="theme-color"/>
  <meta content="Ap6SMBNB0lQoXpXl4I9vyTJqJ7Y0X9tPd6Q6rN697iHdubQQxBcWHy21N3N7uEz7Ba5UKMbN+eLvDczBSbi27AsAAABfeyJvcmlnaW4iOiJodHRwczovL3R3aXR0ZXIuY29tOjQ0MyIsImZlYXR1cmUiOiJCYWRnaW5nIiwiZXhwaXJ5IjoxNTY0NTgyNzY2LCJpc1N1YmRvbWFpbiI6dHJ1ZX0=" http-equiv="origin-trial"/>
  <meta content="Apir4chqTX+4eFxKD+ErQlKRB/VtZ/dvnLfd9Y9Nenl5r1xJcf81alryTHYQiuUlz9Q49MqGXqyaiSmqWzHUqQwAAABneyJvcmlnaW4iOiJodHRwczovL3R3aXR0ZXIuY29tOjQ0MyIsImZlYXR1cmUiOiJDb250YWN0c01hbmFnZXIiLCJleHBpcnkiOjE1NzUwMzUyODMsImlzU3ViZG9tYWluIjp0cnVlfQ==" http-equiv="origin-trial"/>

    <form action="https://mobile.twitter.com/i/nojs_router?path=%2Fsearch%3Fq%3Droad" method="POST" style="background-color: #fff; position: fixed; top: 0; left: 0; right: 0; bottom: 0; z-index: 9999;">

       We've detected that JavaScript is disabled in your browser. Would you like to proceed to legacy Twitter?

       <button style="background-color: #1da1f2; border-radius: 100px; border: none; box-shadow: none; color: #fff; cursor: pointer; font-size: 14px; font-weight: bold; line-height: 20px; padding: 6px 16px;" type="submit">
        Yes
       </button>

    </form>

       <svg class="r-1p0dtai r-13gxpu9 r-4qtqp9 r-yyyyoo r-wy61xf r-1d2f490 r-ywje51 r-dnmrzs r-u8s1d r-zchlnj r-1plcrui r-ipm5af r-lrvibr r-1blnp2b" viewbox="0 0 24 24">
        <g>
         <path d="M23.643 4.937c-.835.37-1.732.62-2.675.733.962-.576 1.7-1.49 2.048-2.578-.9.534-1.897.922-2.958 1.13-.85-.904-2.06-1.47-3.4-1.47-2.572 0-4.658 2.086-4.658 4.66 0 .364.042.718.12 1.06-3.873-.195-7.304-2.05-9.602-4.868-.4.69-.63 1.49-.63 2.342 0 1.616.823 3.043 2.072 3.878-.764-.025-1.482-.234-2.11-.583v.06c0 2.257 1.605 4.14 3.737 4.568-.392.106-.803.162-1.227.162-.3 0-.593-.028-.877-.082.593 1.85 2.313 3.198 4.352 3.234-1.595 1.25-3.604 1.995-5.786 1.995-.376 0-.747-.022-1.112-.065 2.062 1.323 4.51 2.093 7.14 2.093 8.57 0 13.255-7.098 13.255-13.254 0-.2-.005-.402-.014-.602.91-.658 1.7-1.477 2.323-2.41z">
         </path>
        </g>
       </svg>

       <form action="" method="GET">

          Something went wrong, but don’t fret — let’s give it another shot.

         <input name="failedScript" type="hidden" value=""/>
         <input type="submit" value="Try again"/>

       </form>

 </link>

[]
tweets = soup.find_all({"data-testid":'tweet'})
print(tweets)[]
#from fake_useragent import UserAgent
from urllib.request import urlopen, Request
from bs4 import BeautifulSoup

url = "https://twitter.com/search?q=road"
header = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1309.0 Safari/537.17'}

# ua = UserAgent()
# print(ua.chrome)
# header = {'User-Agent':str(ua.chrome)}
# print(header)

htmlContent = Request(url, headers=header)
html = urlopen(htmlContent)
soup = BeautifulSoup(html, "html.parser")
#print(soup.prettify())

# Gets the tweet
tweets = soup.find_all(attrs={"class":'tweet-text'})

for tweet in tweets:
    print(tweet.text.strip())NEW: concern after “fringe” drivers’ group which denies climate science and wants higher speed limits and less focus on walking and cyclist is invited to give evidence to MPs on road safety.

theguardian.com/world/2020/oct…
Free Excel tutorial: excel-easy.com pic.twitter.com/DEFUSbTbD5
More shooting again this morning on Lekki Epe road - Dj Switch pic.twitter.com/sFBemP0MM2
We’re around the Tollgate now but this monsters are still shooting. They’ve occupied the road #EndSARS
[HQ] 201021 Osen Oh! Sketch: Star Road - LOONA

kpopping.com/kpics/details/… pic.twitter.com/R7FuPVjEI9
Kal SHER aa raha hai...💪🏻
BRACE YOURSELVES... 🔥

#RamarajuForBheemTomorrow 🌊🔥

#RRRMovie #RamarajuForBheem pic.twitter.com/ldbl90v8Fh
ALERT ⚠
Nature is unforgiving pic.twitter.com/bLANBJUimv

Via @FauzKhalid
you don't really have to go to Diplomat Hotel or Loakan road at night to experience a ghost. i-message mo lang kadi crush mo, parehas met lang effect na.
#ICYM
Here's our Katojo-Biharwe estate. Only 2.5KM off the main road, electricity on location.
Site has commercial and residential plots. A 50*100 is at 5M.
Just reach out to @LtdAmity pic.twitter.com/zuqQkVAGEd
07:11
People are still being shot at lekki-epe road 🇳🇬💔 pic.twitter.com/vQ97W3AY5E
BREAKING: Another massive shooting happening now at Lekki-Epe road as MILITARY continues to shoot at #EndSARS protesters, leaving many people injured.

#BuhariResignNow #LekkiMassacre #LekkiTollGateMassacre #Lekki
#basedonwhat those at airport road should listen to this pic.twitter.com/jESMwZrgcA
‼TREASURE ROAD TO ROTY DATE REMINDERS ‼
Oct. 26(H-5)-TMA Voting Ends (Fannstar)
Oct. 29(H-8)- MAMA Voting Opens (Mwave)
Oct. 29(H-8)-Super Rookie 2021 Ends (Fanplus)
Nov. 15(H-25)- AAA Voting Ends (Choeaedol)
+ Mubeat,Idol champ, whosfan,mwave for ch3 [tb]
#LOONA press photos from Star Road
#이달의소녀 @loonatheworld pic.twitter.com/0Zl0V3uoi5
[OFFICIAL] 201021 More "Star Road" photos from OSEN V LIVE channel #1

"#LOONA displayed their superb showmanship and flawless teamwork among the members."

🔗channels.vlive.tv/C7A6F1/celeb/0…
@loonatheworld #이달의소녀 pic.twitter.com/nKeHNBurja
anyway they want to make the road near clear creek four lanes wide, which absolutely fucking sucks pic.twitter.com/LbPim4WFDd
Lights went off, bonfires were setup on both ends of the road, soldiers took attack formations and  opened fire on unarmed civilians, seized corpses of those shot then switched the light back on afterwards.
Premeditated murder 💔😭 #LekkiMassacre #EndSARS
Ijoko- Sango Road blocked, no movement. We are trekking today, it's a sacrifice we have to pay.
#EndSARS
Just received a call from my staff....... if you stay at magodo or along CMD road, do not come outside!!!!! Thugs have taken over the road and they are destroying cars once you get to the road blocks they’ve created!!!!! Be warned
#ADMKfailedFarmers
Come to main road pic.twitter.com/yKDa9m48oo
Road Safety
Teaching Road Safety
Lanes Driving
Need of the hour,
Designate ,Bus Lanes,Bike Lanes and Pedestrians Walkway.
Below see the Difference
What required,
What Is at Present
Hat's off to the System pic.twitter.com/plf7rhWCUu
#from fake_useragent import UserAgent
from urllib.request import urlopen, Request
from bs4 import BeautifulSoup

url = "https://mobile.twitter.com/search?q=road"
header = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.4; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2225.0 Safari/537.36'}

# ua = UserAgent()
# print(ua.chrome)
# header = {'User-Agent':str(ua.chrome)}
# print(header)

htmlContent = Request(url, headers=header)
html = urlopen(htmlContent)
soup = BeautifulSoup(html, "html.parser")
print(soup.prettify())

# Gets the tweet
tweets = soup.find_all(attrs={"class":'tweet-text'})

for tweet in tweets:
    print(tweet.text.strip())<!DOCTYPE html>

 <meta charset="utf-8"/>
 <meta content="width=device-width,initial-scale=1,maximum-scale=1,user-scalable=0,viewport-fit=cover" name="viewport"/>
 <link href="//abs.twimg.com" rel="preconnect"/>
 <link href="//api.twitter.com" rel="preconnect"/>
 <link href="//pbs.twimg.com" rel="preconnect"/>
 <link href="//t.co" rel="preconnect"/>
 <link href="//video.twimg.com" rel="preconnect"/>
 <link href="//abs.twimg.com" rel="dns-prefetch"/>
 <link href="//api.twitter.com" rel="dns-prefetch"/>
 <link href="//pbs.twimg.com" rel="dns-prefetch"/>
 <link href="//t.co" rel="dns-prefetch"/>
 <link href="//video.twimg.com" rel="dns-prefetch"/>
 <link as="script" crossorigin="anonymous" href="https://abs.twimg.com/responsive-web/client-web-legacy/polyfills.18e394f5.js" nonce="NDM2ZmUyMzUtMDk4ZS00YTc0LWFmNGQtZWVjZDZmNjdmMjVk" rel="preload"/>
 <link as="script" crossorigin="anonymous" href="https://abs.twimg.com/responsive-web/client-web-legacy/vendors~main.45e01195.js" nonce="NDM2ZmUyMzUtMDk4ZS00YTc0LWFmNGQtZWVjZDZmNjdmMjVk" rel="preload"/>
 <link as="script" crossorigin="anonymous" href="https://abs.twimg.com/responsive-web/client-web-legacy/i18n/en.b2193155.js" nonce="NDM2ZmUyMzUtMDk4ZS00YTc0LWFmNGQtZWVjZDZmNjdmMjVk" rel="preload"/>
 <link as="script" crossorigin="anonymous" href="https://abs.twimg.com/responsive-web/client-web-legacy/main.ed211455.js" nonce="NDM2ZmUyMzUtMDk4ZS00YTc0LWFmNGQtZWVjZDZmNjdmMjVk" rel="preload"/>
 <meta content="2231777543" property="fb:app_id"/>
 <meta content="Twitter" property="og:site_name"/>
 <meta content="V0yIS0Ec_o3Ii9KThrCoMCkwTYMMJ_JYx_RSaGhFYvw" name="google-site-verification"/>
 <link crossorigin="use-credentials" href="/manifest.json" rel="manifest"/>
 <link href="https://mobile.twitter.com/search?q=road" hreflang="x-default" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=ar" hreflang="ar" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=bg" hreflang="bg" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=bn" hreflang="bn" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=ca" hreflang="ca" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=cs" hreflang="cs" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=da" hreflang="da" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=de" hreflang="de" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=el" hreflang="el" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=en" hreflang="en" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=en-GB" hreflang="en-GB" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=es" hreflang="es" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=eu" hreflang="eu" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=fa" hreflang="fa" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=fi" hreflang="fi" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=tl" hreflang="tl" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=fr" hreflang="fr" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=ga" hreflang="ga" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=gl" hreflang="gl" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=gu" hreflang="gu" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=he" hreflang="he" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=hi" hreflang="hi" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=hr" hreflang="hr" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=hu" hreflang="hu" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=id" hreflang="id" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=it" hreflang="it" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=ja" hreflang="ja" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=kn" hreflang="kn" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=ko" hreflang="ko" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=mr" hreflang="mr" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=ms" hreflang="ms" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=nb" hreflang="nb" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=nl" hreflang="nl" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=pl" hreflang="pl" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=pt" hreflang="pt" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=ro" hreflang="ro" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=ru" hreflang="ru" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=sk" hreflang="sk" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=sr" hreflang="sr" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=sv" hreflang="sv" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=ta" hreflang="ta" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=th" hreflang="th" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=tr" hreflang="tr" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=uk" hreflang="uk" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=ur" hreflang="ur" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=vi" hreflang="vi" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=zh" hreflang="zh" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road&lang=zh-Hant" hreflang="zh-Hant" rel="alternate"/>
 <link href="https://mobile.twitter.com/search?q=road" rel="canonical"/>
 <link href="/opensearch.xml" rel="search" title="Twitter" type="application/opensearchdescription+xml"/>
 <link color="#1da1f2" href="https://abs.twimg.com/responsive-web/client-web-legacy/icon-svg.9e211f65.svg" rel="mask-icon" sizes="any"/>
 <link href="//abs.twimg.com/favicons/twitter.ico" rel="shortcut icon" type="image/x-icon"/>
 <link href="https://abs.twimg.com/responsive-web/client-web-legacy/icon-ios.8ea219d5.png" rel="apple-touch-icon" sizes="192x192">
  <meta content="yes" name="mobile-web-app-capable"/>
  <meta content="Twitter" name="apple-mobile-web-app-title"/>
  <meta content="white" name="apple-mobile-web-app-status-bar-style"/>
  <meta content="#ffffff" name="theme-color"/>
  <meta content="Ap6SMBNB0lQoXpXl4I9vyTJqJ7Y0X9tPd6Q6rN697iHdubQQxBcWHy21N3N7uEz7Ba5UKMbN+eLvDczBSbi27AsAAABfeyJvcmlnaW4iOiJodHRwczovL3R3aXR0ZXIuY29tOjQ0MyIsImZlYXR1cmUiOiJCYWRnaW5nIiwiZXhwaXJ5IjoxNTY0NTgyNzY2LCJpc1N1YmRvbWFpbiI6dHJ1ZX0=" http-equiv="origin-trial"/>
  <meta content="Apir4chqTX+4eFxKD+ErQlKRB/VtZ/dvnLfd9Y9Nenl5r1xJcf81alryTHYQiuUlz9Q49MqGXqyaiSmqWzHUqQwAAABneyJvcmlnaW4iOiJodHRwczovL3R3aXR0ZXIuY29tOjQ0MyIsImZlYXR1cmUiOiJDb250YWN0c01hbmFnZXIiLCJleHBpcnkiOjE1NzUwMzUyODMsImlzU3ViZG9tYWluIjp0cnVlfQ==" http-equiv="origin-trial"/>

    <form action="https://mobile.twitter.com/i/nojs_router?path=%2Fsearch%3Fq%3Droad" method="POST" style="background-color: #fff; position: fixed; top: 0; left: 0; right: 0; bottom: 0; z-index: 9999;">

       We've detected that JavaScript is disabled in your browser. Would you like to proceed to legacy Twitter?

       <button style="background-color: #1da1f2; border-radius: 100px; border: none; box-shadow: none; color: #fff; cursor: pointer; font-size: 14px; font-weight: bold; line-height: 20px; padding: 6px 16px;" type="submit">
        Yes
       </button>

    </form>

       <svg class="r-1p0dtai r-13gxpu9 r-4qtqp9 r-yyyyoo r-wy61xf r-1d2f490 r-ywje51 r-dnmrzs r-u8s1d r-zchlnj r-1plcrui r-ipm5af r-lrvibr r-1blnp2b" viewbox="0 0 24 24">
        <g>
         <path d="M23.643 4.937c-.835.37-1.732.62-2.675.733.962-.576 1.7-1.49 2.048-2.578-.9.534-1.897.922-2.958 1.13-.85-.904-2.06-1.47-3.4-1.47-2.572 0-4.658 2.086-4.658 4.66 0 .364.042.718.12 1.06-3.873-.195-7.304-2.05-9.602-4.868-.4.69-.63 1.49-.63 2.342 0 1.616.823 3.043 2.072 3.878-.764-.025-1.482-.234-2.11-.583v.06c0 2.257 1.605 4.14 3.737 4.568-.392.106-.803.162-1.227.162-.3 0-.593-.028-.877-.082.593 1.85 2.313 3.198 4.352 3.234-1.595 1.25-3.604 1.995-5.786 1.995-.376 0-.747-.022-1.112-.065 2.062 1.323 4.51 2.093 7.14 2.093 8.57 0 13.255-7.098 13.255-13.254 0-.2-.005-.402-.014-.602.91-.658 1.7-1.477 2.323-2.41z">
         </path>
        </g>
       </svg>

       <form action="" method="GET">

          Something went wrong, but don’t fret — let’s give it another shot.

         <input name="failedScript" type="hidden" value=""/>
         <input type="submit" value="Try again"/>

       </form>

 </link>

print(soup.find("div", {"class": "stream-container"}))None
https://twitter.com/i/api/1.1/friends/following/list.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&cursor=-1&user_id=1092295577478414336&count=3&with_total_count=true_____no_output_____https://twitter.com/i/api/2/timeline/profile/1092295577478414336.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&send_error_codes=true&simple_quoted_tweet=true&include_tweet_replies=false&count=20&cursor=HBaAwKvxopiBwSQAAA%3D%3D&userId=1092295577478414336&ext=mediaStats%2ChighlightedLabel_____no_output_____


# Notebook from LiekeVanSon/RedshiftEvolution_BBHMergers_WeightyMatter
Path: Code/6_MassDistribution_split_by_Z.ipynb

# The redshift evolution of the BBH merger rate: "a weighty matter"

## Figure 6: the contribution of different metallicities to the BH primary mass distribution
_____no_output_____## Imports and definitions:_____no_output_____

######################################
## Imports
import numpy as np
import h5py as h5

from astropy.table import vstack, Table, Column
import astropy.units as u
from astropy import constants as const

import matplotlib.pyplot as plt
import seaborn as sns
import astropy.units as u

from matplotlib import ticker, cm

from scipy import stats

# Chosen cosmology
from astropy.cosmology import WMAP9 as cosmo
from astropy.cosmology import z_at_value

# Extra python functions
import HelperFunctions as func

######################################
## locations
save_loc    =  '../plots/'
data_dir    = '../output/'

# This will be put in front of the name for every figure we safe
sim_save_str = 'N1e7_'

######################################
## PLOT setttings
plt.rc('font', family='serif')
from matplotlib import rc
import matplotlib
matplotlib.rcParams['mathtext.fontset'] = 'stix'
matplotlib.rcParams['font.family'] = 'STIXGeneral'
rc('font',**{'family':'sans-serif','sans-serif':['Helvetica']})
rc('text', usetex=True)
fsize, SMALL_SIZE, MEDIUM_SIZE, BIGGER_SIZE = 30,25,25,30
for obj in ['axes','xtick','ytick']:
    plt.rc(obj, labelsize=MEDIUM_SIZE)          # controls default text sizes
for obj in ['figure','axes']:
    plt.rc(obj, titlesize=BIGGER_SIZE)    # fontsize of the tick labels
plt.rc('font', size=MEDIUM_SIZE)          # controls default text sizes
plt.rc('legend', fontsize=SMALL_SIZE)    # legend fontsize

######################################
## Widescreen jupyter notebook
from IPython.core.display import display, HTML
display(HTML(""))_____no_output_____

# Read your data

the function `read_data` is part of `HelperFunctions.py` and reads the hdf5 file containing the BBH population data and merger rates. This is expensive memory-wise
Hence the columns specifically needed for this  Figure are loaded right below that

The Bool "DCO_mask" filters for BBHs:
1. with an inspiral time that is less than the age of the Universe
2. excludes systems that experienced a CE from a HG donor (i.e. the flag `Optimistic_CE == False`)
3. excludes systems that experienced RLOF immediately following a CE (i.e. the flag `Immediate_RLOF>CE == False`)

In other words, we treat 2. and 3. as stellar mergers and exclude them from the rest of our analysis

Lastly, we select merging BBHs using the `DCO_mask`, and aditionally exclude systems that evolve Chemically homogeneous.
_____no_output_____

####################################################
## Location of your data
File_location = data_dir+'COMPAS_Output_wWeights.h5'
print(File_location)

# Rate selects wich mergers you are interested in, using a specific SFRD
rate_key      = 'Rates_mu00.025_muz-0.05_alpha-1.77_sigma01.125_sigmaz0.05_zBinned'

####################################################
'''
The following function will read more information (generally useful for all plots in paper)
Warning, this is heavy on memory usage
Bare essential data will be read below
DCO, DCO_mask, redshifts, Average_SF_mass_needed, intrinsic_rate_density, intrinsic_rate_density_z0, = func.read_data(loc = File_location, rate_key = rate_key)
DCO.info()
'''

################################################
## Essential data for this plot
## Open hdf5 file and read relevant columns
File        = h5.File(File_location ,'r')

DCO = Table()
DCO['SEED']                  = File['DoubleCompactObjects']['SEED'][()]
DCO['Metallicity@ZAMS(1)']   = File['DoubleCompactObjects']['Metallicity@ZAMS(1)'][()]
DCO['CE_Event_Count']        = File['DoubleCompactObjects']['CE_Event_Count'][()]
DCO['M_moreMassive']         = np.maximum(File['DoubleCompactObjects']['Mass(1)'][()], File['DoubleCompactObjects']['Mass(2)'][()])

SYS_DCO_seeds_bool           = np.in1d(File['SystemParameters']['SEED'][()], DCO['SEED']) #Bool to point SYS to DCO
DCO['Stellar_Type@ZAMS(1)']  = File['SystemParameters']['Stellar_Type@ZAMS(1)'][SYS_DCO_seeds_bool]

DCO_mask                     = File[rate_key]['DCOmask'][()] # Mask from DCO to merging BBH
intrinsic_rate_density       = File[rate_key]['merger_rate'][()]
redshifts                    = File[rate_key]['redshifts'][()]

File.close()

################################################
# Bools to select merging BBHs w.o. CHE only
nonCHE_bool         = DCO['Stellar_Type@ZAMS(1)'] != 16
rate_nonCHE_bool    = DCO['Stellar_Type@ZAMS(1)'][DCO_mask] != 16

# Filter both the BBH table and the intrinsic rate data
merging_BBH         = DCO[DCO_mask * nonCHE_bool]
Red_intr_rate_dens  = intrinsic_rate_density[rate_nonCHE_bool, :]
../output/COMPAS_Output_wWeights.h5

# Look at contribution from different metallicities to the BH mass distribution_____no_output_____

# Distribution plot function
def plot_dist_by_redshift_Z(tables, x_key, crude_rate_densities, log = False, kde_width = 0.1,
                            fill=True, labels = ['',''],title_label = '',
                            metal_edges = [-3.5,-3,-2.5,-2,-1.5], annotate_stuff=True,print_MBHmax = True,
                            bins = np.arange(0,50,2), leg_args = {'bbox_to_anchor':(1.005, 0.85), 'loc':'upper left'},
                            savestr = '', save_plot=False, x_lim=(0,52.5), y_lim = (0,1000), linewidths = [5,5,5,5,5],
                            colors = ['#488496', '#73bdb3', '#e388b0'], linestyles = ['-', '--', '-.',':', '-'],
                            xlabel = '$M_{\mathrm{BH,1} \ } [\mathrm{M_{\odot}} ]$',
                            ylabel = r'$R_{\mathrm{BBH} \ \mathrm{[Gpc^{-3} \ yr^{-1} \ M_{\odot}^{-1}}]}$',):

    #########################################
    # Metal Labels
    metal_labels     = ['$(%s , %s]$'%(np.round(metal_edges[a],2),np.round(metal_edges[a+1],2) ) for a in range(0,len(metal_edges)-1) ]
    # add one bool for to include the total
    metal_labels.insert(0,'$\mathrm{All} \ Z$') #len(metal_bool_list)

    max_MBH_masses = []

    #########################################
    # Start plotting
    fig, ax = plt.subplots(figsize = (12, 8))

    zorders = [1,2,3,4,0]

    color_i = 0
    #########################################
    # Loop over all tables (i.e. if split by channel)
    for t, table in enumerate(tables):
        ##############################
        ## Average rate density per z-bin
        crude_rate_density = crude_rate_densities[t]

        #########################################
        # METAL bools
        table['logZ'] = np.log10(table['Metallicity@ZAMS(1)'])
        metal_bool_list = [np.logical_and(table['logZ'] > metal_edges[a], table['logZ']<= metal_edges[a+1]) for a in range(0,len(metal_edges)-1) ]
        # add one bool for to include the total
        metal_bool_list.insert(0,table['logZ'] < metal_edges[-1])

        #########################################
        # Loop over all metallicities
        for met, metal_bool in enumerate(metal_bool_list):
            print('t', t, 'met', met, metal_labels[met], 'color_i',color_i)
            ########################
            # X value and weight
            x_vals        = table[x_key][metal_bool]
            m_Weight      = crude_rate_density[metal_bool]

            ########################
            # Get the Hist
            hist, bin_edge = np.histogram(x_vals, weights = m_Weight, bins=bins)
            center_bins = (bin_edge[:-1] + bin_edge[1:])/2.
            # And the KDE
            kernel = stats.gaussian_kde(x_vals, bw_method= kde_width, weights=m_Weight)
            binwidth = np.diff(bin_edge)

            ########################
            # Add KDE
            x_KDE = np.arange(2.5,bins[-1],0.1)
            KDEy_vals = kernel(x_KDE)*sum(hist)*np.diff(bins)[0] #re-normalize the KDE

            # Find the max BH mass and metal bin
            Mbh_max = max(x_KDE[KDEy_vals > 1e-3])

            # Stuff for the legend handles
            if print_MBHmax:
                max_MBH_masses.append(Mbh_max)
                labelstr = metal_labels[met]
            else:
                labelstr = None
                if t == 0: # Only add lables to the first table
                    ax.plot((60,65), (-1, -1), lw=linewidths[met], color='darkgrey', ls = linestyles[met], zorder = 50,label = metal_labels[met] )

            # Plot the actual thing
            ax.plot(x_KDE, KDEy_vals, lw=linewidths[met], color=colors[color_i], ls = linestyles[met], zorder = 50,label = labelstr)

            if fill:
                if np.logical_and(print_MBHmax == False, met == 0):
                    ax.fill_between(x_KDE, y1=0, y2=KDEy_vals, color=colors[1 + t*5], alpha = 0.05, zorder = t+1) # Fill with the low Z color
                elif np.logical_and(print_MBHmax == True, met != 0):
                    ax.fill_between(x_KDE, y1=0, y2=KDEy_vals, color=colors[1 + t*5], alpha = 0.05, zorder = t+1) # Fill with the low Z color
                else:
                    print('dont fill the seperate dists')
            color_i += 1

    print(max_MBH_masses)
    # Annotating max BH masses by hand
    if print_MBHmax:
        #lowest Z == all
        ax.annotate('$ %s \mathrm{M_{\odot}}$'%(int(np.round(max_MBH_masses[1]))), (max_MBH_masses[1]-1,2e-1), xytext = (max_MBH_masses[1] ,5e-1) , color = colors[1], ha = 'left',
                    arrowprops=dict(arrowstyle= '-|>', color=colors[1],lw=3.5), zorder = 100)
        # one but lowest metals
        ax.annotate('$ %s \mathrm{M_{\odot}}$'%(int(np.round(max_MBH_masses[2]))), (max_MBH_masses[2]-1,2e-2), xytext = (max_MBH_masses[2]-6,1e-1) , color = colors[2], ha = 'left',
                    arrowprops=dict(arrowstyle= '-|>', color=colors[2],lw=3.5), zorder = 100)
        # one but highest metals
        ax.annotate('$ %s \mathrm{M_{\odot}}$'%(int(np.round(max_MBH_masses[3]))), (max_MBH_masses[3]-1,2e-2), xytext = (max_MBH_masses[3]-5,5e-2) , color = colors[3], ha = 'left',
                    arrowprops=dict(arrowstyle= '-|>', color=colors[3],lw=3.5), zorder = 100)
        # highest metals
        ax.annotate('$ %s \mathrm{M_{\odot}}$'%(int(np.round(max_MBH_masses[4]))), (max_MBH_masses[4]-1,2e-2), xytext = (max_MBH_masses[4]-5,5e-2) , color = colors[4], ha = 'left',
                    arrowprops=dict(arrowstyle= '-|>', color=colors[4],lw=3.5), zorder = 100)
    else:
        ax.annotate('$\mathrm{CE \ Channel}$', (10,4), xytext = (1,10) , color ='dimgrey', ha = 'left',
                    arrowprops=dict(arrowstyle= '-|>', color='dimgrey',lw=3.5), zorder = 100)
        ax.annotate('$\mathrm{Stable \ RLOF \ Channel}$', (26.5,3), xytext =(18, 15)  , color ='dimgrey', ha = 'left',
                    arrowprops=dict(arrowstyle= '-|>', color='dimgrey',lw=3.5), zorder = 100)

    #########################################
    # plot values
    ax.set_xlim(x_lim)
    ax.set_ylim(y_lim)
    leg = ax.legend(title= '$\log_{10} \mathrm{Metallicity}$ ',fontsize = 22,**leg_args) #+mbhmax_leg_string
    leg.set_zorder(100)
    leg._legend_box.align='right'

    ax.text(0.1, 0.925, title_label, ha = 'left', transform=ax.transAxes)

    ax.set_xlabel(xlabel, fontsize = 35)
    ax.set_ylabel(ylabel, fontsize = 30)
    ax.tick_params(axis='both', labelsize= 35)
    if log:
        ax.set_yscale('log')

    if save_plot:
        plt.savefig(save_loc+'/Mass_dist_by_metallicity'+savestr+'.pdf', bbox_inches='tight')
        plt.savefig(save_loc+'/png_versions/Mass_dist_by_metallicity'+savestr+'.png', bbox_inches='tight')

    plt.show()

    _____no_output_____

# Split by 4 metallicity bins_____no_output_____

z_bin_edges = np.array([0,0.5])

#########################################
# centers of redshif bins
center_z = (redshifts[:-1] + redshifts[1:])/2.
#Centers of your crude redshift bins
center_Crude_bins = (z_bin_edges[:-1] + z_bin_edges[1:])/2. # center points

#########################################
## Calculate average rate density per z-bin
crude_rate_density = func.get_crude_rate_density(Red_intr_rate_dens, redshifts, z_bin_edges)

#########################################
# labels based on z_bin_edges
labels     = ['$%s \leq \mathrm{merger \ redshift,} z < %s$'%(z_bin_edges[a],z_bin_edges[a+1]) for a in range(0,len(z_bin_edges)-1) ]

#########################################
kleuren_lijst = ['grey', '#4FC1E8', '#A0D568', '#FFCE54', '#ed6c55', '#6e6e6e']

#########################################
# !! Only works for one redshift bin
plot_dist_by_redshift_Z([merging_BBH], 'M_moreMassive', [crude_rate_density[:,0]],
                        metal_edges = [-4,-3,-2.5,-2,-1.5],fill=True, title_label = '',#'$\mathrm{Split \ by \ metallicity}$',
                        colors = kleuren_lijst, y_lim = (2e-2,7e1), linestyles = ['-', '-', '-','-', '-'],
                        ylabel = r'$d\overline{R}_{\mathrm{BBH}}/dM_{\mathrm{BH,1}} \ \mathrm{[Gpc^{-3} \ yr^{-1} \ M_{\odot}^{-1}]}$',
                        log = True, savestr = sim_save_str+'0z_leq2', save_plot=True, leg_args = {'bbox_to_anchor':(0.69, 0.49), 'loc':'lower left', 'ncol':1}) #(0.17, 0.015)

i_per_crude_bin 10.0
t 0 met 0 $\mathrm{All} \ Z$ color_i 0
dont fill the seperate dists
t 0 met 1 $(-4 , -3]$ color_i 1
t 0 met 2 $(-3 , -2.5]$ color_i 2
t 0 met 3 $(-2.5 , -2]$ color_i 3
t 0 met 4 $(-2 , -1.5]$ color_i 4
[47.80000000000004, 47.90000000000004, 43.80000000000004, 32.00000000000003, 18.000000000000014]

# Split by channel and by metallicity_____no_output_____

z_bin_edges = np.array([0,0.5])

##########################################
## Calculate average rate density per z-bin
crude_rate_density    = func.get_crude_rate_density(Red_intr_rate_dens[:,:], redshifts, z_bin_edges)
weights               = crude_rate_density[:,0]

# CE channel
BBH_CEchannel         = merging_BBH[merging_BBH['CE_Event_Count'] >0]
CEchannel_weights     = weights[merging_BBH['CE_Event_Count'] >0]

# stable RLOF channel
BBH_stablechannel     = merging_BBH[merging_BBH['CE_Event_Count'] == 0]
stablechannel_weights = weights[merging_BBH['CE_Event_Count'] == 0]

#########################################
# CE channel
CE_col                = np.vstack([sns.light_palette("#4c7d76",reverse=True, n_colors=1) ,sns.light_palette("#73bdb3",reverse=True, n_colors=4)[0:-1]])
# Stable channel
stable_col            = np.vstack([sns.light_palette("#8c546d",reverse=True, n_colors=1) ,sns.light_palette("#e388b0",reverse=True, n_colors=5)[0:-2]])
kleuren_lijst         = np.vstack([CE_col,stable_col])

#########################################
# !! Only works for one redshift bin
plot_dist_by_redshift_Z([BBH_CEchannel,BBH_stablechannel], 'M_moreMassive', [CEchannel_weights, stablechannel_weights],
                        metal_edges = [-4,-3,-2.5,-2] , fill=True,print_MBHmax = False,
                        colors = kleuren_lijst, y_lim = (2e-2,7e1), title_label = '$\mathrm{Split \ by \ metallicity \ and \ channel}$', linewidths = [5,5,5,5,3],
                        ylabel = r'$d\overline{R}_{\mathrm{BBH}}/dM_{\mathrm{BH,1}} \ \mathrm{[Gpc^{-3} \ yr^{-1} \ M_{\odot}^{-1}]}$',
                        log = True, savestr = sim_save_str+'channels_0z_leq2', save_plot=True,
                       leg_args = {'loc':'upper right'})

i_per_crude_bin 10.0
t 0 met 0 $\mathrm{All} \ Z$ color_i 0
t 0 met 1 $(-4 , -3]$ color_i 1
dont fill the seperate dists
t 0 met 2 $(-3 , -2.5]$ color_i 2
dont fill the seperate dists
t 0 met 3 $(-2.5 , -2]$ color_i 3
dont fill the seperate dists
t 1 met 0 $\mathrm{All} \ Z$ color_i 4
t 1 met 1 $(-4 , -3]$ color_i 5
dont fill the seperate dists
t 1 met 2 $(-3 , -2.5]$ color_i 6
dont fill the seperate dists
t 1 met 3 $(-2.5 , -2]$ color_i 7
dont fill the seperate dists
[]


# Notebook from m0n0l0c0/task-risk
Path: notebooks/covid-19-lda-notebook.ipynb

# Importing Key Libraries_____no_output_____

import os
import glob
import json
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer,CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation
from tqdm.notebook import tqdm_____no_output_____

# Reading in the data_____no_output_____

all_json_paths = glob.glob(f'/kaggle/input/CORD-19-research-challenge/comm_use_subset/comm_use_subset/*.json', recursive=True)
len(all_json_paths)_____no_output_____#source:https://www.kaggle.com/amogh05/cord-19-eda-question-topic-modeling-starter
#add more vars as required

class FileReader:
    def __init__(self, file_path):
        with open(file_path) as file:
            content = json.load(file)
            self.paper_id = content['paper_id']
            self.title = content['metadata']['title']
            self.abstract = []
            self.body_text = []
            self.biblio = []
            self.biblio_doi = []
            self.img_tables = []
            self.back_matter = []

            # Abstract
            for entry in content['abstract']:
                self.abstract.append(entry['text'])
            self.abstract = '\n'.join(self.abstract)

            # Body text
            for entry in content['body_text']:
                self.body_text.append(entry['text'])
            self.body_text = '\n'.join(self.body_text)

            # bibliography
            for bib_id, details in content['bib_entries'].items():
                self.biblio.append(details['title'])
                self.biblio_doi.append(details['other_ids'])
            self.biblio = '\n'.join(self.biblio)
            #self.biblio_doi = '\n'.join(self.biblio_doi)

            #img and table references
            for ref_id,details in content['ref_entries'].items():
                self.img_tables.append(details['text'])
            self.img_tables = '\n'.join(self.img_tables)

            #back_matter
            for entry in content['back_matter']:
                self.back_matter.append(entry['text'])
            self.back_matter = '\n'.join(self.back_matter)

    def __repr__(self):
        return f'{self.paper_id}:{self.title}-{self.abstract}... {self.body_text}...{self.biblio}...{self.img_tables}...{self.back_matter}'

dict_ = {'paper_id': [],'title':[], 'abstract': [], 'body_text': [],'biblio':[],'bidoi':[],'img_tables':[]}
for idx, entry in enumerate(all_json_paths):
    if idx % (len(all_json_paths) // 10) == 0:
        print(f'Processing index: {idx} of {len(all_json_paths)}')
    #print(entry)
    content = FileReader(entry)
    dict_['paper_id'].append(content.paper_id)
    dict_['title'].append(content.title)
    dict_['abstract'].append(content.abstract)
    dict_['body_text'].append(content.body_text)
    dict_['biblio'].append(content.biblio)
    dict_['img_tables'].append(content.img_tables)
df_covid = pd.DataFrame(dict_, columns=['paper_id', 'abstract', 'body_text','biblio','img_tables'])
df_covid.head()

#identify dups
df_covid.describe(include='all')

df_covid.drop_duplicates(['abstract'], inplace=True)
df_covid.describe(include='all')Processing index: 0 of 9315
Processing index: 931 of 9315
Processing index: 1862 of 9315
Processing index: 2793 of 9315
Processing index: 3724 of 9315
Processing index: 4655 of 9315
Processing index: 5586 of 9315
Processing index: 6517 of 9315
Processing index: 7448 of 9315
Processing index: 8379 of 9315
Processing index: 9310 of 9315
df_covid['all_text'] = df_covid['abstract'] + '' + df_covid['body_text'] _____no_output_____

# Find synonyms_____no_output_____

#This approach does not work well:  defining a list is better
import nltk
from nltk.corpus import wordnet
synonyms = []

for syn in wordnet.synsets('exposure'):
    for l in syn.lemmas():
        synonyms.append(l.name())
        if l.antonyms():
            antonyms.append(l.antonyms()[0].name())
print(set(synonyms)){'vulnerability', 'photo', 'picture', 'photograph', 'pic', 'exposure'}
#defining a list better
stage_syn_list = ['exposure','vulnerability','vulnerable'] _____no_output_____

# Filter By Stage_____no_output_____

disease_stage_list = ['exposure' ,'acquisition' ,'progression', 'development' ,'complications' ,'fatality', 'disability']_____no_output_____def filterByStage(text,stage_syn_list):
    paper_list =[]

    for idx_num,row in text.iterrows():
        for stage in stage_syn_list:
            stage_found = False
            if stage in row.all_text.split():
                stage_found = True
            else:
                pass
        if stage_found==True:
            paper_list.append(row.all_text)
    return paper_list_____no_output_____stage_dict = {}

stage = disease_stage_list[0]

stage_dict[stage] = filterByStage(df_covid,stage_syn_list)_____no_output_____#for later ease while searching for relevant papers
exposure = pd.DataFrame(stage_dict[stage])_____no_output_____

# NLP Starts_____no_output_____

!pip install scispacy scipy
!pip install https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.2.4/en_core_sci_lg-0.2.4.tar.gz
!pip install tqdm -U
!pip install spacy-langdetectCollecting scispacy
  Downloading scispacy-0.2.4.tar.gz (38 kB)
Requirement already satisfied: scipy in /opt/conda/lib/python3.6/site-packages (1.4.1)
Requirement already satisfied: spacy>=2.2.1 in /opt/conda/lib/python3.6/site-packages (from scispacy) (2.2.3)
Collecting awscli
  Downloading awscli-1.18.31-py2.py3-none-any.whl (3.0 MB)
     |████████████████████████████████| 3.0 MB 3.0 MB/s
Requirement already satisfied: conllu in /opt/conda/lib/python3.6/site-packages (from scispacy) (1.3.1)
Requirement already satisfied: numpy in /opt/conda/lib/python3.6/site-packages (from scispacy) (1.18.1)
Requirement already satisfied: joblib in /opt/conda/lib/python3.6/site-packages (from scispacy) (0.14.1)
Collecting nmslib>=1.7.3.6
  Downloading nmslib-2.0.5-cp36-cp36m-manylinux2010_x86_64.whl (12.9 MB)
     |████████████████████████████████| 12.9 MB 16.3 MB/s
Requirement already satisfied: scikit-learn>=0.20.3 in /opt/conda/lib/python3.6/site-packages (from scispacy) (0.22.2.post1)
Collecting pysbd
  Downloading pysbd-0.2.3-py3-none-any.whl (24 kB)
Requirement already satisfied: requests<3.0.0,>=2.13.0 in /opt/conda/lib/python3.6/site-packages (from spacy>=2.2.1->scispacy) (2.22.0)
Requirement already satisfied: plac<1.2.0,>=0.9.6 in /opt/conda/lib/python3.6/site-packages (from spacy>=2.2.1->scispacy) (0.9.6)
Requirement already satisfied: wasabi<1.1.0,>=0.4.0 in /opt/conda/lib/python3.6/site-packages (from spacy>=2.2.1->scispacy) (0.6.0)
Requirement already satisfied: srsly<1.1.0,>=0.1.0 in /opt/conda/lib/python3.6/site-packages (from spacy>=2.2.1->scispacy) (1.0.2)
Requirement already satisfied: blis<0.5.0,>=0.4.0 in /opt/conda/lib/python3.6/site-packages (from spacy>=2.2.1->scispacy) (0.4.1)
Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /opt/conda/lib/python3.6/site-packages (from spacy>=2.2.1->scispacy) (1.0.2)
Requirement already satisfied: catalogue<1.1.0,>=0.0.7 in /opt/conda/lib/python3.6/site-packages (from spacy>=2.2.1->scispacy) (1.0.0)
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /opt/conda/lib/python3.6/site-packages (from spacy>=2.2.1->scispacy) (2.0.3)
Requirement already satisfied: thinc<7.4.0,>=7.3.0 in /opt/conda/lib/python3.6/site-packages (from spacy>=2.2.1->scispacy) (7.3.1)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.6/site-packages (from spacy>=2.2.1->scispacy) (45.2.0.post20200210)
Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /opt/conda/lib/python3.6/site-packages (from spacy>=2.2.1->scispacy) (3.0.2)
Collecting botocore==1.15.31
  Downloading botocore-1.15.31-py2.py3-none-any.whl (6.0 MB)
     |████████████████████████████████| 6.0 MB 47.3 MB/s
Requirement already satisfied: docutils<0.16,>=0.10 in /opt/conda/lib/python3.6/site-packages (from awscli->scispacy) (0.15.2)
Requirement already satisfied: colorama<0.4.4,>=0.2.5; python_version != "3.4" in /opt/conda/lib/python3.6/site-packages (from awscli->scispacy) (0.4.3)
Collecting rsa<=3.5.0,>=3.1.2
  Downloading rsa-3.4.2-py2.py3-none-any.whl (46 kB)
     |████████████████████████████████| 46 kB 3.3 MB/s
Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.6/site-packages (from awscli->scispacy) (0.3.3)
Requirement already satisfied: PyYAML<5.4,>=3.10; python_version != "3.4" in /opt/conda/lib/python3.6/site-packages (from awscli->scispacy) (5.3)
Requirement already satisfied: pybind11>=2.2.3 in /opt/conda/lib/python3.6/site-packages (from nmslib>=1.7.3.6->scispacy) (2.4.3)
Requirement already satisfied: psutil in /opt/conda/lib/python3.6/site-packages (from nmslib>=1.7.3.6->scispacy) (5.6.7)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.6/site-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.1->scispacy) (1.24.3)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/lib/python3.6/site-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.1->scispacy) (3.0.4)
Requirement already satisfied: idna<2.9,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.1->scispacy) (2.8)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.6/site-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.1->scispacy) (2019.11.28)
Requirement already satisfied: importlib-metadata>=0.20; python_version < "3.8" in /opt/conda/lib/python3.6/site-packages (from catalogue<1.1.0,>=0.0.7->spacy>=2.2.1->scispacy) (1.5.0)
Requirement already satisfied: tqdm<5.0.0,>=4.10.0 in /opt/conda/lib/python3.6/site-packages (from thinc<7.4.0,>=7.3.0->spacy>=2.2.1->scispacy) (4.42.0)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.6/site-packages (from botocore==1.15.31->awscli->scispacy) (0.9.5)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /opt/conda/lib/python3.6/site-packages (from botocore==1.15.31->awscli->scispacy) (2.8.0)
Requirement already satisfied: pyasn1>=0.1.3 in /opt/conda/lib/python3.6/site-packages (from rsa<=3.5.0,>=3.1.2->awscli->scispacy) (0.4.8)
Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.6/site-packages (from importlib-metadata>=0.20; python_version < "3.8"->catalogue<1.1.0,>=0.0.7->spacy>=2.2.1->scispacy) (2.2.0)
Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.6/site-packages (from python-dateutil<3.0.0,>=2.1->botocore==1.15.31->awscli->scispacy) (1.14.0)
Building wheels for collected packages: scispacy
  Building wheel for scispacy (setup.py) ... - \ done
  Created wheel for scispacy: filename=scispacy-0.2.4-py3-none-any.whl size=35203 sha256=7488a7b3aed301c771550773180a058b18acd8887186860502f8a9b52f2632fe
  Stored in directory: /root/.cache/pip/wheels/80/01/69/37a2ab4f9b61773c187d83257ffb31365a5ad57e7779ae5e92
Successfully built scispacy
ERROR: allennlp 0.9.0 has requirement spacy<2.2,>=2.1.0, but you'll have spacy 2.2.3 which is incompatible.
Installing collected packages: botocore, rsa, awscli, nmslib, pysbd, scispacy
  Attempting uninstall: botocore
    Found existing installation: botocore 1.15.13
    Uninstalling botocore-1.15.13:
      Successfully uninstalled botocore-1.15.13
  Attempting uninstall: rsa
    Found existing installation: rsa 4.0
    Uninstalling rsa-4.0:
      Successfully uninstalled rsa-4.0
Successfully installed awscli-1.18.31 botocore-1.15.31 nmslib-2.0.5 pysbd-0.2.3 rsa-3.4.2 scispacy-0.2.4
Collecting https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.2.4/en_core_sci_lg-0.2.4.tar.gz
  Downloading https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.2.4/en_core_sci_lg-0.2.4.tar.gz (500.6 MB)
     |████████████████████████████████| 500.6 MB 9.8 kB/s
Requirement already satisfied: spacy>=2.2.1 in /opt/conda/lib/python3.6/site-packages (from en-core-sci-lg==0.2.4) (2.2.3)
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /opt/conda/lib/python3.6/site-packages (from spacy>=2.2.1->en-core-sci-lg==0.2.4) (2.0.3)
Requirement already satisfied: thinc<7.4.0,>=7.3.0 in /opt/conda/lib/python3.6/site-packages (from spacy>=2.2.1->en-core-sci-lg==0.2.4) (7.3.1)
Requirement already satisfied: srsly<1.1.0,>=0.1.0 in /opt/conda/lib/python3.6/site-packages (from spacy>=2.2.1->en-core-sci-lg==0.2.4) (1.0.2)
Requirement already satisfied: blis<0.5.0,>=0.4.0 in /opt/conda/lib/python3.6/site-packages (from spacy>=2.2.1->en-core-sci-lg==0.2.4) (0.4.1)
Requirement already satisfied: requests<3.0.0,>=2.13.0 in /opt/conda/lib/python3.6/site-packages (from spacy>=2.2.1->en-core-sci-lg==0.2.4) (2.22.0)
Requirement already satisfied: plac<1.2.0,>=0.9.6 in /opt/conda/lib/python3.6/site-packages (from spacy>=2.2.1->en-core-sci-lg==0.2.4) (0.9.6)
Requirement already satisfied: numpy>=1.15.0 in /opt/conda/lib/python3.6/site-packages (from spacy>=2.2.1->en-core-sci-lg==0.2.4) (1.18.1)
Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /opt/conda/lib/python3.6/site-packages (from spacy>=2.2.1->en-core-sci-lg==0.2.4) (3.0.2)
Requirement already satisfied: wasabi<1.1.0,>=0.4.0 in /opt/conda/lib/python3.6/site-packages (from spacy>=2.2.1->en-core-sci-lg==0.2.4) (0.6.0)
Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /opt/conda/lib/python3.6/site-packages (from spacy>=2.2.1->en-core-sci-lg==0.2.4) (1.0.2)
Requirement already satisfied: catalogue<1.1.0,>=0.0.7 in /opt/conda/lib/python3.6/site-packages (from spacy>=2.2.1->en-core-sci-lg==0.2.4) (1.0.0)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.6/site-packages (from spacy>=2.2.1->en-core-sci-lg==0.2.4) (45.2.0.post20200210)
Requirement already satisfied: tqdm<5.0.0,>=4.10.0 in /opt/conda/lib/python3.6/site-packages (from thinc<7.4.0,>=7.3.0->spacy>=2.2.1->en-core-sci-lg==0.2.4) (4.42.0)
Requirement already satisfied: idna<2.9,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.1->en-core-sci-lg==0.2.4) (2.8)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.6/site-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.1->en-core-sci-lg==0.2.4) (2019.11.28)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.6/site-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.1->en-core-sci-lg==0.2.4) (1.24.3)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/lib/python3.6/site-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.1->en-core-sci-lg==0.2.4) (3.0.4)
Requirement already satisfied: importlib-metadata>=0.20; python_version < "3.8" in /opt/conda/lib/python3.6/site-packages (from catalogue<1.1.0,>=0.0.7->spacy>=2.2.1->en-core-sci-lg==0.2.4) (1.5.0)
Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.6/site-packages (from importlib-metadata>=0.20; python_version < "3.8"->catalogue<1.1.0,>=0.0.7->spacy>=2.2.1->en-core-sci-lg==0.2.4) (2.2.0)
Building wheels for collected packages: en-core-sci-lg
  Building wheel for en-core-sci-lg (setup.py) ... - \ | / - \ | / - \ | / - \ done
  Created wheel for en-core-sci-lg: filename=en_core_sci_lg-0.2.4-py3-none-any.whl size=501343161 sha256=ddd54d846138dcc1fe929574c4e2b7106ed376ed104fd7c22db58ea7602aeb38
  Stored in directory: /root/.cache/pip/wheels/43/12/d5/dd85b5deab7738797c4d5358672df3616dda39acf570a3ef96
Successfully built en-core-sci-lg
Installing collected packages: en-core-sci-lg
Successfully installed en-core-sci-lg-0.2.4
Collecting tqdm
  Downloading tqdm-4.44.1-py2.py3-none-any.whl (60 kB)
     |████████████████████████████████| 60 kB 101 kB/s
ERROR: pandas-profiling 2.5.0 has requirement tqdm==4.42.0, but you'll have tqdm 4.44.1 which is incompatible.
ERROR: hyperopt 0.2.3 has requirement networkx==2.2, but you'll have networkx 2.4 which is incompatible.
ERROR: allennlp 0.9.0 has requirement spacy<2.2,>=2.1.0, but you'll have spacy 2.2.3 which is incompatible.
Installing collected packages: tqdm
  Attempting uninstall: tqdm
    Found existing installation: tqdm 4.42.0
    Uninstalling tqdm-4.42.0:
      Successfully uninstalled tqdm-4.42.0
Successfully installed tqdm-4.44.1
Collecting spacy-langdetect
  Downloading spacy_langdetect-0.1.2-py3-none-any.whl (5.0 kB)
Requirement already satisfied: langdetect==1.0.7 in /opt/conda/lib/python3.6/site-packages (from spacy-langdetect) (1.0.7)
Requirement already satisfied: pytest in /opt/conda/lib/python3.6/site-packages (from spacy-langdetect) (5.0.1)
Requirement already satisfied: six in /opt/conda/lib/python3.6/site-packages (from langdetect==1.0.7->spacy-langdetect) (1.14.0)
Requirement already satisfied: py>=1.5.0 in /opt/conda/lib/python3.6/site-packages (from pytest->spacy-langdetect) (1.8.1)
Requirement already satisfied: packaging in /opt/conda/lib/python3.6/site-packages (from pytest->spacy-langdetect) (20.1)
Requirement already satisfied: attrs>=17.4.0 in /opt/conda/lib/python3.6/site-packages (from pytest->spacy-langdetect) (19.3.0)
Requirement already satisfied: more-itertools>=4.0.0 in /opt/conda/lib/python3.6/site-packages (from pytest->spacy-langdetect) (8.2.0)
Requirement already satisfied: atomicwrites>=1.0 in /opt/conda/lib/python3.6/site-packages (from pytest->spacy-langdetect) (1.3.0)
Requirement already satisfied: pluggy<1.0,>=0.12 in /opt/conda/lib/python3.6/site-packages (from pytest->spacy-langdetect) (0.13.1)
Requirement already satisfied: importlib-metadata>=0.12 in /opt/conda/lib/python3.6/site-packages (from pytest->spacy-langdetect) (1.5.0)
Requirement already satisfied: wcwidth in /opt/conda/lib/python3.6/site-packages (from pytest->spacy-langdetect) (0.1.8)
Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.6/site-packages (from packaging->pytest->spacy-langdetect) (2.4.6)
Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.6/site-packages (from importlib-metadata>=0.12->pytest->spacy-langdetect) (2.2.0)
Installing collected packages: spacy-langdetect
Successfully installed spacy-langdetect-0.1.2
import spacy
import en_core_sci_lg
nlp = en_core_sci_lg.load()

# We also need to detect language, or else we'll be parsing non-english text
# as if it were English.
from spacy_langdetect import LanguageDetector
nlp.add_pipe(LanguageDetector(), name='language_detector', last=True)

nlp.max_length=2000000

# New stop words list
customize_stop_words = [
    'doi', 'preprint', 'copyright', 'peer', 'reviewed', 'org', 'https', 'et', 'al', 'author', 'figure',
    'rights', 'reserved', 'permission', 'used', 'using', 'biorxiv', 'fig', 'fig.', 'al.',
    'di', 'la', 'il', 'del', 'le', 'della', 'dei', 'delle', 'una', 'da',  'dell',  'non', 'si'
]

# Mark them as stop words
for w in customize_stop_words:
    nlp.vocab[w].is_stop = True_____no_output_____

# LDA : Kaggle Notebook Approach
https://www.kaggle.com/danielwolffram/topic-modeling-finding-related-articles_____no_output_____

def spacy_tokenizer(sentence):
    return [word.lemma_ for word in nlp(sentence) if not (word.like_num or word.is_stop or word.is_punct or word.is_space or len(word)==1)]
    # remove numbers (e.g. from references [1], etc.)_____no_output_____

### Create vector representation of data_____no_output_____

tf_vectorizer = CountVectorizer(tokenizer = spacy_tokenizer, max_features=800000)
tf = tf_vectorizer.fit_transform(tqdm(stage_dict[stage]))

print(tf.shape)

import joblib
joblib.dump(tf_vectorizer, '/kaggle/working/tf_vectorizer.csv')
joblib.dump(tf, '/kaggle/working/tf.csv')_____no_output_____

### LDA_____no_output_____

lda_tf = LatentDirichletAllocation(n_components=50, random_state=0)
lda_tf.fit(tf)
joblib.dump(lda_tf, '/kaggle/working/lda.csv')_____no_output_____

### Discover Topics_____no_output_____

tfidf_feature_names = tf_vectorizer.get_feature_names()

def print_top_words(model, feature_names, n_top_words):
    for topic_idx, topic in enumerate(model.components_):
        message = "\nTopic #%d: " % topic_idx
        message += " ".join([feature_names[i]
                             for i in topic.argsort()[:-n_top_words - 1:-1]])
        print(message)
    print()

print_top_words(lda_tf, tfidf_feature_names, 25)
Topic #0: specie host wildlife bat virus flavivirus model high forest wnv study datum predict habitat number body include animal individual probability human identify devil denv distribution

Topic #1: cardiovascular mortality air pm pollution pollutant effect day significant increase lag impact daily non-accidental μg/m concentration particulate er elderly change exposure cumulative haze single-day row

Topic #2: disease network change country increase dengue population infectious case area high node climate human health spread africa time global include travel year specie number new

Topic #3: protein cell virus viral antibody binding residue mutation show peptide epitope domain membrane site glycoprotein mab virion hiv-1 activity study surface interaction bind fusion target

Topic #4: adam17 upr ace2 protein receptor study mouse increase human strain effect level epitope stroke brain show ii er stress gii.4 role ang-(1 induce ang vsv

Topic #5: datum suicide descriptor s. set study report compound h. influenzae om model pneumoniae activity antibiotic test identify serotype aom incentive isolate bacteria charcoal patient research

Topic #6: rate contact temperature study country household population increase age high influenza year change policy disease pathogen specie individual find transmission datum different infection number virus

Topic #7: sequence g4 region gene structure recombination dna recombinant breakpoint rsv household form strand genome adc7αrsv show anti-rsv find identify number nucleotide administration vector pattern distribution

Topic #8: disease human risk research virus animal intervention system vaccine infectious zoonotic vaccination health development pathogen global new approach potential collaboration production problem knowledge increase threat

Topic #9: zikv infection study cell control ibv strain sample fork asthma show day fruit fetal vegetable blood intake human offspring group table zika macrophage report level

Topic #10: group animal level cssa number follicle system resident worry age show high low study bid minnow recipient increase day kong hong doc result rare nh

Topic #11: mouse virus cell ifnar infection animal viral titer rig-i vaccine protein show antibody serum day peptide response particle level wt result rna mda5 challenge dose

Topic #12: virus aptamer lectin influenza study protein group target stress ha show plant mbt antibody report aptasensor monoclonal calf detect result viral detection bcov program binding

Topic #13: ccr5 cd4 hev mouse cell je −/− foxp3 cn tregs refugee infection /+ dpi veterinary child veterinarian progression result number hi disease ic response tb

Topic #14: health public system surveillance disease country outbreak information response datum include need service national provide report care case plan medical local patient practice global emergency

Topic #15: model number estimate case disease transmission datum time individual epidemic population method infection parameter value rate network contact probability distribution death result different average outbreak

Topic #16: cell piglet group microglia mouse neutrophil control hotspot study brain increase number show dpi bacteria result day nrf2 lesion e. pr8 pcv2 coli microglial tissue

Topic #17: rna cell disruption nucleolus value isg15 temperature rrna uniformity method polyamine nucleolar standard irt study result base alu dna stability mrtd different docetaxel high show

Topic #18: risk group mer disease perceive high perception show control labor behavior market al study decrease compare sow vulnerability trust effect epidemic pedv result model factor

Topic #19: antibody show infection domain cell result scfv expression datum trial compare analysis sequence treatment observe am binding usa t. cc-11050 study control parasite macaque nano-7

Topic #20: study influenza patient case infection disease risk report group high respiratory participant rate year include age hospital vaccination severe virus test datum illness death analysis

Topic #21: surveillance datum system influenza nanoparticle drug sample veterinary result control ili medicine test environment study effect show performance site material animal activity high score cause

Topic #22: cell infection response mouse cd8 immune expression cn viral cytokine antigen cd4 role induce tissue increase memory ms level virus disease ebv control function signaling

Topic #23: mers-cov pedv human infection case virus sars-cov animal camel pig outbreak contact disease study report day transmission high mer respiratory aged coronavirus strain antibody epidemic

Topic #24: sgta mrsa result sdab detection protein assay ata scfvs test antibody patient np show scfv cell sample ward target site isnv yoph study arm method

Topic #25: herd respiratory hyponatremia variable bigg analysis high cow study disease infection bhv-1 h3n2 human level influenza event seed number h1n1 virus bovine associate sequence find

Topic #26: hrv strain hrv-qpm a2 receptor vp1 icam-1 sequence site hev specie hrvs group structure rhinovirus datum predict deletion structural major minor acid amino identify divergent

Topic #27: patient datum tb mask care study pattern child claim provide use treatment cybercare flu hospital family trial telemedicine health include ward code region technology eps

Topic #28: study respondent report review care score include high family health questionnaire question triage sample datum knowledge dengue animal number work level time population meat nurse

Topic #29: bat cell gp120 dc virus study expression apoptosis atmuv infection response human modc immune specie control apmv-1 duck bent-winged level serum ifn increase dc-sign chicken

Topic #30: participant pandemic health study nurse care risk work hcws response disease emergency influenza practice knowledge experience report staff plan need management preparedness measure lack perception

Topic #31: diabete type piglet emodin group mer incidence tregs allograft year people pedv pn-sia49 show h7n9 ha anxiety study isolate cmah csa hcp ko recipient isolation

Topic #32: mouse lung treatment airway study nicu infection model show l. asthma cell reduce bacterial animal macrophage compare level sample effect response new gene day increase

Topic #33: hospital contact datum outbreak patient model system stream ed network group mer surge tace year shift sar index disease transmission target selection visit droplet bead

Topic #34: infection disease pathogen bee host wild population human specie parasite greenhouse distress animal sars-cov high study prevalence infected infectious bumble commercial respiratory psychological c. b.

Topic #35: organ cancer rdna transplant prostate overseas sexual transplantation genome gene partner prisoner increase china number hcov-nl63 death human man recipient transcription domestic kt rpm lt

Topic #36: virus sample sequence viral rna detect genome study detection pcr specie analysis strain human pathogen isolate acid assay positive host gene identify sequencing infection respiratory

Topic #37: cuprizone demyelination alcohol remyelination week mouse model oligodendrocyte cell microglia c200 b. study brain nps opc animal tio astrocyte factor effect expression human suggest group

Topic #38: vaccine vaccination behaviour pandemic public infection perceive people flu population participant study risk swine respiratory social intention perception concern ph1n1 control government health measure group

Topic #39: health disaster research study review article resilience emergency include identify public workforce theme literature community country outbreak preparedness need system model lri work develop result

Topic #40: intubation lynx robot prv system pig lma rate iberian wild study success perform time rrais plate infection tongue medical pre-hospital depressor laryngoscope eti tube group

Topic #41: maternal pneumonia pregnancy child vaccine immunization study dental infant pregnant woman tetraspanin neonatal factor cd151 cd63 health birth infection carie peptide hpv pneumococcal vaccination associate

Topic #42: cell virus viral infection protein host replication rna gene expression response antiviral immune type human ifn activity cellular show influenza pathway increase result mechanism receptor

Topic #43: child infection respiratory rsv study year virus age patient case clinical infant viral report pathogen pneumonia include old pneumoniae month pertussis young high symptom adult

Topic #44: health policy population social disease datum public support include regional country international approach level development global change program project initiative migrant economic region increase access

Topic #45: care mucus gene sink peacock analysis condor sample bird study show coli pathogen result e. viscosity disinfectant specie eol different model high perform ppe critical

Topic #46: pathogen air infection ventilation airborne aerosol patient droplet high cough rate system model bed increase concentration different room study result method city human respiratory condition

Topic #47: patient study group icu day mortality case high result clinical infection level treatment therapy increase sepsis hospital associate include severe disease care rate method admission

Topic #48: cell t. gondii cancer dvgs sev immune ln aa ua group production protein effect include gondii-infected acid expression infection activity increase restriction show compare cantell

Topic #49: vaccine virus infection human cell antibody study protein disease response development camel viral immune strain antigen neutralize cause hiv activity high include mab bst2 result

### Create Topic Distance Matrix_____no_output_____

topic_dist = pd.DataFrame(lda_tf.transform(tf))
topic_dist.to_csv('/kaggle/working/topic_dist.csv', index=False)_____no_output_____topic_dist.head()_____no_output_____

### Get Paper Related to Stage of Disease_____no_output_____

#get most similar paper
from scipy.spatial import distance
def get_k_nearest_docs(doc_dist, k=5, lower=1950, upper=2020, only_covid19=False, get_dist=False):
    '''
    doc_dist: topic distribution (sums to 1) of one article

    Returns the index of the k nearest articles (as by Jensen–Shannon divergence in topic space).
    '''

    #relevant_time = df.publish_year.between(lower, upper)

   # if only_covid19:
   #     is_covid19_article = df.body_text.str.contains('COVID-19|SARS-CoV-2|2019-nCov|SARS Coronavirus 2|2019 Novel Coronavirus') #TODO: move outside
   #     topic_dist_temp = topic_dist[relevant_time & is_covid19_article]
   #
   # else:
    #    topic_dist_temp = topic_dist[relevant_time]

    distances = topic_dist.apply(lambda x: distance.jensenshannon(x, doc_dist), axis=1)
    k_nearest = distances[distances != 0].nsmallest(n=k).index

    if get_dist:
        k_distances = distances[distances != 0].nsmallest(n=k)
        return k_nearest, k_distances
    else:
        return k_nearest

#d = get_k_nearest_docs(topic_dist[1].iloc[0],k=10)_____no_output_____def relevant_articles(df,tasks, k=3, lower=1950, upper=2020, only_covid19=False):
    tasks = [tasks] if type(tasks) is str else tasks

    tasks_tf = tf_vectorizer.transform(tasks)
    tasks_topic_dist = pd.DataFrame(lda_tf.transform(tasks_tf))

    for index, bullet in enumerate(tasks):
        print(bullet)
        recommended = get_k_nearest_docs(tasks_topic_dist.iloc[index], k, lower, upper, only_covid19)
        print(list(recommended))
        recommended = df.iloc[recommended] #stage_dict[stage][','.join(list(recommended))]#
    return recommended_____no_output_____task = ['exposure']
relevant_articles(exposure,task,k=10) #k is the number of relevant articlesexposure
[101, 463, 129, 321, 3, 199, 341, 99, 2, 513]


# Notebook from QPoland/basics-of-quantum-computing-pl
Path: bronze/B03_One_Bit.ipynb

            prepared by Abuzer Yakaryilmaz (QLatvia)

_____no_output_____

This cell contains some macros. If there is a problem with displaying mathematical formulas, please run this cell to load these macros.

$ \newcommand{\bra}[1]{\langle #1|} $
$ \newcommand{\ket}[1]{|#1\rangle} $
$ \newcommand{\braket}[2]{\langle #1|#2\rangle} $
$ \newcommand{\dot}[2]{ #1 \cdot #2} $
$ \newcommand{\biginner}[2]{\left\langle #1,#2\right\rangle} $
$ \newcommand{\mymatrix}[2]{\left( \begin{array}{#1} #2\end{array} \right)} $
$ \newcommand{\myvector}[1]{\mymatrix{c}{#1}} $
$ \newcommand{\myrvector}[1]{\mymatrix{r}{#1}} $
$ \newcommand{\mypar}[1]{\left( #1 \right)} $
$ \newcommand{\mybigpar}[1]{ \Big( #1 \Big)} $
$ \newcommand{\sqrttwo}{\frac{1}{\sqrt{2}}} $
$ \newcommand{\dsqrttwo}{\dfrac{1}{\sqrt{2}}} $
$ \newcommand{\onehalf}{\frac{1}{2}} $
$ \newcommand{\donehalf}{\dfrac{1}{2}} $
$ \newcommand{\hadamard}{ \mymatrix{rr}{ \sqrttwo & \sqrttwo \\ \sqrttwo & -\sqrttwo }} $
$ \newcommand{\vzero}{\myvector{1\\0}} $
$ \newcommand{\vone}{\myvector{0\\1}} $
$ \newcommand{\vhadamardzero}{\myvector{ \sqrttwo \\  \sqrttwo } } $
$ \newcommand{\vhadamardone}{ \myrvector{ \sqrttwo \\ -\sqrttwo } } $
$ \newcommand{\myarray}[2]{ \begin{array}{#1}#2\end{array}} $
$ \newcommand{\X}{ \mymatrix{cc}{0 & 1 \\ 1 & 0}  } $
$ \newcommand{\Z}{ \mymatrix{rr}{1 & 0 \\ 0 & -1}  } $
$ \newcommand{\Htwo}{ \mymatrix{rrrr}{ \frac{1}{2} & \frac{1}{2} & \frac{1}{2} & \frac{1}{2} \\ \frac{1}{2} & -\frac{1}{2} & \frac{1}{2} & -\frac{1}{2} \\ \frac{1}{2} & \frac{1}{2} & -\frac{1}{2} & -\frac{1}{2} \\ \frac{1}{2} & -\frac{1}{2} & -\frac{1}{2} & \frac{1}{2} } } $
$ \newcommand{\CNOT}{ \mymatrix{cccc}{1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 1 \\ 0 & 0 & 1 & 0} } $
$ \newcommand{\norm}[1]{ \left\lVert #1 \right\rVert } $_____no_output_____
 One Bit

[Watch Lecture](https://youtu.be/kn53Qvl-h28)

In daily life, we use decimal number system. It is also called base-10 system, because we have 10 digits:

$ 0,~1,~2,~3,~4,~5,~6,~7,~8, \mbox{ and } 9  $.

In computer science, on the other hand, the widely used system is binary, which has only two digits:

$ 0 $ and $ 1 $.

Bit (or binary digit) is the basic unit of information used in computer science and information theory.

It can also be seen as the smallest "useful" memory unit, which has two states named 0 and 1.

At any moment, a bit can be in either state 0 or state 1._____no_output_____
 Four operators

How many different operators can be defined on a single bit?

An operator, depending on the current state of the bit, updates the state of bit (the result may be the same state).

We can apply four different operators to a single bit:

 Identity: $ I(0) = 0 $ and $ I(1) = 1 $

 Negation: $ NOT(0) = 1 $ and $ NOT(1) = 0 $

 Constant (Zero): $ ZERO(0) = 0 $ and $ ZERO(1) = 0 $

 Constant (One): $ ONE(0) = 1 $ and $ ONE(1) = 1 $

The first operator is called IDENTITY, because it does not change the content/value of the bit.

The second operator is named NOT, because it negates (flips) the value of bit.

Remark that 0 and 1 also refer to Boolean values False and True, respectively, and, False is the negation of True, and True is the negation of False.

The third (resp., fourth) operator returns a constant value 0 (resp., 1), whatever the input is._____no_output_____
 Table representation

We can represent the transitions of each operator by a table:

$
I = \begin{array}{lc|cc}
    & & initial & states \\
    & & \mathbf{0} & \mathbf{1} \\ \hline
    final & \mathbf{0} & \mbox{goes-to} & \emptyset \\
    states & \mathbf{1} & \emptyset & \mbox{goes-to}  \end{array} ,
$
where
- the header (first row) represents the initial values, and
- the first column represents the final values.

We can also define the transitions numerically:
- we use 1 if there is a transition between two values, and,
- we use 0 if there is no transition between two values.

$
I = \begin{array}{lc|cc}
    & & initial & states \\
    & & \mathbf{0} & \mathbf{1} \\ \hline
    final & \mathbf{0} & 1 & 0 \\
    states & \mathbf{1} & 0 & 1  \end{array}
$_____no_output_____The values in bold are the initial and final values of the bits. The non-bold values represent the transitions.

 The top-left non-bold 1 represents the transtion $ 0 \rightarrow 0 $.

 The bottom-right non-bold 1 represents the transtion $ 1 \rightarrow 1 $.

 The top-right non-bold 0 means that there is no transition from 1 to 0.

 The bottom-left non-bold 0 means that there is no transition from 0 to 1.

The reader may think that the values 0 and 1 are representing the transitions as False (Off) and True (On), respectively.

Similarly, we can represent the other operators as below:

$
NOT = \begin{array}{lc|cc} & & initial & states \\ & & \mathbf{0} & \mathbf{1} \\ \hline final & \mathbf{0} & 0 & 1 \\
    states & \mathbf{1} & 1 & 0  \end{array}
~~~
ZERO = \begin{array}{lc|cc} & & initial & states \\ & & \mathbf{0} & \mathbf{1} \\ \hline final & \mathbf{0} & 1 & 1 \\
    states & \mathbf{1} & 0 & 0  \end{array}
~~~
ONE = \begin{array}{lc|cc} & & initial & states \\ & & \mathbf{0} & \mathbf{1} \\ \hline final & \mathbf{0} & 0 & 0 \\
    states & \mathbf{1} & 1 & 1  \end{array}
.
$_____no_output_____
 Task 1

Convince yourself with the correctness of each table._____no_output_____
 Reversibility and Irreversibility

After applying Identity or NOT operator, we can easily determine the initial value by checking the final value.

 In the case of Identity operator, we simply say the same value.

 In the case of NOT operator, we simply say the other value, i.e., if the final value is 0 (resp., 1), then we say 1 (resp., 0).

However, we cannot know the initial value by checking the final value after applying ZERO or ONE operator.

Based on this observation, we can classify the operators into two types: Reversible and Irreversible.

 If we can recover the initial value(s) from the final value(s), then the operator is called reversible like Identity and NOT operators.

 If we cannot know the initial value(s) from the final value(s), then the operator is called irreversible like ZERO and ONE operators.

 This classification is important, as the quantum evolution operators are reversible  (as long as the system is closed).

The Identity and NOT operators are two fundamental quantum operators._____no_output_____