# Notebook from deepfates/fastpages
Path: _notebooks/2021-07-05-twitter-test.ipynb

# Can I extract tweets to this page

- toc: true
- badges: true
- comments: true
- categories: [code, twitter]
- hide: true_____no_output_____

import nest_asyncio
nest_asyncio.apply()
_____no_output_____

I want to be able to programmatically scrape tweets from my account (and maybe others), push the URLs into a markdown cell, and let `fastpages` do the work of embedding the tweets in a blog page. I can use `twint` to get the tweets without using Twitter API. If replies are involved I might need more complicated data structures, but simple tweet-unrolling should be linear and straightforward.

First, I'll fetch the info about a top-level tweet. Then get all tweets from that user, to that user, filtered for conversation_id. TThen print them in chronological order to a markdown display! (Can it be so easy??)

The top level tweet is located at
```
https://twitter.com/deepfates/status/1412086693713358850?s=20
```

and should be displayed like

>twitter: https://twitter.com/deepfates/status/1412086693713358850_____no_output_____

# Get the tweet's location with free ARGH endpoint
op = 'https://twitter.com/deepfates/status/1412086693713358850'
op_id = op.split('/')[-1]
import requests

r = requests.get(f'https://argh.tweeter.workers.dev/tw/metadata?id={op}')_____no_output_____twt = r.json()
twt_____no_output_____import twint
c = twint.Config()
_____no_output_____op_uname = twt['user']['screen_name']
op_date = twt['created_at'][:10]
op_id = twt['id_str']
tweets = []

op_uname, op_date, op_id_____no_output_____#hide_output
c.To = op_uname
c.By = op_uname
c.Since = op_date

c.Store_object = True
c.Store_object_tweets_list = tweets

twint.run.Search(c)

self_replies = [tweet for tweet in tweets if tweet.conversation_id == op_id and tweet.reply_to == []]

self_replies.sort(key=lambda x: x.datetime)

urls = [t.link for t in self_replies]
urls1412213789274083331 2021-07-05 18:56:19 -0600 <chopstickfury01> @deepfates I assume this is a depreciation joke?
1412211345399812115 2021-07-05 18:46:37 -0600 <maxisawesome538> @deepfates I'm playing tf2 in mine
1412205612075831296 2021-07-05 18:23:50 -0600 <j0lms> @deepfates Beat you to it, to the personal website rebuilding that is! Probably whimsical Twitter embedding, or something with curated tweets is a must. Also if you're gonna make it technical make it silly technical, like waitbutwhy for AI enthusiasts (or enthusiastic AIs).
1412201736010272777 2021-07-05 18:08:25 -0600 <ykgoon> @deepfates Yes that's totally fine if it's visitor counter.
1412191763599552513 2021-07-05 17:28:48 -0600 <dannycolspector> @deepfates yes, and our potential is unbelievably massive!
1412191205505568782 2021-07-05 17:26:35 -0600 <OliverCarefull> @deepfates @ctjlewis For me, I want to be able to navigate both worlds. I could make my own clothes and dress like a wizard, but I already have one leg so really all regular rules are suspended anyway.   https://t.co/LV1YNkNKEm
1412186307275591688 2021-07-05 17:07:07 -0600 <sam_havens> @deepfates I like it, BUT that it is reasonable is an indictment. "let's just package up not just the application, but its dependencies and the entire OS, so we actually have something that we can have confidence will work"
1412180851387863040 2021-07-05 16:45:26 -0600 <jejunestarfcker> @deepfates have you read this? great article about smarm  https://t.co/HmZPWQur7x
1412176032623300612 2021-07-05 16:26:17 -0600 <acnebs> @deepfates I didn’t realize there were other types of shirt.
1412175583186931713 2021-07-05 16:24:30 -0600 <pee_zombie> @deepfates it's kinda dumb but it's way less dumb than many other things
1412175286414761987 2021-07-05 16:23:19 -0600 <ben_papillon> @deepfates bc I have no idea what I’m doing here lol
1412166419425472512 2021-07-05 15:48:05 -0600 <erocdrahs> @deepfates Virtualisation is the devils business.
1412161193356181518 2021-07-05 15:27:19 -0600 <JovanJester> @deepfates Sorry for doubting, I am certain it is totally legit now. Blue checks never lie.
1412160320559206403 2021-07-05 15:23:51 -0600 <maxisawesome538> @deepfates docker system prune
1412157737253818376 2021-07-05 15:13:35 -0600 <maxisawesome538> @deepfates it's not that bad
1412156980920295428 2021-07-05 15:10:35 -0600 <JovanJester> @deepfates This star rating stuff looks kinda sus to me
1412156164809408515 2021-07-05 15:07:20 -0600 <reconfigurthing> @deepfates congrats king
1412155256117420032 2021-07-05 15:03:44 -0600 <_rajkhare> @deepfates Well it does help to protect your local env from going haywire. But yeah another level of indirection 😑
1412154319508488198 2021-07-05 15:00:00 -0600 <blisstweeting> @deepfates lol no I've taken it in your profile 😂
1412153341614956550 2021-07-05 14:56:07 -0600 <Al_D_Smith> @deepfates @sirsfurther looks like the natural log of your twitter follow count or something. damn it
1412153165257056256 2021-07-05 14:55:25 -0600 <sirsfurther> @deepfates @Al_D_Smith I wonder how Gary bought his new car 🤔🤔🤔🤔
1412153135619993601 2021-07-05 14:55:18 -0600 <skugern> @deepfates don't forget to pay the membership dues to retain the half star
1412152922054529029 2021-07-05 14:54:27 -0600 <blisstweeting> @deepfates I'm starstruck 🤩  https://t.co/nWRSP7my61
1412152765493628949 2021-07-05 14:53:50 -0600 <mayfer> @deepfates docker and webpack are the Juiceros of software
1412152343609610244 2021-07-05 14:52:09 -0600 <erinbeess> @deepfates it is
1412152271559925762 2021-07-05 14:51:52 -0600 <Kirsten3531> @deepfates I count 7 🤨
1412152113367506948 2021-07-05 14:51:15 -0600 <epicidity> @deepfates not just you
1412151770088886272 2021-07-05 14:49:53 -0600 <taalumot> @deepfates what’s next for you
1412151768742613000 2021-07-05 14:49:52 -0600 <Al_D_Smith> @deepfates @sirsfurther watch out, he's trying to one-and-a-half-up you
1412151618829795334 2021-07-05 14:49:17 -0600 <taalumot> @deepfates congrats
1412151608868278274 2021-07-05 14:49:14 -0600 <deepfates> This was a typo that I'm keeping it
1412151200401739778 2021-07-05 14:47:37 -0600 <vishyssoise> @deepfates yet u live in one, curious?  bottomtext
1412150905303179266 2021-07-05 14:46:26 -0600 <nickcammarata> @deepfates at least make it so boring it’s really out there then
1412146373324165129 2021-07-05 14:28:26 -0600 <nickcammarata> @deepfates whatever you choose make sure it’s really out there
1412144906223570952 2021-07-05 14:22:36 -0600 <plesmasquerade> @deepfates I dont think the world where im an ancestor to many civilizations is a good one
1412142733938937857 2021-07-05 14:13:58 -0600 <Cytosknee> @deepfates Alright alright... yes I'm full of shit, I want aliens to come to earth wielding big green mommy milkers for me to suck on while they cradle me and tell me the secrets of the universe
1412140346432278534 2021-07-05 14:04:29 -0600 <nicholatian> @deepfates because i had this uniform tailor-made
1412138578197417984 2021-07-05 13:57:27 -0600 <Cytosknee> @deepfates Maybe rejoice is not appropriate because I'm too indifferent to life outside of earth that I don't find delight in the idea of another world like ours existing. I just hope it's not as fucked as ours. It not existing at all is fine with me, if it does, ok, but god help them.
1412138349553360896 2021-07-05 13:56:33 -0600 <asingleoat> @deepfates like this  https://t.co/GyTSJmhuZQ
1412136448459874304 2021-07-05 13:49:00 -0600 <Cytosknee> @deepfates I hope the agonies of life in an advanced civilization is a localized event and the rest of the universe is spared from the experience. People fall into despair thinking about the fermi paradox,  I rejoice in it.
1412135274721599488 2021-07-05 13:44:20 -0600 <pervexists69> @deepfates felt cute, might delete later
1412135063601307649 2021-07-05 13:43:30 -0600 <KGoth999> @deepfates Not mixing hermeticism with dope, when.
1412133741674188800 2021-07-05 13:38:14 -0600 <KGoth999> @deepfates who cares, run it thru the wash a couple times
1412132474629640192 2021-07-05 13:33:12 -0600 <deepfates> I fucked it up but whatever
1412131187343212545 2021-07-05 13:28:05 -0600 <chopstickfury01> @deepfates @selentelechia All I know is that everything needs to be shorter for me.  ESPECIALLY LIGHTING.
1412131167067848711 2021-07-05 13:28:01 -0600 <aeranothaur> @deepfates I actually really like this paperback design you have now. it's one of the cleanest and clearest personal sites I've seen in ages
1412130952067829771 2021-07-05 13:27:09 -0600 <chopstickfury01> @deepfates I... I actually remember this.
1412128251158401032 2021-07-05 13:16:25 -0600 <hyperfekt> @deepfates no 👏 more 👏 witnessing 👏
1412126291281821696 2021-07-05 13:08:38 -0600 <bantablack> @deepfates What watered-down Greek wine looks like  https://t.co/Z6dH8pej18
1412121295471321114 2021-07-05 12:48:47 -0600 <Cytosknee> @deepfates When the kids all start fedposting
1412120383172452366 2021-07-05 12:45:09 -0600 <caelanhuntress> @deepfates My shirt has no collar but I take your point
1412119596610428930 2021-07-05 12:42:02 -0600 <mattdsegal> @deepfates Write down goals of website Map out user journey  Loosely related:  https://t.co/EDtTHn78TK
1412114826529325062 2021-07-05 12:23:05 -0600 <_avichalp_> @deepfates everyone will be cancelled eventually
1412113631274209282 2021-07-05 12:18:20 -0600 <pantinkins> @deepfates The things described here need tangible examples, maybe a page for each topic. What specifically have you done? What specifically do you think?  https://t.co/JtNqtuRxtL
1412110584561156099 2021-07-05 12:06:13 -0600 <laundryandry> @deepfates @deepfates_notes what different representations come to mind ?
1412110386493595650 2021-07-05 12:05:26 -0600 <pantinkins> @deepfates Gotta catch me first
1412109337741303810 2021-07-05 12:01:16 -0600 <mankemens> @deepfates I thought I made that bit transparent 😶
1412108706599346178 2021-07-05 11:58:46 -0600 <chriscooladini> @deepfates tempus fugit, memenTOS
1412108684797300740 2021-07-05 11:58:40 -0600 <DaltonDEmery> @deepfates Inshallah
1412108311462400002 2021-07-05 11:57:11 -0600 <laundryandry> @deepfates every account will be banned for 15 minutes
1412108164053409793 2021-07-05 11:56:36 -0600 <judahwantssocks> @deepfates soon inshallah
1412108158357606407 2021-07-05 11:56:35 -0600 <ELFonthebeatMG> @deepfates Jack be like:  https://t.co/jknHVaRC9q
1412108097976406017 2021-07-05 11:56:20 -0600 <laundryandry> @deepfates @deepfates_notes yeah mastodon as a destination for replicated tweets is less interesting than you shaping it for your personal website. might be clumsy to figure out what you want the UX to be like but might be a fun challenge.
1412108079332663305 2021-07-05 11:56:16 -0600 <skugern> @deepfates that's the goal
1412107144992800773 2021-07-05 11:52:33 -0600 <laundryandry> @deepfates @deepfates_notes x_whatever_app being the interim connection between twitter and your static website?
1412106390873718786 2021-07-05 11:49:33 -0600 <laundryandry> @deepfates tbh i prefer twitter as an interface, and it's just (whispers quietly) *where people are* but going _from_ twitter _to_ X decentralized whatever is an interesting idea. idk how replies and such would work but at least for your own content it would be a decent 80% solution
1412104105930207236 2021-07-05 11:40:29 -0600 <gene_minkov> @deepfates "My lady got two men" by Atmosphere
1412103137637371908 2021-07-05 11:36:38 -0600 <laundryandry> @deepfates ooo i wonder if you could set up a service essentially as a follower to your twitter activity that could re-post ActivityPub protocol compliant requests to a server your control, that your website then feeds from
1412102826629615617 2021-07-05 11:35:24 -0600 <benbohmer> @deepfates Second BrainFriend
1412099928340447238 2021-07-05 11:23:53 -0600 <ctjlewis> @deepfates yes, it's also trivial because embeds
1412099615990554624 2021-07-05 11:22:38 -0600 <judahwantssocks> @deepfates you can download an archive of them all, if that's what you mean
1412099497346297861 2021-07-05 11:22:10 -0600 <danehighwind> @deepfates Most personal websites are vanity projects. That's fine, just know it going into it.  If your goal is engagement, abandon the personal site and devote that time to Twitter, GitHub, etc. directly  (Of course there are exceptions, e.g. SlateStarCodex)
1412099240252248064 2021-07-05 11:21:09 -0600 <KGoth999> @deepfates whatever
1412099130776768513 2021-07-05 11:20:42 -0600 <deepfates> is it possible to just extract all of my tweets into a static site that I host myself?
1412099083972534283 2021-07-05 11:20:31 -0600 <Mimi_Rose_01> @deepfates Depends what you're hoping to gain from having the site?  As a follower I'd click bc I want context for a tweet (like approx age or location or prof bg), or your tweets have me intrigued enough to click through for long-form more, or if you tweet a link to a book list or smthg
1412098709068795907 2021-07-05 11:19:02 -0600 <judahwantssocks> @deepfates yeah. links to twitter accounts, sites, music, blog posts etc.   twitter is cool but i'd like better formatting + persistence.   also i wanna embed my small drawings someplace
1412098394365984768 2021-07-05 11:17:47 -0600 <KGoth999> @deepfates *You tube team ranting*
1412098191097352195 2021-07-05 11:16:58 -0600 <judahwantssocks> @deepfates ooh wait. check this out.   https://t.co/p59tooLnDJ
1412097810648899584 2021-07-05 11:15:28 -0600 <judahwantssocks> @deepfates very tempting to just do one of these instead   https://t.co/ykzCE6m0oL
1412097698602262529 2021-07-05 11:15:01 -0600 <judahwantssocks> @deepfates me two years ago.  confused teens, curious teens, soft teens.  also i would mainly end up doing curation i think
1412097678759038982 2021-07-05 11:14:56 -0600 <deepfates> very wise question. hmm...  Meet people? yes. get job offers? eventually. Display my writing/hide my writing if it embarrasses me later. Get more points on Twitter and GitHub. Host weird tech experiments.   😐 is this a portfolio?   https://t.co/DctcK0QCLT
1412097372428062720 2021-07-05 11:13:43 -0600 <Mimi_Rose_01> @deepfates Should have appended "Wrong answers only." to your initial ask- guarantees more than half the replies will take you literally 😏
1412096674525159429 2021-07-05 11:10:57 -0600 <VividVoid_> @deepfates  https://t.co/ShyyCESAbE  Was literally about to ask *you* this question
1412096153550659585 2021-07-05 11:08:53 -0600 <danehighwind> @deepfates What's your desired outcome? Sell stuff? Meet people? Get job offers?
1412095448437825552 2021-07-05 11:06:05 -0600 <skugern> @deepfates @sirsfurther @infra_naut @plesmasquerade i fully endorse this website. I personally pay a monthly patreon for the amazing content
1412095434261102595 2021-07-05 11:06:01 -0600 <judahwantssocks> @deepfates i am honestly wracking my brains trying to come up with something completely bonkers for my own place but i come back to "blog" every single time.  damn
1412094715009380353 2021-07-05 11:03:10 -0600 <afterlxss> @deepfates Like a canon bookshelf kinda UI idk, but make the source code hidden just for the added chaos
1412094653780959232 2021-07-05 11:02:55 -0600 <deepfates> All right a real information architecture problem:  Right now, I have some front facing about page type stuff, and then a "digital garden" style notebook that I update too rarely.  what if it was more like a technical blog, with posts organized by category and time? Good or na
1412093548581818373 2021-07-05 10:58:32 -0600 <deepfates> This is a very good and literal answer Can I have some more of those   https://t.co/rQ0SfNCRz1
1412093512422670343 2021-07-05 10:58:23 -0600 <Historycourses> @deepfates Bayonets!!!!!
1412093296411918337 2021-07-05 10:57:31 -0600 <Al_D_Smith> @deepfates Henley exemption
1412093177234874368 2021-07-05 10:57:03 -0600 <Mimi_Rose_01> @deepfates Greatest hit Tweets, by personal selection and by most likes.  Link to GitHub account with an explanation of what you're working on.  Top 100 book list with short reviews and  humorous tagging.
1412092605500903428 2021-07-05 10:54:47 -0600 <GENIC0N> @deepfates you should burn books out back of the store to release stress, or smash coffee mugs made with love or something
1412092085885456389 2021-07-05 10:52:43 -0600 <GENIC0N> @deepfates just read Ogilvy on Advertising and create a content strategy and then create a content distribution strategy. I hope I'm opening pandora's box for you here, because you asked.
1412091934760374273 2021-07-05 10:52:07 -0600 <sirsfurther> @deepfates Lots of gifs jumping about the screen
1412091539908706311 2021-07-05 10:50:33 -0600 <deepfates> everybody wants to be funny but i need advice. how to present deepfates
1412091507784523783 2021-07-05 10:50:25 -0600 <parafactual> @deepfates @kyzcralotSxaM LMAO
1412091129898672133 2021-07-05 10:48:55 -0600 <blisstweeting> @deepfates actually this is a nice idea ☝
1412090975686701059 2021-07-05 10:48:18 -0600 <parafactual> @deepfates @kyzcralotSxaM hmmm...eventually i am gonna have to fill my about page. maybe then some legal page can take on the role?
1412090312273633283 2021-07-05 10:45:40 -0600 <afterlxss> @deepfates Like a Canon style but make the source hidden
1412089995825979392 2021-07-05 10:44:25 -0600 <parafactual> @deepfates @kyzcralotSxaM some are buttons that create dialog boxes instead
1412089905736392726 2021-07-05 10:44:03 -0600 <ZetaRayZac> @deepfates don't tell anyone but i forgot to finish coloring the hair in my avatar
1412089429443887108 2021-07-05 10:42:09 -0600 <sirsfurther> @deepfates Yes.
1412089418157068290 2021-07-05 10:42:07 -0600 <blisstweeting> @deepfates btw I'm closing in on finally cloning the repo
1412089176258797568 2021-07-05 10:41:09 -0600 <ykgoon> @deepfates For everybody's sake, lighten the load of the web.
1412089080842698754 2021-07-05 10:40:46 -0600 <ELFonthebeatMG> @deepfates Check out how… professional i am
1412088865670709249 2021-07-05 10:39:55 -0600 <sirsfurther> @deepfates It's Charles Fox
1412088615379812353 2021-07-05 10:38:55 -0600 <blisstweeting> @deepfates memery
1412088528486371331 2021-07-05 10:38:35 -0600 <zachcaceres> @deepfates really cool projects. also im so jelly that you and your fam are functional enough to run a book store together.
1412088298714042370 2021-07-05 10:37:40 -0600 <deepfates> When I say 'look like" I don't mean just the styling, although I'm accepting advice on that too. I mean, what do you want to see when you click through from this account to my website?   what am I supposed to put on there
1412088201561317381 2021-07-05 10:37:17 -0600 <rplevy> @deepfates @zachcaceres Oh cool I like that it looks like a pocket paperback book
1412088201154543621 2021-07-05 10:37:17 -0600 <sirsfurther> @deepfates Heyyyyy
1412088161493258241 2021-07-05 10:37:07 -0600 <DaltonDEmery> @deepfates What’s a shirt
1412088138403565570 2021-07-05 10:37:02 -0600 <deepfates> currently i have  https://t.co/UEM7H2vuZk styled like an old Pelican paperback but i don't really know how to present myself within that
1412088089128980483 2021-07-05 10:36:50 -0600 <VectorOfBasis> @deepfates geocities
1412088038730108929 2021-07-05 10:36:38 -0600 <blisstweeting> @deepfates The two genders  https://t.co/FUBpL6g3Cg
1412087873898225670 2021-07-05 10:35:59 -0600 <kyzcralotSxaM> @deepfates minimalist
1412087571803369477 2021-07-05 10:34:47 -0600 <uberstuber> @deepfates invisible
1412087545752653834 2021-07-05 10:34:40 -0600 <OliverCarefull> @deepfates Couldn't be me
1412087511069777921 2021-07-05 10:34:32 -0600 <ctjlewis> @deepfates max normie points
1412087223151828993 2021-07-05 10:33:23 -0600 <zachcaceres> @deepfates whats your current one?
1412087170672771085 2021-07-05 10:33:11 -0600 <ctjlewis> @deepfates immersive 3D shit, cc @SeanMombo
1412087103618531334 2021-07-05 10:32:55 -0600 <loopholekid> @deepfates Flippantly: beautiful
1412086945803431938 2021-07-05 10:32:17 -0600 <ykgoon> @deepfates Text only, no images
1412081674721153031 2021-07-05 10:11:21 -0600 <beckness> @deepfates I suspect smarm is in the eye of the beholder.
1412076456109391874 2021-07-05 09:50:36 -0600 <seeclarkrun> @deepfates Alright tell the beaver I’ll be nice. I’m over it
1412074316452225040 2021-07-05 09:42:06 -0600 <seeclarkrun> @deepfates is this because I was mean to a wet rodent
1412073565051043842 2021-07-05 09:39:07 -0600 <seeclarkrun> @deepfates less gooo
1412071002029400072 2021-07-05 09:28:56 -0600 <seeclarkrun> @deepfates wrong take
1411959980152877058 2021-07-05 02:07:46 -0600 <caelanhuntress> @deepfates @Tjdriii Did you know that the word ‘barbarian’ comes from how Greeks made fun of foreign speech  Those xenos be like ‘bar bar bar bar bar bar’
[!] No more data! Scraping will stop now.
found 0 deleted tweets in this search.
from IPython.display import Markdown as md
_____no_output_____templated_links = '>twitter: ' + '\n\n>twitter: '.join(urls)_____no_output_____md(f'''These should be the appropriate links: \n\n>twitter:
{templated_links}
   ''')
_____no_output_____

If I make a raw cell and copy the `% include` magic from the converted post, I can get it to show up:_____no_output_____{% include twitter.html content='https://twitter.com/deepfates/status/1412097678759038982' %}_____no_output_____And I can make one programmatically, but I can't figure out how to insert that as the next cell._____no_output_____

include = '''{% include twitter.html content='https://twitter.com/deepfates/status/1412097678759038982' %}'''_____no_output_____import nbformat as nbf_____no_output_____raw_cell = nbf.v4.new_raw_cell(include)_____no_output_____from IPython import display_____no_output_____display.display(raw_cell)_____no_output_____from IPython.core.getipython import get_ipython
shell = get_ipython()
_____no_output_____shell.set_next_input(templated_links)_____no_output_____

>twitter: https://twitter.com/deepfates/status/1412088138403565570

>twitter: https://twitter.com/deepfates/status/1412088298714042370

>twitter: https://twitter.com/deepfates/status/1412091539908706311

>twitter: https://twitter.com/deepfates/status/1412093548581818373

>twitter: https://twitter.com/deepfates/status/1412094653780959232

>twitter: https://twitter.com/deepfates/status/1412097678759038982

>twitter: https://twitter.com/deepfates/status/1412099130776768513_____no_output_____Well, if i want I can send them into a new cell and manually make it into a markdown cell and run it. That's reasonable. Alternatively, I could insert them into a new notebook entirely, as raw includes.

The other problem is that this Twitter embed includs the previous tweet, so the idea of posting abuch ofthem as a thread is kind of awkward. Might work better for a "popular tweets" post?

Yes... In fact, it could even be a notebook I run every so often taht prints to a .md file in the `pages` folder. All it has to have is YAML frontmatter and a list of includes..._____no_output_____


# Notebook from LDSSA/ds-prep-course-2021
Path: Week 11/SLU18_3 - Exam Prep II/Exercise notebook.ipynb

import csv
import json
import numpy as np
import utils_____no_output_____

### Exercise 1 (5 points)

This exercise is a quiz with 14 multiple answer questions.
In the cell below, we've declared a dictionary called `answers`.

You should fill in that dictionary with your answers, using as keys the question number, like `question_x`, and as values the number from `1` to `4` that corresponds to the right answer.
For each question, you should provide only one answer, i.e, the dict values should have type **int**.

For example, if you want to answer Question 1 with choice number 2, then you do:
```
answers["question_1"] = 2
```_____no_output_____

answers = {}_____no_output_____

#### Question 1
In the terminal of a Unix based machine, which of these commands will take you to your home directory?
1. cd ..
2. ls ..
3. cd ~/
4. ls ~/_____no_output_____

# YOUR CODE HERE

# YOUR CODE HERE
raise NotImplementedError()

# answers["question_1"] = ..._____no_output_____

#### Question 2
In the terminal of a Unix based machine, which of these commands would you use to see the entire content of a file named `data.csv`?
1. head -5 data.csv
2. tail -5 data.csv
3. cat data.csv
4. ls -la data.csv_____no_output_____

# YOUR CODE HERE
# YOUR CODE HERE
raise NotImplementedError()
# answers["question_2"] = ..._____no_output_____

#### Question 3
After running one of these commands in the terminal of a Unix based machine, you won't have a file named `data.csv` in your current directory. Which command is it?
1. cp dataset.csv data.csv
2. rm data.csv
3. touch data.csv
4. mv dataset.csv data.csv_____no_output_____

# YOUR CODE HERE
# YOUR CODE HERE
raise NotImplementedError()

# answers["question_3"] = ..._____no_output_____

#### Question 4
In the terminal of a Unix based machine, if you want to see the size of the files in the current directory, in a human readable format, which command is the best option?
1. ls
2. ls .
3. ls -la
4. ls -lh_____no_output_____

# YOUR CODE HERE
# YOUR CODE HERE
raise NotImplementedError()

# answers["question_4"] = ..._____no_output_____

#### Question 5
There are certain types of files that you usually don't want to have in your git repositories, like cache files, or local configuration files that contain passwords. In order to have git not tracking these files, you can use the `.gitignore` file. Say you're currently in the root directory of your project, and you have a `.gitignore` file there. In the terminal of a Unix based machine, which of these commands should you use to list the files and see the `.gitignore` file on that list?
1. ls -la
2. ls .
3. ls -lh
4. ls -l_____no_output_____

# YOUR CODE HERE
# YOUR CODE HERE
raise NotImplementedError()

# answers["question_5"] = ..._____no_output_____

#### Question 6
Consider the following git commands: `add`, `commit`, `status`, `pull`, `push`. Which is the most natural order to use them?
1. `pull` > `status` > `add` > `commit` > `push`
2. `status` > `commit` > `add` > `push` > `pull`
3. `push` > `add` > `status` > `commit` > `pull`
4. `pull` > `push` > `status` > `add` > `commit`_____no_output_____

# YOUR CODE HERE
# YOUR CODE HERE
raise NotImplementedError()

# answers["question_6"] = ..._____no_output_____

#### Question 7
How do you change from the `master` branch to an existing `feature_dev` branch?
1. git checkout master
2. git checkout master feature_dev
3. git checkout -b feature_dev
4. git checkout feature_dev_____no_output_____

# YOUR CODE HERE
# YOUR CODE HERE
raise NotImplementedError()

# answers["question_7"] = ..._____no_output_____

#### Question 8
Which of these commands should you use to find the hash of your previous commit?
1. git status
2. git log
3. git diff
4. git commit_____no_output_____

# YOUR CODE HERE
# YOUR CODE HERE
raise NotImplementedError()

# answers["question_8"] = ..._____no_output_____

#### Question 9
Which one of these is false:
1. You can use a tuple as a dictionary key.
2. You can add new elements to a tuple.
3. Tuples can have duplicated elements.
4. You can define a tuple with length 0._____no_output_____

# YOUR CODE HERE
# YOUR CODE HERE
raise NotImplementedError()

# answers["question_9"] = ..._____no_output_____

#### Question 10
If you want to incorporate the master branch into another branch called `dev`, which 2 commands should you run sequentially?
1. git checkout dev; git merge master
2. git merge master; git checkout dev
3. git checkout master; git merge dev
4. git merge dev; git checkout master_____no_output_____

# YOUR CODE HERE
# YOUR CODE HERE
raise NotImplementedError()

# answers["question_10"] = ..._____no_output_____

#### Question 11
Which command should you use to bring back changes that you've stashed?
1. git stash
2. git stash pop
3. git revert
4. git revert apply_____no_output_____

# YOUR CODE HERE
# YOUR CODE HERE
raise NotImplementedError()

# answers["question_11"] = ..._____no_output_____

#### Question 12
Which one of these files will you always find in the root directory of a Python package?
1. README.md
2. requirements.txt
3. \_\_init\_\_.py
4. setup.py_____no_output_____

# YOUR CODE HERE
# YOUR CODE HERE
raise NotImplementedError()

# answers["question_12"] = ..._____no_output_____

#### Question 13
How do you import function `my_function` from module `my_module`?
1. import my_function
2. import my_module
3. from my_function import my_module
4. from my_module import my_function_____no_output_____

# YOUR CODE HERE
# YOUR CODE HERE
raise NotImplementedError()

# answers["question_13"] = ..._____no_output_____

#### Question 14
Which one of these should not raise an Error?
1. '4' + 'None'
2. 4 + None
3. 4 + 'None'
4. '4' + None_____no_output_____

# YOUR CODE HERE
# YOUR CODE HERE
raise NotImplementedError()

# answers["question_14"] = ..._____no_output_____# Run this cell to see if your answers are correct
utils.exercise_1(answers)_____no_output_____

### Exercise 2

Consider a csv file with four columns: `city`, `latitude`, `longitude`, `country`.

File `cities_locations.csv` is an example of such file and was included in the zip file you downloaded with this notebook, so you can preview it using some command line arguments._____no_output_____#### Part I (2 points)

Implement a function that reads a file with the same format as the `cities_locations.csv` file, and stores the data in a dictionary with the following structure:

```
{
    "city_name": {
        "idx": row number in the input file, not considering the header,
               i.e, the first row with data values has idx=0 (type: int),
        "lat": latitude value (type: float),
        "lng": longitude value (type: float),
        "country": country (type: str)
    },
}

```

The function should be called `read_cities`, receives an argument called `filepath` (which is the file that the function should read the data from), and returns the dictionary that was created._____no_output_____

def read_cities(filepath):
    """
    Reads the file in filepath, parses it and returns the data in a dictionary.

    Parameters:
    filepath (str): Path to the input file to be parsed.

    Returns:
    cities_parsed_data (dict): Cities locations dataset stored as a dictionary, where the keys are city names
          and the values are data about the city.
    """

    # YOUR CODE HERE
# YOUR CODE HERE
raise NotImplementedError()_____no_output_____cities_locations_data = read_cities('cities_locations.csv')

assert len(cities_locations_data) == 10

assert 'Barcelona' in cities_locations_data
barcelona = cities_locations_data['Barcelona']
assert barcelona['idx'] == 5
assert barcelona['country'] == 'Spain'
np.testing.assert_almost_equal(barcelona['lat'], 41.3833, decimal=3)
np.testing.assert_almost_equal(barcelona['lng'], 2.1834, decimal=3)

from more_tests import test_exercise_2_I
test_exercise_2_I(read_cities)_____no_output_____

#### Part II (3 points)

We want to calculate the distance in km between each pair of cities from our dataset, and store this in a numpy array.

In order to calculate these distances, you should use the **geodesic** function from the python package [geopy](https://pypi.org/project/geopy/).

You'll need to read the package's docs, in order to install it in your local environment and to learn how to use the **geodesic** function. The version of geopy that is installed in our grader is `1.22.0`, so, to make sure that you'll have the same behaviour in your local environment and in our grader, make sure to install the same version.

The matrix with the distances will have shape `(n_cities, n_cities)`, where `n_cities` is the number of cities in the dataset. Each city's position in the matrix's rows and columns should be the `idx` value from the dataset.

Implement a function called `create_dist_matrix` that receives as input a dictionary like the one we created in the first part of this exercise, and returns the numpy array representing the distances matrix._____no_output_____

from geopy.distance import geodesic

def create_dist_matrix(cities_data):
    """
    Computes the geodesic distance between each pair of cities in the input dataset, and returns a matrix with the
    computed distances.

    Parameters:
    cities_data (dict): Dictionary with the city locations dataset.

    Returns:
    matrix (np.array): Matrix with shape (n_cities, n_cities) where the value in position (i, j) represents
        the geodesic distance in km between city_i and city_j.
    """

    # YOUR CODE HERE
# YOUR CODE HERE
raise NotImplementedError()_____no_output_____cities_locations_data = {
    'Rome': {'idx': 0, 'lat': 41.896, 'lng': 12.4833, 'country': 'Italy'},
    'Milan': {'idx': 1, 'lat': 45.47, 'lng': 9.205, 'country': 'Italy'},
    'Lisbon': {'idx': 2, 'lat': 38.7227, 'lng': -9.1449, 'country': 'Portugal'},
    'Porto': {'idx': 3, 'lat': 41.15, 'lng': -8.62, 'country': 'Portugal'},
    'Madrid': {'idx': 4, 'lat': 40.4, 'lng': -3.6834, 'country': 'Spain'},
    'Barcelona': {'idx': 5, 'lat': 41.3833, 'lng': 2.1834, 'country': 'Spain'}
}

dists = create_dist_matrix(cities_locations_data)

assert type(dists) == np.ndarray
assert dists.shape == (6, 6)

for i in range(dists.shape[0]):
    assert dists[i][i] == 0

np.testing.assert_almost_equal(dists[0][1], 476.93, decimal=1)
np.testing.assert_almost_equal(dists[0][5], 859.51, decimal=1)

from more_tests import test_exercise_2_II
test_exercise_2_II(create_dist_matrix)_____no_output_____

#### Part III (2 points)

Create a function called `furthest_cities` that finds in our distances matrix, the pair of cities furthest away from each other.

It should return a tuple of city indexes, where the first one is smaller than the second._____no_output_____

def furthest_cities(dist_matrix):
    """
    Finds the pair of cities that is furthest away from each other.

    Parameters:
    dist_matrix (np.array): Matrix that stores geodesic distances between cities.

    Returns:
    furthest_cities_idx (tuple): Indexes of the two cities. The first element in the tuple is the lower index.
    """

    # YOUR CODE HERE
# YOUR CODE HERE
raise NotImplementedError()_____no_output_____# Creating a random distances matrix for this test
np.random.seed(19)
dists = np.random.rand(5, 5) * 1000
for i in range(dists.shape[0]):
    dists[i][i] = 0
    for j in range(dists.shape[1]):
        if j > i:
            dists[i][j] = dists[j][i]

idx_1, idx_2 = furthest_cities(dists)
assert idx_1 == 1
assert idx_2 == 4

from more_tests import test_exercise_2_III
test_exercise_2_III(furthest_cities)_____no_output_____

### Exercise 3

In this exercise, we'll use object oriented programming concepts to model [Pokemons](https://en.wikipedia.org/wiki/Pok%C3%A9mon).

In case you're not familiar with Pokemon, they are wild (imaginary) creatures. These are some examples.

When they're living in the wild, Pokemon can be captured by Pokemon Trainers. Once they belong to a Trainer, they will obey the Trainer's commands. They're usually sent out to non-lethal battles against other Pokémon, in order to gain experience and level up.

When they reach certain levels, they can undergo a form of metamorphosis and transform into a similar but stronger species of Pokémon: this is called evolution.

_____no_output_____#### Part I (4 points)

Your first assignment is to implement a class that represents a `Pokemon`.

You'll need to store the following information about a Pokemon:
* `name`: the Pokemon's name.
* `max_health`: the number of health points that this Pokemon has with full health.
* `speed`: a measure of how fast this Pokemon is. Faster Pokemons usually attack first in battles.
* `hp`: current number of health points. Pokemons may lose health points during battles.
* `level`: Pokemon's current level. This is a measure of the Pokemon's experience. Pokemons in higher levels have more chances of winning battles.

Some additional information:
* All the stats points described above (`max_health`, `speed`, `hp`, `level`) should be measured with non-negative integers.
* When a Pokemon is born, its `level` is always 1 and its `hp` is always the same as `max_health`, but its `name`, `max_health`and `speed` vary from Pokemon to Pokemon.

Our Pokemon class should implement 4 methods, described below.

##### 1. Method `is_knocked_out`

During battles, Pokemon take damage, which translates into losing health points.
Method `is_knocked_out` receives no arguments, and checks whether the Pokemon is knocked out by checking if the Pokemon's `hp` is equal to 0.
This method should return a bool.

##### 2. Method `level_up`

When a Pokemon wins a battle, it will level up.
Method `level_up` receives no arguments, and doesn't return anything.

This method should:
* Increase the Pokemon's `level` by one.
* Increase the Pokemon's `max_health` by 20 points.
* Increase the Pokemon's `speed` by 10%. But speed must be an integer, so round it down.
* It shouldn't change the Pokemon's `hp`!

##### 3. Method `take_damage`

As explained above, during battles, Pokemon take damage, which translates into losing health points.
Method `take_damage` receives as argument the integer `damage_points`, and doesn't return anything.

This method should decrease the Pokemon's `hp` by `damage_points`. Make sure that the Pokemon's `hp`doesn't fall below 0.

##### 4. Method `attack_damage`

This method calculates the number of damage points that our Pokemon's attack will inflict on an enemy Pokemon, during a battle. More experienced Pokemon inflict more damage and have higher chances of having successful attacks than less experienced Pokemon.
Method `attack_damage` receives as argument an `enemy_pokemon` (which is another instance of the Pokemon class) and returns an integer representing the number of damage points.

This method should compute the damage points in the following way:
* Create a variable `level_diff` that stores the difference in levels of the two Pokemon. If our Pokemon is more experienced, `level_diff` should be positive. If our Pokemon is less experienced, `level_diff` should be 0.
* Create a variable `max_level` that stores the maximum level between the levels of the two Pokemon.
* Create a variable called `p_success`, that represents the probability of success of your Pokemon's attack, and set it to: $$0.5 + \frac{level\_diff}{2 * max\_level}$$
* Create a variable called `attack_success`, that represents whether the attack was successful or not. Calculate its value by drawing one sample from a binomial distribution: use numpy's [binomial function](https://numpy.org/doc/stable/reference/random/generated/numpy.random.binomial.html), with parameters `n=1` and `p=p_success`. This function will output a 0 (which means the attack was not successful) or a 1 (which means the attack was successful).
* Create a variable called `damage_points`. Calculate its value by multiplying your Pokemon's `level` by `attack_success`.
* Return `damage_points`.
_____no_output_____

class Pokemon:
    # YOUR CODE HERE
# YOUR CODE HERE
raise NotImplementedError()_____no_output_____score = 0

try:
    pika = Pokemon(name='Pikachu', max_health=20, speed=5)
    assert pika.name == 'Pikachu'
    assert pika.max_health == 20
    assert pika.speed == 5
    assert pika.hp == pika.max_health
    assert pika.level == 1
    assert not pika.is_knocked_out()
except AssertionError:
    pass
else:
    score += 1

try:
    pika.level_up()
    assert pika.max_health == 40
    assert pika.hp == 20
    assert pika.speed == 5
    assert pika.level == 2
except AssertionError:
    pass
else:
    score += 1

try:
    enemy = Pokemon(name='Squirtle', max_health=10, speed=10)
    np.random.seed(42)
    assert pika.attack_damage(enemy) == 2
    assert pika.attack_damage(enemy) == 0
    assert pika.attack_damage(enemy) == 2

    np.random.seed(13)
    assert pika.attack_damage(enemy) == 0
    assert pika.attack_damage(enemy) == 2
    assert pika.attack_damage(enemy) == 0
except AssertionError:
    pass
else:
    score += 1

try:
    pika.take_damage(5)
    assert pika.hp == 15

    pika.take_damage(20)
    assert pika.hp == 0
    assert pika.is_knocked_out()
except AssertionError:
    pass
else:
    score += 1

if score == 0:
    raise AssertionError("Not enough correct answers to score points :(")

score_____no_output_____

#### Part II (2 points)

Now we'll implement a battle between two Pokemon.

Write a function called `battle` that receives as arguments `p1` and `p2`, both instances of the Pokemon class, and returns the Pokemon that wins the battle and the Pokemon that loses the battle (in this order!).

A battle is a sequence of attacks. In the first turn, the Pokemon who attacks is the fastest (if they have the same speed, `p1` attacks first); the slowest Pokemon attacks in the second turn, and then the fastest Pokemon attacks again. No Pokemon is allowed to attack twice, they always switch places as the attacker.

In an attack, the attacker inflicts as many damage points as indicated by the `attack_damage` method. Remember that this method has a random component, so you have to explicitly call it everytime the Pokemon attacks.
The defender may suffer damage points. In order to record that, you should use the `take_tamage` method.

The battle ends as soon as one of the Pokemon is knocked out.

Grader Tip: If you see a `KeyboardInterrupt` error on the grader feedback, that is because your cell is taking too long to run, which is probably due to an infinite loop._____no_output_____

def battle(p1, p2):
    """
    Represents a battle between two Pokemon, where each Pokemon attacks at a time.

    Parameters:
    p1 (Pokemon): A Pokemon fighting in the battle.
    p2 (Pokemon): The other Pokemon fighting in the battle.

    Returns:
    winner (Pokemon): The winner Pokemon
    loser (Pokemon): The loser Pokemon
    """
    # YOUR CODE HERE
# YOUR CODE HERE
raise NotImplementedError()_____no_output_____pika = Pokemon(name='Pikachu', max_health=20, speed=5)
squirtle = Pokemon(name='Squirtle', max_health=10, speed=10)

np.random.seed(19)
winner, loser = battle(pika, squirtle)

assert winner.name == "Pikachu"
assert loser.name == "Squirtle"
assert winner.hp == 15
assert loser.hp == 0

from more_tests import test_exercise_3_II
test_exercise_3_II(battle, Pokemon)_____no_output_____

### Exercise 4

Consider that you have a matrix, that represents a field, where you can go for a walk.

In order to refer to a specific position in the matrix, we'll use the notation `(i,j)`, where `i` represents the row number, `j` represents the column number, using 0-indexing (the first row has index 0).

You start walking at some position in the matrix, and the value on that position will guide you towards your next position. The possible values in the matrix are:
* `l`: means that you'll move to the column on your **left**, in the same row
* `r`: means that you'll move to the column on your **right**, in the same row
* `u`: means that you'll move to the row **above**, in the same column
* `d`: means that you'll move to the row **below**, in the same column

You'll be able to keep walking until you're guided to a position outside the boundaries of the field.

Let's see an example. This is a field.

[['r', 'r', 'd', 'l', 'u'],
 ['r', 'd', 'l', 'd', 'd'],
 ['u', 'r', 'd', 'r', 'l'],
 ['l', 'd', 'd', 'u', 'r']]

If you start your walk in position (0,0), your path will be:
* `(0,0)`
* `r` => `(0,1)`
* `r` => `(0,2)`
* `d` => `(1,2)`
* `l` => `(1,1)`
* `d` => `(2,1)`
* `r` => `(2,2)`
* `d` => `(3,2)`
* `d` => `end`

In red you can see the positions where you passed.

[['r', 'r', 'd', 'l', 'u'],
 ['r', 'd', 'l', 'd', 'd'],
 ['u', 'r', 'd', 'r', 'l'],
 ['l', 'd', 'd', 'u', 'r']]

The length of your path is the number of positions where you have been in the field.
The path in the example has length 8.

***

#### Part I (1 point)

Write a function, named `walk` that receives as arguments:
* a numpy matrix that represents a **field**, and
* a tuple `(i, j)` that represents a **starting position**,

and returns the length of the path that you get if you start walking at the starting position and continue until you leave the field.

You can assume that the field is not empty, i.e, it has at least one position, and that the starting position is within the field.

#### Part II (1 point)

For an extra point, add the following validations to your function.

If you find an unknown instruction while walking your path, raise a **ValueError** with message: `Invalid instruction found on position (i, j)`, where `i` and `j` are the row and column numbers of the position where you found the unknown instruction. If the field has unknown instructions, but you don't stumble upon them while walking, no exception should be raised.

And be careful! Your field might tell you to start walking in circles! If during your walk, you find yourself in a position where you have been before, you should raise a **ValueError** with message: `"I'm running in circles"`

Grader Tip: If you see a `KeyboardInterrupt` error on the grader feedback, that is because your cell is taking too long to run, which is probably due to an infinite loop._____no_output_____

def walk(field, start):
    """
    Returns the length of the path that begins on a starting position and continues according to the
    field's instructions.

    Parameters:
    field (np.array): Matrix with instructions that describes a field where one can go for a walk.
    start (tuple):    Position where to start walking.
                      The first value is the row index and the second is the column index.

    Returns:
    walk_length (int): Length of the walk.

    Raises:
    ValueError: if an invalid instruction is found.
    ValueError: if the walk has a loop.
    """

    # YOUR CODE HERE
# YOUR CODE HERE
raise NotImplementedError()_____no_output_____# test field with single position
field = np.array([['d']])
assert walk(field, (0, 0)) == 1

# test field with successful walk
field = np.array([
    ['d', 'l', 'd'],
    ['d', 'u', 'l'],
    ['d', 'r', 'd'],
    ['d', 'd', 'd'],
])
assert walk(field, (1, 2)) == 7
assert walk(field, (2, 2)) == 2

# test another field with successful walk
field = np.array([
    ['r', 'r', 'd', 'l', 'u'],
    ['r', 'd', 'l', 'd', 'd'],
    ['u', 'r', 'd', 'r', 'l'],
    ['l', 'd', 'd', 'u', 'r'],
])
assert walk(field, (0, 0)) == 8_____no_output_____# test field with loop
field = np.array([
        ['r', 'r', 'd', 'l', 'u'],
        ['r', 'd', 'l', 'd', 'd'],
        ['u', 'r', 'd', 'r', 'l'],
        ['l', 'd', 'd', 'u', 'r'],
    ])
try:
    walk(field, (1, 3))
    raise AssertionError("Should raise a ValueError when running in circles!")
except ValueError as e:
    assert str(e) == "I'm running in circles"
else:
    raise AssertionError("Should have raised the right error type!")

# test another field with loop
field = np.array([
    ['d', 'l', 'l'],
    ['d', 'w', 'u'],
    ['d', 'u', 'u'],
    ['r', 'r', 'u'],
])
try:
    walk(field, (3, 1))
    raise AssertionError("Should raise a ValueError when running in circles!")
except ValueError as e:
    assert str(e) == "I'm running in circles"
else:
    raise AssertionError("Should have raised the right error type!")

# test field with error
field = np.array([
    ['d', 'l', 'l'],
    ['d', 'w', 'u'],
    ['d', 'u', 'u'],
    ['r', 'r', 'u'],
])
try:
    walk(field, (2, 1))
    raise AssertionError("Should raise a ValueError when an invalid instruction is found!")
except ValueError as e:
    assert str(e) == "Invalid instruction found on position (1, 1)"
else:
    raise AssertionError("Should have raised the right error type!")

# test another field with error
field_with_error = np.array([
    ['r', 'x', 'd', 'l', 'u'],
    ['r', 'd', 'l', 'd', 'd'],
    ['u', 'r', 'd', 'r', 'l'],
    ['l', 'd', 'd', 'u', 'r'],
])
try:
    walk(field_with_error, (0, 0))
    raise AssertionError("Should raise a ValueError when an invalid instruction is found!")
except ValueError as e:
    assert str(e) == "Invalid instruction found on position (0, 1)"
else:
    raise AssertionError("Should have raised the right error type!")_____no_output_____

---_____no_output_____### Last but not least, submit your work!_____no_output_____To submit your work, fill your slack ID in the `slack_id` variable (as a string).

Example: `slack_id = "x-men"`

Help: if you forgot your slack ID, [read this](https://moshfeu.medium.com/how-to-find-my-member-id-in-slack-workspace-d4bba942e38c)._____no_output_____

# Submit your work!

#slack_id =

# YOUR CODE HERE
raise NotImplementedError()_____no_output_____from submit import submit
assert isinstance(slack_id, str)

slu = 18_3
submit(slack_id, slu)_____no_output_____

---_____no_output_____


# Notebook from ShepherdCode/IEEE.BIBM.2020
Path: notebooks/GRU_240.ipynb

# GRU 240
Test the RNA length effect._____no_output_____

NC_FILENAME='ncRNA.gc34.processed.fasta'
PC_FILENAME='pcRNA.gc34.processed.fasta'
DATAPATH=""
try:
    from google.colab import drive
    IN_COLAB = True
    PATH='/content/drive/'
    drive.mount(PATH)
    DATAPATH=PATH+'My Drive/data/'  # must end in "/"
    NC_FILENAME = DATAPATH+NC_FILENAME
    PC_FILENAME = DATAPATH+PC_FILENAME
except:
    IN_COLAB = False
    DATAPATH=""

EPOCHS=100  # use 2 for debug, 200 for real
K=3
VOCABULARY_SIZE=4**K+1   # e.g. K=3 => 64 DNA K-mers + 'NNN'
EMBED_DIMEN=16
FILENAME='GRU240'
NEURONS=16
ACT="tanh"
DROP=0.5
Drive already mounted at /content/drive/; to attempt to forcibly remount, call drive.mount("/content/drive/", force_remount=True).
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import ShuffleSplit
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedKFold
from sklearn.model_selection import StratifiedKFold
import tensorflow as tf
from tensorflow import keras
from keras.wrappers.scikit_learn import KerasRegressor
from keras.models import Sequential
from keras.layers import Bidirectional
from keras.layers import GRU
from keras.layers import Dense
from keras.layers import LayerNormalization
import time
dt='float32'
tf.keras.backend.set_floatx(dt)_____no_output_____

## Build model_____no_output_____

def compile_model(model):
    adam_default_learn_rate = 0.001
    schedule = tf.keras.optimizers.schedules.ExponentialDecay(
        initial_learning_rate = adam_default_learn_rate*10,
        #decay_steps=100000, decay_rate=0.96, staircase=True)
        decay_steps=10000, decay_rate=0.99, staircase=True)
    # learn rate = initial_learning_rate * decay_rate ^ (step / decay_steps)
    alrd = tf.keras.optimizers.Adam(learning_rate=schedule)
    bc=tf.keras.losses.BinaryCrossentropy(from_logits=False)
    print("COMPILE...")
    #model.compile(loss=bc, optimizer=alrd, metrics=["accuracy"])
    model.compile(loss=bc, optimizer="adam", metrics=["accuracy"])
    print("...COMPILED")
    return model

def build_model():
    embed_layer  = keras.layers.Embedding(
        #VOCABULARY_SIZE, EMBED_DIMEN, input_length=1000, input_length=1000, mask_zero=True)
        #input_dim=[None,VOCABULARY_SIZE], output_dim=EMBED_DIMEN, mask_zero=True)
        input_dim=VOCABULARY_SIZE, output_dim=EMBED_DIMEN, mask_zero=True)
    #rnn1_layer = keras.layers.Bidirectional(
    rnn1_layer = keras.layers.GRU(NEURONS, return_sequences=False,  #   True,
          input_shape=[1000,EMBED_DIMEN], activation=ACT, dropout=DROP)#)#bi
    #rnn2_layer = keras.layers.Bidirectional(
    rnn2_layer = keras.layers.GRU(NEURONS, return_sequences=False,
        activation=ACT, dropout=DROP)#)#bi
    dense1_layer = keras.layers.Dense(NEURONS, activation=ACT,dtype=dt)
    drop1_layer = keras.layers.Dropout(DROP)
    #dense2_layer = keras.layers.Dense(NEURONS, activation=ACT,dtype=dt)
    #drop2_layer = keras.layers.Dropout(DROP)
    output_layer = keras.layers.Dense(1, activation="sigmoid", dtype=dt)
    mlp = keras.models.Sequential()
    mlp.add(embed_layer)
    mlp.add(rnn1_layer)
    #mlp.add(rnn2_layer)
    mlp.add(dense1_layer)
    mlp.add(drop1_layer)
    #mlp.add(dense2_layer)
    #mlp.add(drop2_layer)
    mlp.add(output_layer)
    mlpc = compile_model(mlp)
    return mlpc_____no_output_____

## Load and partition sequences_____no_output_____

# Assume file was preprocessed to contain one line per seq.
# Prefer Pandas dataframe but df does not support append.
# For conversion to tensor, must avoid python lists.
def load_fasta(filename,label):
    DEFLINE='>'
    labels=[]
    seqs=[]
    lens=[]
    nums=[]
    num=0
    with open (filename,'r') as infile:
        for line in infile:
            if line[0]!=DEFLINE:
                seq=line.rstrip()
                num += 1   # first seqnum is 1
                seqlen=len(seq)
                nums.append(num)
                labels.append(label)
                seqs.append(seq)
                lens.append(seqlen)
    df1=pd.DataFrame(nums,columns=['seqnum'])
    df2=pd.DataFrame(labels,columns=['class'])
    df3=pd.DataFrame(seqs,columns=['sequence'])
    df4=pd.DataFrame(lens,columns=['seqlen'])
    df=pd.concat((df1,df2,df3,df4),axis=1)
    return df

def separate_X_and_y(data):
    y=   data[['class']].copy()
    X=   data.drop(columns=['class','seqnum','seqlen'])
    return (X,y)

_____no_output_____

## Make K-mers_____no_output_____

def make_kmer_table(K):
    npad='N'*K
    shorter_kmers=['']
    for i in range(K):
        longer_kmers=[]
        for mer in shorter_kmers:
            longer_kmers.append(mer+'A')
            longer_kmers.append(mer+'C')
            longer_kmers.append(mer+'G')
            longer_kmers.append(mer+'T')
        shorter_kmers = longer_kmers
    all_kmers = shorter_kmers
    kmer_dict = {}
    kmer_dict[npad]=0
    value=1
    for mer in all_kmers:
        kmer_dict[mer]=value
        value += 1
    return kmer_dict

KMER_TABLE=make_kmer_table(K)

def strings_to_vectors(data,uniform_len):
    all_seqs=[]
    for seq in data['sequence']:
        i=0
        seqlen=len(seq)
        kmers=[]
        while i < seqlen-K+1 -1:  # stop at minus one for spaced seed
            #kmer=seq[i:i+2]+seq[i+3:i+5]    # SPACED SEED 2/1/2 for K=4
            kmer=seq[i:i+K]
            i += 1
            value=KMER_TABLE[kmer]
            kmers.append(value)
        pad_val=0
        while i < uniform_len:
            kmers.append(pad_val)
            i += 1
        all_seqs.append(kmers)
    pd2d=pd.DataFrame(all_seqs)
    return pd2d   # return 2D dataframe, uniform dimensions_____no_output_____def make_kmers(MAXLEN,train_set):
    (X_train_all,y_train_all)=separate_X_and_y(train_set)
    X_train_kmers=strings_to_vectors(X_train_all,MAXLEN)
    # From pandas dataframe to numpy to list to numpy
    num_seqs=len(X_train_kmers)
    tmp_seqs=[]
    for i in range(num_seqs):
        kmer_sequence=X_train_kmers.iloc[i]
        tmp_seqs.append(kmer_sequence)
    X_train_kmers=np.array(tmp_seqs)
    tmp_seqs=None
    labels=y_train_all.to_numpy()
    return (X_train_kmers,labels)_____no_output_____def make_frequencies(Xin):
    Xout=[]
    VOCABULARY_SIZE= 4**K + 1  # plus one for 'NNN'
    for seq in Xin:
        freqs =[0] * VOCABULARY_SIZE
        total = 0
        for kmerval in seq:
            freqs[kmerval] += 1
            total += 1
        for c in range(VOCABULARY_SIZE):
            freqs[c] = freqs[c]/total
        Xout.append(freqs)
    Xnum = np.asarray(Xout)
    return (Xnum)
def make_slice(data_set,min_len,max_len):
    slice = data_set.query('seqlen < '+str(max_len)+' & seqlen>= '+str(min_len))
    return slice_____no_output_____

## Cross validation_____no_output_____

bestname=DATAPATH+FILENAME+".best"
def just_train(X,y): # give me training set
    model=build_model()
    print(model.summary())
    mycallbacks = [keras.callbacks.ModelCheckpoint(
        filepath=bestname, save_best_only=True,
        monitor='accuracy', mode='max')]
    print("FIT")
    start_time=time.time()
    history=model.fit(X_train, y_train, # batch_size=10, default=32 works nicely
            epochs=EPOCHS, verbose=1,  # verbose=1 for ascii art, verbose=0 for none
            callbacks=mycallbacks )  # no validation data
    end_time=time.time()
    elapsed_time=(end_time-start_time)
    print("Train %d epochs, %d sec"%(EPOCHS,elapsed_time))
    pd.DataFrame(history.history).plot(figsize=(8,5))
    plt.grid(True)
    plt.gca().set_ylim(0,1)
    plt.show()
def just_evaluate(X,y):  # give me validation set
    best_model=keras.models.load_model(bestname)
    scores = best_model.evaluate(X, y, verbose=0)
    print("%s: %.2f%%" % (best_model.metrics_names[1], scores[1]*100))_____no_output_____

## Train on RNA length-based slice_____no_output_____

#      A    B    C     D     E     F
# 200, 500, 750, 1000, 1500, 3000, 30000
print("Load data from files.")
nc_seq=load_fasta(NC_FILENAME,0)
pc_seq=load_fasta(PC_FILENAME,1)
together=pd.concat((nc_seq,pc_seq),axis=0)
train_set=together.sample(frac=1)   # shuffle
together=None
nc_seq=None
pc_seq=None
def get_x_and_y(train_set,minlen,maxlen):
  subset=make_slice(train_set,minlen,maxlen)
  (X_set,y_set)=make_kmers(maxlen,subset)
  print("Shape of X is "+str(X_set.shape))
  L=len(X_set)
  T=int(0.80*L+0.5)
  print("Total %d, Train portion %d"%(L,T))
  X_train=X_set[:T]
  y_train=y_set[:T]
  X_valid=X_set[T:]
  y_valid=y_set[T:]
  return (X_train,y_train,X_valid,y_valid)

MINLEN=200
MAXLEN=600
(X_train,y_train,X_valid,y_valid)=get_x_and_y(train_set,MINLEN,MAXLEN)
print("Loaded")Load data from files.
Shape of X is (8165, 600)
Total 8165, Train portion 6532
Loaded
print ("Train on ",MINLEN,"-",MAXLEN)
just_train(X_train,y_train)
Train on  200 - 600
COMPILE...
...COMPILED
Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
embedding_1 (Embedding)      (None, None, 16)          1040
_________________________________________________________________
gru_2 (GRU)                  (None, 16)                1632
_________________________________________________________________
dense_2 (Dense)              (None, 16)                272
_________________________________________________________________
dropout_1 (Dropout)          (None, 16)                0
_________________________________________________________________
dense_3 (Dense)              (None, 1)                 17
=================================================================
Total params: 2,961
Trainable params: 2,961
Non-trainable params: 0
_________________________________________________________________
None
FIT
Epoch 1/100
204/205 [============================>.] - ETA: 0s - loss: 0.5600 - accuracy: 0.7737INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 15s 71ms/step - loss: 0.5601 - accuracy: 0.7737
Epoch 2/100
205/205 [==============================] - ETA: 0s - loss: 0.5098 - accuracy: 0.7812INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 15s 75ms/step - loss: 0.5098 - accuracy: 0.7812
Epoch 3/100
205/205 [==============================] - 5s 26ms/step - loss: 0.5020 - accuracy: 0.7805
Epoch 4/100
203/205 [============================>.] - ETA: 0s - loss: 0.4949 - accuracy: 0.7843INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 15s 72ms/step - loss: 0.4952 - accuracy: 0.7841
Epoch 5/100
204/205 [============================>.] - ETA: 0s - loss: 0.4922 - accuracy: 0.7843INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 16s 77ms/step - loss: 0.4920 - accuracy: 0.7844
Epoch 6/100
205/205 [==============================] - ETA: 0s - loss: 0.4875 - accuracy: 0.7903INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 15s 71ms/step - loss: 0.4875 - accuracy: 0.7903
Epoch 7/100
205/205 [==============================] - 5s 25ms/step - loss: 0.5050 - accuracy: 0.7786
Epoch 8/100
205/205 [==============================] - 5s 25ms/step - loss: 0.4844 - accuracy: 0.7890
Epoch 9/100
205/205 [==============================] - 5s 25ms/step - loss: 0.4858 - accuracy: 0.7901
Epoch 10/100
204/205 [============================>.] - ETA: 0s - loss: 0.4761 - accuracy: 0.7960INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 16s 76ms/step - loss: 0.4761 - accuracy: 0.7959
Epoch 11/100
205/205 [==============================] - 5s 25ms/step - loss: 0.4725 - accuracy: 0.7941
Epoch 12/100
203/205 [============================>.] - ETA: 0s - loss: 0.4627 - accuracy: 0.7988INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 15s 72ms/step - loss: 0.4621 - accuracy: 0.7994
Epoch 13/100
204/205 [============================>.] - ETA: 0s - loss: 0.4598 - accuracy: 0.8013INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 16s 77ms/step - loss: 0.4597 - accuracy: 0.8013
Epoch 14/100
204/205 [============================>.] - ETA: 0s - loss: 0.4616 - accuracy: 0.8042INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 15s 72ms/step - loss: 0.4616 - accuracy: 0.8040
Epoch 15/100
203/205 [============================>.] - ETA: 0s - loss: 0.4556 - accuracy: 0.8087INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 16s 76ms/step - loss: 0.4550 - accuracy: 0.8089
Epoch 16/100
205/205 [==============================] - 5s 25ms/step - loss: 0.4511 - accuracy: 0.8066
Epoch 17/100
204/205 [============================>.] - ETA: 0s - loss: 0.4447 - accuracy: 0.8140INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 15s 72ms/step - loss: 0.4445 - accuracy: 0.8141
Epoch 18/100
204/205 [============================>.] - ETA: 0s - loss: 0.4425 - accuracy: 0.8142INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 15s 75ms/step - loss: 0.4424 - accuracy: 0.8143
Epoch 19/100
205/205 [==============================] - 5s 24ms/step - loss: 0.4443 - accuracy: 0.8129
Epoch 20/100
205/205 [==============================] - ETA: 0s - loss: 0.4414 - accuracy: 0.8161INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 14s 70ms/step - loss: 0.4414 - accuracy: 0.8161
Epoch 21/100
203/205 [============================>.] - ETA: 0s - loss: 0.4353 - accuracy: 0.8167INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 15s 75ms/step - loss: 0.4358 - accuracy: 0.8164
Epoch 22/100
203/205 [============================>.] - ETA: 0s - loss: 0.4342 - accuracy: 0.8199INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 14s 70ms/step - loss: 0.4338 - accuracy: 0.8203
Epoch 23/100
205/205 [==============================] - ETA: 0s - loss: 0.4279 - accuracy: 0.8241INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 15s 75ms/step - loss: 0.4279 - accuracy: 0.8241
Epoch 24/100
204/205 [============================>.] - ETA: 0s - loss: 0.4144 - accuracy: 0.8249INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 14s 71ms/step - loss: 0.4150 - accuracy: 0.8247
Epoch 25/100
204/205 [============================>.] - ETA: 0s - loss: 0.4030 - accuracy: 0.8329INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 15s 74ms/step - loss: 0.4029 - accuracy: 0.8330
Epoch 26/100
204/205 [============================>.] - ETA: 0s - loss: 0.3910 - accuracy: 0.8408INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 14s 70ms/step - loss: 0.3912 - accuracy: 0.8406
Epoch 27/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3839 - accuracy: 0.8397
Epoch 28/100
203/205 [============================>.] - ETA: 0s - loss: 0.3797 - accuracy: 0.8444INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 16s 76ms/step - loss: 0.3792 - accuracy: 0.8448
Epoch 29/100
204/205 [============================>.] - ETA: 0s - loss: 0.3755 - accuracy: 0.8456INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 14s 69ms/step - loss: 0.3756 - accuracy: 0.8455
Epoch 30/100
204/205 [============================>.] - ETA: 0s - loss: 0.3603 - accuracy: 0.8506INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 16s 76ms/step - loss: 0.3603 - accuracy: 0.8506
Epoch 31/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3691 - accuracy: 0.8455
Epoch 32/100
204/205 [============================>.] - ETA: 0s - loss: 0.3638 - accuracy: 0.8509INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 14s 70ms/step - loss: 0.3636 - accuracy: 0.8510
Epoch 33/100
203/205 [============================>.] - ETA: 0s - loss: 0.3602 - accuracy: 0.8533INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 15s 75ms/step - loss: 0.3607 - accuracy: 0.8527
Epoch 34/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3648 - accuracy: 0.8497
Epoch 35/100
204/205 [============================>.] - ETA: 0s - loss: 0.3560 - accuracy: 0.8534INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 14s 70ms/step - loss: 0.3560 - accuracy: 0.8533
Epoch 36/100
203/205 [============================>.] - ETA: 0s - loss: 0.3603 - accuracy: 0.8538INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 15s 74ms/step - loss: 0.3594 - accuracy: 0.8543
Epoch 37/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3608 - accuracy: 0.8512
Epoch 38/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3554 - accuracy: 0.8533
Epoch 39/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3599 - accuracy: 0.8491
Epoch 40/100
205/205 [==============================] - 5s 26ms/step - loss: 0.3565 - accuracy: 0.8526
Epoch 41/100
205/205 [==============================] - 5s 26ms/step - loss: 0.3577 - accuracy: 0.8520
Epoch 42/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3624 - accuracy: 0.8468
Epoch 43/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3568 - accuracy: 0.8475
Epoch 44/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3551 - accuracy: 0.8497
Epoch 45/100
205/205 [==============================] - ETA: 0s - loss: 0.3512 - accuracy: 0.8549INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 15s 74ms/step - loss: 0.3512 - accuracy: 0.8549
Epoch 46/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3533 - accuracy: 0.8536
Epoch 47/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3522 - accuracy: 0.8518
Epoch 48/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3539 - accuracy: 0.8504
Epoch 49/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3530 - accuracy: 0.8509
Epoch 50/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3552 - accuracy: 0.8529
Epoch 51/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3481 - accuracy: 0.8527
Epoch 52/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3467 - accuracy: 0.8541
Epoch 53/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3521 - accuracy: 0.8517
Epoch 54/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3465 - accuracy: 0.8544
Epoch 55/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3463 - accuracy: 0.8536
Epoch 56/100
203/205 [============================>.] - ETA: 0s - loss: 0.3449 - accuracy: 0.8587INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 14s 70ms/step - loss: 0.3446 - accuracy: 0.8590
Epoch 57/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3438 - accuracy: 0.8550
Epoch 58/100
205/205 [==============================] - 5s 26ms/step - loss: 0.3423 - accuracy: 0.8556
Epoch 59/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3456 - accuracy: 0.8526
Epoch 60/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3449 - accuracy: 0.8541
Epoch 61/100
205/205 [==============================] - ETA: 0s - loss: 0.3408 - accuracy: 0.8593INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 14s 70ms/step - loss: 0.3408 - accuracy: 0.8593
Epoch 62/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3407 - accuracy: 0.8590
Epoch 63/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3444 - accuracy: 0.8555
Epoch 64/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3412 - accuracy: 0.8553
Epoch 65/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3369 - accuracy: 0.8561
Epoch 66/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3432 - accuracy: 0.8564
Epoch 67/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3358 - accuracy: 0.8582
Epoch 68/100
205/205 [==============================] - ETA: 0s - loss: 0.3342 - accuracy: 0.8619INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 15s 74ms/step - loss: 0.3342 - accuracy: 0.8619
Epoch 69/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3415 - accuracy: 0.8552
Epoch 70/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3409 - accuracy: 0.8572
Epoch 71/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3320 - accuracy: 0.8602
Epoch 72/100
203/205 [============================>.] - ETA: 0s - loss: 0.3284 - accuracy: 0.8635INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 15s 73ms/step - loss: 0.3291 - accuracy: 0.8631
Epoch 73/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3400 - accuracy: 0.8607
Epoch 74/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3365 - accuracy: 0.8619
Epoch 75/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3345 - accuracy: 0.8624
Epoch 76/100
203/205 [============================>.] - ETA: 0s - loss: 0.3299 - accuracy: 0.8639INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 14s 69ms/step - loss: 0.3294 - accuracy: 0.8641
Epoch 77/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3400 - accuracy: 0.8570
Epoch 78/100
204/205 [============================>.] - ETA: 0s - loss: 0.3307 - accuracy: 0.8673INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 15s 75ms/step - loss: 0.3306 - accuracy: 0.8674
Epoch 79/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3347 - accuracy: 0.8618
Epoch 80/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3297 - accuracy: 0.8651
Epoch 81/100
205/205 [==============================] - 5s 26ms/step - loss: 0.3291 - accuracy: 0.8645
Epoch 82/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3284 - accuracy: 0.8642
Epoch 83/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3332 - accuracy: 0.8598
Epoch 84/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3315 - accuracy: 0.8630
Epoch 85/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3321 - accuracy: 0.8650
Epoch 86/100
205/205 [==============================] - 5s 23ms/step - loss: 0.3290 - accuracy: 0.8673
Epoch 87/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3278 - accuracy: 0.8662
Epoch 88/100
203/205 [============================>.] - ETA: 0s - loss: 0.3256 - accuracy: 0.8693INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 15s 71ms/step - loss: 0.3250 - accuracy: 0.8699
Epoch 89/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3248 - accuracy: 0.8674
Epoch 90/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3249 - accuracy: 0.8641
Epoch 91/100
205/205 [==============================] - ETA: 0s - loss: 0.3178 - accuracy: 0.8720INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 16s 76ms/step - loss: 0.3178 - accuracy: 0.8720
Epoch 92/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3226 - accuracy: 0.8671
Epoch 93/100
205/205 [==============================] - 5s 26ms/step - loss: 0.3204 - accuracy: 0.8683
Epoch 94/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3244 - accuracy: 0.8694
Epoch 95/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3222 - accuracy: 0.8677
Epoch 96/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3276 - accuracy: 0.8670
Epoch 97/100
203/205 [============================>.] - ETA: 0s - loss: 0.3137 - accuracy: 0.8733INFO:tensorflow:Assets written to: /content/drive/My Drive/data/GRU240.best/assets
205/205 [==============================] - 14s 70ms/step - loss: 0.3143 - accuracy: 0.8729
Epoch 98/100
205/205 [==============================] - 5s 25ms/step - loss: 0.3191 - accuracy: 0.8706
Epoch 99/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3155 - accuracy: 0.8720
Epoch 100/100
205/205 [==============================] - 5s 24ms/step - loss: 0.3156 - accuracy: 0.8709
Train 100 epochs, 869 sec
print ("Evaluate on A")
(X_train,y_train,X_valid_A,y_valid_A)=get_x_and_y(train_set,200,600)
just_evaluate(X_valid_A,y_valid_A)
(X_valid_A,y_valid_A)=(None,None)
print ("Evaluate on B")
(X_train,y_train,X_valid_B,y_valid_B)=get_x_and_y(train_set,600,900)
just_evaluate(X_valid_B,y_valid_B)
(X_valid_B,y_valid_B)=(None,None)
print ("Evaluate on C")
(X_train,y_train,X_valid_C,y_valid_C)=get_x_and_y(train_set,900,1300)
just_evaluate(X_valid_C,y_valid_C)
(X_valid_C,y_valid_C)=(None,None)
print ("Evaluate on D")
(X_train,y_train,X_valid_D,y_valid_D)=get_x_and_y(train_set,1300,1900)
just_evaluate(X_valid_D,y_valid_D)
(X_valid_D,y_valid_D)=(None,None)
print ("Evaluate on E")
(X_train,y_train,X_valid_E,y_valid_E)=get_x_and_y(train_set,1900,3000)
just_evaluate(X_valid_E,y_valid_E)
(X_valid_E,y_valid_E)=(None,None)
print ("Evaluate on F")
(X_train,y_train,X_valid_F,y_valid_F)=get_x_and_y(train_set,3000,30000)
just_evaluate(X_valid_F,y_valid_F)
(X_valid_F,y_valid_F)=(None,None)

print ("Done")Evaluate on A
Shape of X is (8165, 600)
Total 8165, Train portion 6532
accuracy: 88.18%
Evaluate on B
Shape of X is (6442, 900)
Total 6442, Train portion 5154
accuracy: 78.49%
Evaluate on C
Shape of X is (4478, 1300)
Total 4478, Train portion 3582
accuracy: 69.75%
Evaluate on D
Shape of X is (4421, 1900)
Total 4421, Train portion 3537
accuracy: 66.29%
Evaluate on E
Shape of X is (4590, 3000)
Total 4590, Train portion 3672
accuracy: 64.05%
Evaluate on F
Shape of X is (3900, 30000)
Total 3900, Train portion 3120
accuracy: 75.64%
Done


# Notebook from yangjy0826/Quantum-Study
Path: edX-UToronto-QML/Coding assignments/Quantum Systems/03_Evolution_in_Closed_and_Open_Systems.ipynb

Before you begin, execute this cell to import numpy and packages from the D-Wave Ocean suite, and all necessary functions the gate-model framework you are going to use, whether that is the Forest SDK or Qiskit. In the case of Forest SDK, it also starts the qvm and quilc servers._____no_output_____

%run -i "assignment_helper.py"Available frameworks:
Forest SDK
Qiskit
D-Wave Ocean

# Unitary evolution

**Exercise 1** (1 point). The conjugate transpose of a unitary matrix is its inverse. Write a function that checks if a $2\times 2$ numpy array is unitary._____no_output_____

def is_unitary(matrix):
    ###
    ### YOUR CODE HERE
    ###
    conjugate_transpose = np.array(np.matrix(matrix).getH())
    inverse = np.linalg.inv(matrix)
    #print(conjugate_transpose)
    #print(inverse)
    if ((conjugate_transpose - inverse < 0.01).all()):
        return True
    else:
        return False
    #return ((conjugate_transpose == inverse).all())
X = np.array([[0, 1], [1, 0]])
H = 1/np.sqrt(2)*np.array([[1, 1], [1, -1]])
random_matrix = np.random.rand(2, 2)
print(is_unitary(X))
print(is_unitary(H))
print(is_unitary(random_matrix))True
True
False
X = np.array([[0, 1], [1, 0]])
H = 1/np.sqrt(2)*np.array([[1, 1], [1, -1]])
random_matrix = np.random.rand(2, 2)
assert is_unitary(X)
assert is_unitary(H)
assert is_unitary(random_matrix) is False_____no_output_____

**Exercise 2** (1 point). The eigenvalues of unitary operators always lie on the complex unit circle. We exploit this property over and over again. For instance, the quantum phase estimation algorithm estimates the eigenvalues through the phase that describes the eigenvalues angle on the complex unit circle. Calculate the eigenvalues of the Hadamard gate and store them in a numpy array called `eigenvalues`._____no_output_____

###
### YOUR CODE HERE
###
H = 1/np.sqrt(2)*np.array([[1, 1], [1, -1]])
w, v = np.linalg.eig(H)
eigenvalues = w_____no_output_____###
### AUTOGRADER TEST - DO NOT REMOVE
###
_____no_output_____

**Exercise 3** (1 point). Unitary operations are reversible. In fact, we often reverse them on purpose: this is called uncomputing, and used, for instance, in the quantum matrix inversion algorithm that underlies many coherent quantum machine learning protocols. The reason we do that is to remove the entanglement we might have created, so that tracing out a subsystem would not interfere with our calculations (the remaining system will not be a mixed state). Create a circuit in your preferred framework that applies the X and H gates in this sequence. The circuit should be in an object called `circuit`._____no_output_____

###
### YOUR CODE HERE
###
q = QuantumRegister(1, 'q') # Create a Quantum Register with n qubits.
register = QuantumCircuit(q) # Create a Quantum Circuit acting on the q register
register.x(q[0])
register.h(q[0])

# Create a Classical Register with n bits.
c = ClassicalRegister(1, 'c')
# Create a Quantum Circuit
measure = QuantumCircuit(q, c)

circuit = register + measure_____no_output_____amplitudes = get_amplitudes(circuit)
assert all(np.isclose(amplitudes, np.array([1/np.sqrt(2), -1/np.sqrt(2)])))_____no_output_____

**Exercise 4** (1 point). Extend this circuit to reverse the calculations you have done._____no_output_____

###
### YOUR CODE HERE
###
q = QuantumRegister(1, 'q') # Create a Quantum Register with n qubits.
register = QuantumCircuit(q) # Create a Quantum Circuit acting on the q register
register.x(q[0])
register.h(q[0])

register.h(q[0])
register.x(q[0])

# Create a Classical Register with n bits.
c = ClassicalRegister(1, 'c')
# Create a Quantum Circuit
measure = QuantumCircuit(q, c)

circuit = register + measure_____no_output_____amplitudes = get_amplitudes(circuit)
program_length = get_circuit_length(circuit)
assert all(np.isclose(amplitudes, np.array([1, 0]))), "The circuit is not successfully reversed"
assert program_length == 4, "The circuit does not have the right number of gates"_____no_output_____

# Interaction with the environment: open systems

**Exercise 5** (2 points). Contemporary quantum computers are open systems: they interact with the environment. The nature of the interaction can often be characterized by noise models. Even if the system starts in a pure state, the interaction with the environment will turn it into a mixed state. The state loses coherence (the off-diagonal terms in the density matrix). Furthermore, increasing noise may completely destroy entanglement, so we can no longer use it as a resource for calculations. In general, it is a very hard problem to tell how much noise can be tolerated so that the remaining state is still useful for some calculations.

We introduced a simple function called `mixed_state` that interpolated a pure state with the maximally mixed state $\mathbb{1}/d$, where $d$ is the dimension of the system. We also talked about thermal states: at infinite temperature, you recover the uniform distribution over the energy levels of the system. This is, in fact, is just the maximally mixed state.

We talk about energy levels and Hamiltonians in the subsequent notebook. For the time being, imagine that you have a qubit system: this has two levels, in other words, two possible energies. Let's make these -1.0 and +1.0. The thermal state of this qubit will have the Boltzmann distribution in its diagonal. Calculate this density matrix at temperature $T=1$ and store it in a numpy array called `dm`._____no_output_____

import numpy as np
energies = np.array([-1.0, +1.0])
T = 1
###
### YOUR CODE HERE
###
#mixed_state = np.array([[0.5, 0], [0, 0.5]])
p_unnorm0 = np.exp(-energies[0]/T)
p_unnorm1 = np.exp(-energies[1]/T)
p_0 = p_unnorm0/(p_unnorm0+p_unnorm1)
p_1 = p_unnorm1/(p_unnorm0+p_unnorm1)
q0 = np.array([[1], [0]])
q1 = np.array([[0], [1]])
dm = p_0 * np.dot(q0, q0.T) + p_1 * np.dot(q1, q1.T)
print(dm)[[0.88079708 0.        ]
 [0.         0.11920292]]
###
### AUTOGRADER TEST - DO NOT REMOVE
###
_____no_output_____

The thermal state itself is interesting for calculations. It is the property of a system in equilibrium with the environment. Classical simulations of achieving an equilibrium are often computationally expensive: for instance, inference in probabilistic graphical models requires this. This is where quantum protocols that create a thermal state become important._____no_output_____


# Notebook from soundofives/hpu-classes
Path: physics/classical/SHO/SHOundamped.ipynb

import numpy as np
from scipy import special
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit_____no_output_____#Parameters
omega0 = 1                       #angular frequency

x0 = 1                           #initial position
v0 = 0                           #initial velocity
beta = 0.5
n = 2

tmax = 100                       #max integration time
dt = 0.01                        #time step
Nt = int(np.floor(tmax/dt)+1)    #number of time steps (fixed by tmax, dt)

t = np.linspace(0,tmax,Nt)       #array of time values
x = np.zeros(Nt)                 #array to store x values
v = np.zeros(Nt)                 #array to store v values

x[0] = x0
v[0] = v0_____no_output_____

$x_{0}e^{\frac{-\beta\omega_{0}^2}{4k^{2}m^{2}}t} = A$

$\frac{1}{\frac{2\beta\omega_{0}^{2}t}{3km}+x_{0}^{-1}} = A$

$\frac{-\beta}{km}t+x_{0} = A$_____no_output_____

#time evolution
i=0
fv = beta*np.abs(v[i])**(n-1)*v[i]
for i in range(0,Nt-1):

    v[i+1] = v[i] + dt*(-(omega0**2)*x[i]) - fv #Simple forward-Euler integration
    x[i+1] = x[i] + dt*v[i+1]                   #BUT using v[i+1] here actually improves the
                                                #accuracy more than you'd expect if you ever
                                                #coded RK4 from scratch (that's still better)_____no_output_____

## UNDAMPED_____no_output_____

#analytic undamped
xa = x0*np.cos(omega0*t)

#for n = 0
#xa = x0*np.cos(omega0*t)*((-beta*t)/(k*m))
#analytic solution for n = 1
#xa = x0*np.sin(omega0*t)*np.exp((-beta*omega0**2*t)/(4*k**2*m**2))
#for n = 2
#xa = x0*np.sin(omega0*t)*1/(((2*beta*omega0**2*t)/(3*k*m))+x0**(-1))_____no_output_____plt.plot(t,x, '.')
plt.plot(t,xa, 'r-')
plt.legend(['numerical','analytic'])
plt.xlabel('time (s)')
plt.ylabel('displacement (m)')
plt.show()_____no_output_____plt.figure()
plt.plot(t,np.abs(xa-x))
plt.title('Error in Euler Integration')_____no_output_____

1. At long time scales, the error in the Euler numerical integration grows significantly in time.

2. The error appears to grow slightly more than linearly with time, though not much more. The error can be significantly lowered with lower time steps. With a $dt$ of $0.01$, our error reached $0.6$ at a time of $100$, whereas with a $dt$ of $0.001$ our error at the same time was $0.05$.

3. Adding some basic RK4 fitting gives a basically non-increasing error or $0.0005$ with the previous $dt$.

7. The approximations seems somewhat weak--the n=0 analytic expression is especially off. It grows in time instead of depreciating! The n=2 is very close, especially at long time scales. The n=1 is similarly good, though less accurate than the n=2 case._____no_output_____## N = 0 CASE_____no_output_____

#This case needs a special condition--eventually the A(t) function goes negative--at this
#point the amplitude should die to 0.
def A0(x0,t):
    if (-beta*t/(k*m) + x0) > 0:
        return (-beta*t/(k*m) + x)
    else:
        return 0.0_____no_output_____drag2 = np.genfromtxt('Drag2.csv', delimiter=',')

k = 5
m = 1
x = .38
w = 5.3
beta = 0.2

t = np.linspace(0,10,1000)

a0 = np.zeros(np.size(t))
for i in range(0,np.size(t)):
    a0[i] = A0(x,t[i])*np.sin(w*t[i])

drag2t = drag2[27153:, 0]
drag2v = drag2[27153:, 1]
drag2t = drag2t-drag2t[0]

plt.figure(figsize=[20,10])
plt.plot(drag2t,drag2v, 'r-',label='experimental')
plt.plot(t,a0,'b-',label='theoretical')
plt.xlabel('Time (s)')
plt.ylabel('Voltage (V)')
plt.title('Theoretical and Experimental Comparison for Frictional Damping')
plt.show()_____no_output_____xpeaks=[]
tpeaks=[]
a=10
for i in range(a,np.size(t)-a):
    if((a0[i]-a0[i-a])*(a0[i+a]-a0[i])<0):
        xpeaks.append(np.abs(a0[i]))
        tpeaks.append(np.abs(t[i]))_____no_output_____line = lambda x, m, b: m*x+b

xA = np.asarray(xpeaks)
tA = np.asarray(tpeaks)

fit, covar = curve_fit(line, tpeaks, xpeaks)
err = np.sqrt(np.diag(covar))

print(fit)[-0.03962794  0.37570944]
plt.figure(figsize=[20,10])
plt.plot(tpeaks,xpeaks,label='peak trend line')
plt.plot(tpeaks, line(tA, *fit),label='functional fit')
plt.legend(loc='best')
plt.xlabel('Time (s)')
plt.ylabel('Voltage (V)')
plt.title('Fit for the decay trend for the Frictional Damping Case')
plt.show()_____no_output_____

## N = 1_____no_output_____

def A1(x,t):
    return x*np.exp(-beta*w*t/(4*m*k))_____no_output_____drag1 = np.genfromtxt('Drag1.csv', delimiter=',')

k = 2
m = 1
x = .38
w = 5.35
beta = 0.3

t = np.linspace(0,10,1000)

drag1t = drag1[:, 0]
drag1v = drag1[:, 1]

a1 = A1(x,t)*np.sin(w*t)

plt.figure(figsize=[20,10])
plt.plot(drag2t,drag2v, 'r-',label='experimental')
plt.plot(t,a1,'b-',label='theoretical')
plt.xlabel('Time (s)')
plt.ylabel('Voltage (V)')
plt.title('Theoretical and Experimental Comparison for Magnetic Damping')
plt.show()_____no_output_____xpeaks2=[]
tpeaks2=[]
a=10
for i in range(a,np.size(t)-a):
    if((a1[i]-a1[i-a])*(a1[i+a]-a1[i])<0):
        xpeaks2.append(np.abs(a1[i]))
        tpeaks2.append(np.abs(t[i]))

quad = lambda x,a,b,c: a*x**2+b*x+c
xA2 = np.asarray(xpeaks2)
tA2 = np.asarray(tpeaks2)
fit2, covar2 = curve_fit(quad, tpeaks2, xpeaks2)
err = np.sqrt(np.diag(covar2))
print(fit2)

plt.figure(figsize=[20,10])
plt.plot(tpeaks2,xpeaks2,label='peak trend line')
plt.plot(tpeaks2, quad(tA2, *fit2),label='functional fit')
plt.legend(loc='best')
plt.xlabel('Time (s)')
plt.ylabel('Voltage (V)')
plt.title('Fit for the decay trend for the Magnetic Damping Case')
plt.show()[ 1.19137795e-04 -1.20457673e-02  3.63717978e-01]

## N = 2_____no_output_____

def A2(x,t):
    return x/(1+(2*beta*t)/(m*w*k))_____no_output_____paddle = np.genfromtxt('PaddleData.csv', delimiter=',')

k = .1
m = 4
x = 1
w = 2.6

t = np.linspace(0,50,1000)

paddlet = paddle[4:,0]
paddlev = paddle[4:,1]

a2 = A2(x,t)*np.cos(w*t)

plt.figure(figsize=[20,10])
plt.plot(paddlet,paddlev, 'r-',label='experimental')
plt.plot(t,a2,'b-',label='theoretical')
plt.xlabel('Time (s)')
plt.ylabel('Voltage (V)')
plt.legend(loc='best')
plt.title('Theoretical and Experimental Comparison for Damping by Air Resistance')
plt.show()_____no_output_____expon = lambda x,a,lam: a*np.exp(x*lam)_____no_output_____xpeaks3=[]
tpeaks3=[]
a=2
for i in range(a,np.size(t)-a):
    if((a2[i]-a2[i-a])*(a2[i+a]-a2[i])<0):
        xpeaks3.append(np.abs(a2[i]))
        tpeaks3.append(np.abs(t[i]))

xA3 = np.asarray(xpeaks3)
tA3 = np.asarray(tpeaks3)

fit3, covar3 = curve_fit(expon, tpeaks3, xpeaks3,p0=(1,0))
err = np.sqrt(np.diag(covar3))
print(fit3)

plt.figure(figsize=[20,10])
plt.plot(tpeaks3,xpeaks3,label='peak trend line')
plt.plot(tpeaks3, expon(tA3, *fit3),label='functional fit')
plt.legend(loc='best')
plt.xlabel('Time (s)')
plt.ylabel('Voltage (V)')
plt.title('Fit for the decay trend for Damping by Air Resistance')
plt.show()[ 0.53470449 -0.11030997]


# Notebook from revendrat/TemperatureR
Path: assignment.ipynb

<code>
import pandas as pd
import random_____no_output_____data=pd.read_csv(r'D:\MBA BITS Pilani\Python Programming\data.csv')_____no_output_____#Store and print the rows of the column 'headline_category' in a single variable 'category'
data["category"]=data["headline_category"]_____no_output_____data.head()_____no_output_____#Store and print the rows of the column 'headline_text' in a single variable 'headlines'
data["headlines"]=data["headline_text"]_____no_output_____#Check the data-type of each column of the dataset.
data.dtypes_____no_output_____# Pick a line from the 'headline_text' using a generated random number and store it in variable 'line'.
line=data.loc[random.randint(0,len(data))][{"headlines"}]
line
# Challenge : Do it in a single line of code. (It can be displayed using another line of code.)_____no_output_____# Convert each letter of the 'line' variable to uppercase.
line_upper=line.str.upper()
line_upper_____no_output_____# Convert each letter of the 'line' variable to lowercase.
line_lower=line.str.lower()
line_lower_____no_output_____# Find the length of the string stored in 'line'.
line.str.len()_____no_output_____# Capitalize the string store in 'line'.
line_capitalize=line.str.title()
line_capitalize_____no_output_____line.str.title()[0]_____no_output_____# Check if the string begins with the word 'The'.
"The" in line.str.title()[0]_____no_output_____# Challenge: Find the number of strings that start with the word 'The' in the dataset column 'headline_text'.
for k in data.index:
    if "The" in data["headline_text"][k]:
        print(data["headline_text"][k])
# It may take time if you use a for loop. Try some other approach?The best music of the year
They are here to help us
These noisy spots are hurting your ears
The string that pulled Stephen Hawking to India
The saga of Usha Stud Farm
The string that pulled Stephen Hawking to India
The string that pulled Stephen Hawking to India
The saga of Usha Stud Farm
The Sphinx can act too
New year hash: They paid Rs 18;000; went hungry
Kamal Haasan: Hey (Thenali) Ram
The Bali beat goes beyond bhangra
Kamal Haasan: Hey (Thenali) Ram
The policemen find a new excuse: Television in slums
The policemen find a new excuse: Television in slums
The Kamasutra girl is real cool
The best things in life
Towards a Theory of Everything - II
The Veggie Abroad
Security bandobast: The one that got away
'Temptation Island' beats 'The Mole'
The court has faith in us: Jagmohan
The sound of Alabina
The life-style disease leads to blindness
The unpredictable calamity
The Learning Curve
The unpredictable calamity
The town that was once Bhuj
The Word Factory
The Kiteman cometh
The town that lay forgotten for 36 hours
The Enchanted Forester
The question that started the Hinduja affair
The sound of Alabina
The sound of Alabina
The Bhuj Quake: FAQs
The Enemy Within
Sikander: The son also rises!
Sikander: The son also rises!
There will be more taxes; says Vajpayee
There will be more taxes; says Vajpayee
The Dalai Lama: A Man With A Mission
The final verdict: It touched 8.1 on Richter
The Ground beneath their feet
The Ground beneath their feet
The well-oiled machinery of civil society
These criminals must get their due
They 'bought' enough gelatin to blow up Karnataka
The Match Factor
The Kachchi Sprint
The Art of Healing
The Real and the Unreal
The other Rajeev
The inimitable Mamata
The Page Three Politician
The art of science
The three Sunitas
The many avataars Subbirami Reddy
Remember The Titans
These young citizens are voicing their concern
The old man in his labyrinth
The Purpose of Our Life is to 'Know'
The killing mines of Dhanbad
Ban on smoking: The Kerala experience
Human Evolution and The Ten Avatars
There's no use shying away from shyness
The Golden Genome
Anupama Verma: The grahan has been wiped off
The Many Benefits Of Daily Prayer
The salsa was superb; and who's your caterer?
The healing touch
Their bamboo music is hard to beat
The city on a school wall
They help women overcome barrier; achieve success
Janasabha: The catalyst between people; departments
The emperor of cricket is dead
The Don misses another century
The Planet's Plight
The Kingdom of God Is Within Each of Us
The Arthashastra
The Buddha Smiles
The Power of One
The man behind the hot seat
De-Islamizing Islam: The Taliban Project
The Lady's not for Turning
The power broker
The restless style guru
The Sonia magic hasn't quite faded
Exposed! Sonia's letter to Jaya on playing a Thehelka trick
The PMO strikes back
The death of a freedom fighter
The Sound of Music
The Learning Curve
The Bingo Brigade
The Comeback Kids
The Ultimate Holy Book of Humanity
The magical world of P C Sorcar
The happiness of a long-distance runner
The censors seem to have lost their scissors
Question Time The appointment of George Verghese
Anil Kapoor: They've heard his Pukar!
The sound of Alabina
The Middle Path
The master blaster as a spin-doctor
The master blaster as a spin-doctor
The flip side of ayurvedic products
Tarun Tejpal: The shock jock
'There's genius in all of us'
The earth mother
The dashing dadima
The devil in Delhiites
The Amish People
The long and short of the growing problem
The long and short of the growing problem
All The Pretty Horses
The candidates are polls apart
The Good Wife
The Cross Symbolises Life & Redemption
The Stray Pixels
These girls talk suspension; not boybands
The Stray Pixels
The Stray Pixels
The inner landscapes
The mind-body
The lady
Where The Tax Axe Falls
The Philosophical Basis of Religion
The Other Abdullah
Azim Premji: The brain behind Wipro's winning streak
The Bushman
Mahesh Manjrekar: The man behind the gimmicks
The road not taken
There's a serial killer on the Net
The Bandh Wagon
Puru Rajkumar: The word is 'mum'!
The Road Not Taken
The Forgotten Scam
Fare Thee Well
The Forgotten Scam
Fare Thee Well
# Extract the first 6 characters of each row of the column 'headline_text' of the dataset.
data["headline_text"].str[0:7]_____no_output_____# Split the words of the string stored in the 'line' variable and print the first 2 words.
print(line.str.split()[0][0:2])['Cola', 'wars']
#Challenge: Print all the unique words in the variable 'line'. Hint: Use the 'split' function.
uniqueList=[]
for word in line.str.split()[0]:
    if word not in uniqueList:
        uniqueList.append(word)

uniqueList_____no_output_____#Find the position of the word 'The' in the 'line' variable, if not present then search for any word that
#you know that is present in that string.
uniqueList.index("The")
#What is printed when the word isn't present in the string? Google why!_____no_output_____#Find the number of times the word 'The' occurs in the 'line' variable.
line[0].count("The")_____no_output_____# Print characters from string stored in 'line' variable by skipping each second letter.
# Note: Whitespaces are counted as characters._____no_output_____# Challenge: Find the word that occurs with the highest frequency in the string stored in the 'line' variable.
hcount=0
hmax=0
for word in line[0].split():
    hcount=line[0].count(word)
    if hcount>hmax:
        hmax=hcount
        maxindex=line[0].index(word)

print(line[0].split()[maxindex])
# Print 'None' if all words have the same frequency.The
for word in line[0].split():
    print(word)The
Learning
Curve
# Final Challenge: Find the frequency of each character in the string stored in 'line' variable in the form of a dictionary.
wordDict={}
trackWord=[]
for word in line.str.replace(" ","")[0]:
    if word not in trackWord:
        trackWord.append(word)
        wordDict[word]=0
    if word in trackWord:
        wordDict[word]+=1
print(wordDict)
# No output has been provided to make sure the student tries to solve a problem with no hint. :){'T': 1, 'h': 1, 'e': 3, 'L': 1, 'a': 1, 'r': 2, 'n': 2, 'i': 1, 'g': 1, 'C': 1, 'u': 1, 'v': 1}

</code>