# Notebook from karthikchiru12/SARAMSH-TFIDF-Text-Summarizer
Path: demo.ipynb

<code>
from saramsh_package.saramsh import Saramsh_____no_output_____data='''
She isn’t one to shy away from taking up roles which are out of her comfort zone. Actor Divya Dutta, be it in Delhi 6 (2009), Bhaag Milkha Bhaag (2013), or Badlapur (2015), has always managed to leave an impact on the audience, the length of the role irrespective.

Ask her however, if initially she had plans to become a commercial, mainstream film actor which people aspire, complete with drama and dance, she quips, “Yes!... I realised however, after Veer-Zara (2004), that the significance of a role is not just about being glamorous. What you give to a role is what’s integral to the story. Now everything is so story and character oriented… in fact, one of my directors was saying ‘you know that big heroine? She wants to do your role’.”

She goes on to add that she has seen actor turn down roles because they were ‘small’. “Earlier, we used to say chhota role nahi karenge. I never said that, but I have seen girls say it. Now they say ‘chhota hai, koi baat nahi, impactful hai na?’”

As we talk about the length of the role not mattering, we mention to her the debates that arise around casting, such as Bhumi Pednekar and Taapsee Pannu playing older characters on screen (instead of old actors being cast), Bhumi being made to change her skin’s colour for Bala.
Divya herself had to colour her skin for Delhi 6. “You had Anupam Kher play an old man in Saraansh, Nargis ji played Mother India at the peak of her career… it’s not about who is playing it, it’s how you play it. The film you mentioned, if something was questionable, it was makeup and VFX. I once lost a role because I was told I am fair. They were looking for a village woman who is darker. They told me ‘you suit the role completely’. I asked ‘then why am I not doing it?’ They replied ‘you are too fair for the role’. I felt a little cheated at not being able to do something because of skin colour. But I am an actor. I was darkened for Delhi 6. I used to go to set at 4 in the morning, mera mazaak udta tha for the call time. I used to take so long to get ready. But here was a director (Rakeysh Omprakash Mehra) who interested me with a role. I saw Saand Ki Aankh, I thought the girls were young, they did a good job. Thodi der ke baad main bhool gayi (they are younger actors playing older characters),” says Divya.
'''
inshorts='''Actress Divya Dutta has revealed she once lost a role because she was told she was "too fair". "They were looking for a village woman who's darker. They told me 'you suit the role completely'," she added, stating her complexion was cited as the reason behind rejecting her. Divya further revealed she had to darken her skin for 'Delhi-6'.'''
title = '''Divya Dutta: ‘I once lost a role because I was told I am fair’''' _____no_output_____sm = Saramsh(data,title)
sm.summarize()0.9810933498889896

  Divya Dutta: ‘I once lost a role because I was told I am fair’

She wants to do your role.Earlier, we used to say chhota role nahi karenge.I never said that, but I have seen girls say it.Now they say chhota hai, koi baat nahi, impactful hai na?Divya herself had to colour her skin for Delhi 6.They were looking for a village woman who is darker.They told me you suit the role completely.I asked then why am I not doing it?They replied you are too fair for the role.But I am an actor.I was darkened for Delhi 6.
print(inshorts)Actress Divya Dutta has revealed she once lost a role because she was told she was "too fair". "They were looking for a village woman who's darker. They told me 'you suit the role completely'," she added, stating her complexion was cited as the reason behind rejecting her. Divya further revealed she had to darken her skin for 'Delhi-6'.
f1 = open("Sample data/aghoststoryofchristmas.txt")
book = f1.read()
s2 = Saramsh(book," A christmas ghost story")
s2.summarize()1.593220530668083

   A christmas ghost story

Mind!Of course he did.Oh!But what did Scrooge care!"A merry Christmas, uncle!God save you!"cried a cheerful voice."Bah!"said Scrooge, "Humbug!""Christmas a humbug, uncle!"said Scrooge's nephew."I do," said Scrooge."Merry Christmas!You're poor enough."You're rich enough.""Don't be cross, uncle!"said the nephew.Merry Christmas!Out upon merry Christmas!He should!""Uncle!"pleaded the nephew."Nephew!""Keep it!"repeated Scrooge's nephew."But you don't keep it.""Christmas among the rest."Don't be angry, uncle.Come!Dine with us to-morrow.""But why?"cried Scrooge's nephew."Why?""Why did you get married?"said Scrooge."Because I fell in love.""Because you fell in love!""Good afternoon!""Good afternoon," said Scrooge."Good afternoon," said Scrooge.So A Merry Christmas, uncle!""Good afternoon!"said Scrooge."And A Happy New Year!""Good afternoon!"said Scrooge.I'll retire to Bedlam."Marley?""Are there no prisons?"asked Scrooge."And the Union workhouses?"demanded Scrooge."Are they still in operation?""They are.said Scrooge."Both very busy, sir.""Oh!"Nothing!"Scrooge replied."You wish to be anonymous?"Besides--excuse me--I don't know that."Mine occupies me constantly.Good afternoon, gentlemen!"The cold became intense.Foggier yet, and colder.Piercing, searching, biting cold.May nothing you dismay!"said Scrooge."If quite convenient, sir."The clerk smiled faintly.Marley's face.Sitting-room, bedroom, lumber-room.Lumber-room as usual."Humbug!""It's humbug still!"said Scrooge."I won't believe it."and fell again."How now!""Much!""Who are you?""Ask me who I was.""Who were you then?"said Scrooge, raising his voice."Can you--can you sit down?""I can.""Do it, then.""I don't," said Scrooge."I don't know," said Scrooge."You see this toothpick?""I do," replied the Ghost."Well!"Humbug, I tell you!humbug!""Mercy!"he said."Man of the worldly mind!""I do," said Scrooge."I must."Tell me why?"Scrooge trembled more and more.It is a ponderous chain!""Jacob," he said, imploringly.Speak comfort to me, Jacob!""Slow!"the Ghost repeated."Seven years dead," mused Scrooge."And travelling all the time!""No rest, no peace.Incessant torture of remorse.""You travel fast?"said Scrooge."Oh!Yet such was I!Oh!such was I!""Business!""Mankind was my business."Hear me!"cried the Ghost."I will," said Scrooge.Don't be flowery, Jacob!Pray!""Thank'ee!""It is."hinted Scrooge.Scrooge stopped.He looked out.He tried to say "Humbug!"Twelve!The clock was wrong.Twelve!Marley's Ghost bothered him exceedingly."Ding, dong!""Ding, dong!""Half-past!"said Scrooge."Ding, dong!""Ding, dong!"asked Scrooge."I am!"The voice was soft and gentle."Who, and what are you?"Scrooge demanded."Long Past?""No.Your past.""What!""Your welfare!"said the Ghost.Take heed!""Rise!and walk with me!"The city had entirely vanished."Good Heaven!"I was a boy here!""You recollect the way?"inquired the Spirit."Remember it!"observed the Ghost."Let us go on."Out upon merry Christmas!Scrooge said he knew it.And he sobbed."Why, it's Ali Baba!"Scrooge exclaimed in ecstasy.Yes, yes, I know!Poor boy!Serve him right.I'm glad of it."There's the Parrot!"cried Scrooge.Halloa!Hoop!Halloo!"and cried again."What is the matter?"asked the Spirit."Nothing," said Scrooge."Nothing."Home, little Fan?"returned the boy."Yes!""Home, for good and all.Home, for ever and ever.exclaimed the boy."So she had," cried Scrooge."You're right.God forbid!""One child," Scrooge returned."True," said the Ghost."Your nephew!""Know it!"said Scrooge."Was I apprenticed here!"They went in.Ebenezer!Dick!""Dick Wilkins, to be sure!"said Scrooge to the Ghost."Bless me, yes.There he is.Poor Dick!Dear, dear!""Yo ho, my boys!"said Fezziwig."No more work to-night.Christmas Eve, Dick.Christmas, Ebenezer!"Hilli-ho!"Hilli-ho, Dick!Chirrup, Ebenezer!"Clear away!"Small!"echoed Scrooge.Is it not?"It isn't that, Spirit."What is the matter?"asked the Ghost."Nothing particular," said Scrooge."Something, I think?"the Ghost insisted."No," said Scrooge, "No.That's all.""Quick!"For again Scrooge saw himself."To you, very little."What Idol has displaced you?"he rejoined."A golden one."he said.Have I not?""What then?"he retorted.She shook her head."Am I?"You are changed."I am."Have I ever sought release?""In words.No.Never.""In what, then?"Ah, no!""Spirit!"Conduct me home."One shadow more!"exclaimed the Ghost."No more!"cried Scrooge."No more.Show me no more!"to save my life.They are all indescribable alike."Who was it?""Guess!""How can I?Tut, don't I know?""Mr.Scrooge.""Mr. Scrooge it was."Spirit!""Remove me!"Scrooge exclaimed, "I cannot bear it!""Leave me!Take me back.Haunt me no longer!"He obeyed.It was his own room."Come in!"exclaimed the Ghost."Come in!and know me better, man!""Look upon me!"Scrooge reverently did so.exclaimed the Spirit.pursued the Phantom.muttered Scrooge."Touch my robe!"The Grocers'!oh, the Grocers'!And so it was!asked Scrooge."There is.My own."asked Scrooge."To any kindly given.To a poor one most.""Why to a poor one most?"asked Scrooge."Because it needs it most.""I!"cried the Spirit."Wouldn't you?""I!"cried the Spirit.said Scrooge."I seek!"exclaimed the Spirit.Think of that!said Mrs. Cratchit."And your brother, Tiny Tim!"Here's Martha, mother!""Here's Martha, mother!""Hurrah!There's such a goose, Martha!""Well!"No, no!"Hide, Martha, hide!""Why, where's our Martha?"cried Bob Cratchit, looking round."Not coming," said Mrs. Cratchit."Not coming!"Hallo!A great deal of steam!A smell like a washing-day!That was the cloth.That was the pudding!Oh, a wonderful pudding!God bless us!"Which all the family re-echoed."No, no," said Scrooge."Oh, no, kind Spirit!say he will be spared."What then?Oh God!"Mr.Scrooge!"cried Mrs. Cratchit, reddening.Christmas Day."You know he is, Robert!Long life to him!"What place is this?"asked Scrooge."But they know me.See!"Not to sea?To sea."Ha, ha!"laughed Scrooge's nephew."Ha, ha, ha!""Ha, ha!Ha, ha, ha, ha!"cried Scrooge's nephew."He believed it too!""More shame for him, Fred!"said Scrooge's niece, indignantly.They are always in earnest.Oh, perfectly satisfactory."What of that, my dear!"said Scrooge's nephew."Oh, I have!"said Scrooge's nephew.Himself, always.What's the consequence?"Well!What do you say, Topper?"Stop!Of course there was.I know what it is!""What is it?"cried Fred."It's your Uncle Scro-o-o-o-oge!"Which it certainly was."Well!Uncle Scrooge!"they cried.said Scrooge's nephew.Uncle Scrooge!""Are spirits' lives so short?"asked Scrooge."It ends to-night.""To-night!"cried Scrooge."To-night at midnight.Hark!The time is drawing near.""Look here.""Oh, Man!look here.Look, look, down here!"exclaimed the Ghost.Scrooge started back, appalled."Spirit!are they yours?"Scrooge could say no more.This boy is Ignorance.This girl is Want.Deny it!"And bide the end!"cried Scrooge."Are there no prisons?""Are there no workhouses?"The bell struck twelve.said Scrooge."Is that so, Spirit?""Ghost of the Future!"Will you not speak to me?"It gave him no reply."Lead on!"said Scrooge."Lead on!Lead on, Spirit!"I only know he's dead.""When did he die?"inquired another."Last night, I believe.""I thought he'd never die."That's all I know."Another laugh.Bye, bye!"He knew these men, also, perfectly."How are you?"said one."How are you?"returned the other."Well!"said the first."Cold, isn't it?""Seasonable for Christmas time."No.No.Something else to think of.Good morning!"Not another word.cried she who had entered first."Come into the parlour.Ah!How it skreeks!Ha, ha!Come into the parlour.Come into the parlour.""What odds then!What odds, Mrs.Dilber?"said the woman.He always did.""That's true, indeed!"said the laundress."No man more so.""No, indeed!""We should hope not.""Very well, then!"cried the woman."That's enough."It's a judgment on him."Speak out plain.
data ='''Jaipur: The National Institute of Virology (NIV), Pune, on Tuesday confirmed that the 69-year-old Italian tourist, admitted in city’s SMS hospital, is suffering from Covid-19 (the disease caused by the Coronavirus). The tourist’s wife has also tested positive in initial tests and her sample has been sent to NIV, Pune, for confirmation, said Rajasthan chief secretary DB Gupta.

The Italian couple was part of a 21-member group, which landed in Delhi on February 21 and visited six districts of the state till February 28. The rest of the group is in Agra.

Government officials said the group reached Mandawa in Rajasthan’s Jhunjhunu district on Februaruy 21 in a bus arranged by a local travel agent. The group stayed in a local hotel in Mandawa, they added.

According to them, the group went to Bikaner on February 22, and left for Jaisalmer, the next day. They were in Jaisalmer on February 23 and 24 and before leaving for Jodhpur on February 25. From Jodhpur, they went to Udaipur on Feb 26, where they stayed for two days.

On February, 28, the group came to Jaipur, and the 69-year-old man complained of uneasiness and was taken to a local private hospital.

Gupta said the Italian tourist complained of breathlessness, after which he was taken to Fortis Hospital and kept overnight. On February 29 around noon, he was shifted to the isolation ward of Ithe nfectious Diseases Hospital (IDH) of SMS Medical College where he was screened for novel coronavirus and H1N1.

“While in isolation ward, the screening test result of COVID-19 was received at 11pm which was negative. The patient was shifted to H1N1 ICU for observation,” said SMS Medical College principal Dr Sudhir Bhandari.

On Monday, the result of the second screening test came out positive after which the Italian was shifted back to the isolation ward’s ICU and his sample was sent to Pune for confirmation. On Tuesday evening, the union health ministry confirmed that the Italian tourist had Covid-19.

“The patient is in isolation and is stable,” the health ministry said in a statement.

The tourist’s wife was tested on Monday, after the husband’s positive test, and the initial report came out positive.

The chief secretary said all medical colleges in the state have been asked to set up rapid response teams to test and monitor anyone who came in contact with the Italian tourists.

“The rapid response teams will screen people in Mandawa, Bikaner, Jaisalmer, Jodhpur, Udaipur and Jaipur and sanitise hotels and other places the tourists visited,” he said.

Rohit Kumar Singh, additional chief secretary, health department, said the department would send samples of 52 people, who came in contact with the tourist, in Jaipur, to NIV Pune. The 52 include 37 people from SMS Hospital, 11 from Fortis Hospital and four from the hotel where the tourist stayed, he added.

“All of them have been kept in quarantine.”

The rest of the Italians were tested in Agra according to Agra district health officer S K Verma who said the tests came negative. They reached Delhi on March 1, and have now been taken to the Indo Tibetan Border Police’s quarantine facility in Chawala, Delhi. They are undergoing a second round of testing.

Chief Secretary Gupta said five international flights operate out of Rajasthan airports.

“We have been screening people at our airports since January 28 and have screened 24,458 people on 165 flights since then,” he said.'''_____no_output_____sm2 = Saramsh(data,"DBMS")
sm2.summarize()0.1646238965048969

  DBMS

The rest of the group is in Agra.The group stayed in a local hotel in Mandawa, they added.They were in Jaisalmer on February 23 and 24 and before leaving for Jodhpur on February 25.From Jodhpur, they went to Udaipur on Feb 26, where they stayed for two days.On Tuesday evening, the union health ministry confirmed that the Italian tourist had Covid-19.The patient is in isolation and is stable, the health ministry said in a statement.All of them have been kept in quarantine.
sm2.idf______no_output_____sm2.tf______no_output_____sm2.sentenceScores______no_output_____
</code>


# Notebook from artanderson/experiments
Path: notebooks/java-working_with_lists.ipynb

Table of Contents

1  Aerospike Java Client – Reading and Updating Lists

1.1  Notebook Setup

1.1.1  Import Jupyter Java Integration

1.1.2  Start Aerospike

1.1.3  Download the Aerospike Java Client

1.1.4  Start the Aerospike Java Client and Connect

1.2  CREATE Lists in the Database and READ List Records

1.3  READING Lists and List Metrics From the Server

1.4  UPDATE Lists on the Server

1.5  Sorting the Lists

1.6  DELETING the Records and Closing Server Connection

1.7  Aerospike Does Lists

1.8  What's Next?

1.8.1  Next Steps

1.8.2  Additional Resources

_____no_output_____# Aerospike Java Client – Reading and Updating Lists

This notebook demonstrates Java Aerospike CRUD operations (Create, Read, Update, Delete) for lists of data, focusing on server-side **read** and **update** operations, including **sort**.

This [Jupyter Notebook](https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/examples_index.html) requires the Aerospike database running locally with Java kernel and Aerospike Java Client. To create a Docker container that satisfies the requirements and holds a copy of these notebooks, visit the [Aerospike Notebooks Repo](https://github.com/aerospike-examples/interactive-notebooks)._____no_output_____## Notebook Setup

Run these first to initialize Jupyter, download the Java Client, and make sure the Aerospike Database is running._____no_output_____### Import Jupyter Java Integration

Make it easier to work with Java in Jupyter._____no_output_____

import io.github.spencerpark.ijava.IJava;
import io.github.spencerpark.jupyter.kernel.magic.common.Shell;

IJava.getKernelInstance().getMagics().registerMagics(Shell.class);_____no_output_____

### Start Aerospike

Ensure Aerospike database is running locally._____no_output_____

%sh asd_____no_output_____

### Download the Aerospike Java Client

Ask Maven to download and install the project object model (POM) of the Aerospike Java Client._____no_output_____

%%loadFromPOM
<dependencies>
  <dependency>
    <groupId>com.aerospike</groupId>
    <artifactId>aerospike-client</artifactId>
    <version>5.0.0</version>
  </dependency>
</dependencies>_____no_output_____

### Start the Aerospike Java Client and Connect

Create an instance of the Aerospike Java Client, and connect to the demo cluster.

The default cluster location for the Docker container is *localhost* port *3000*. If your cluster is not running on your local machine, modify *localhost* and *3000* to the values for your Aerospike cluster._____no_output_____

import com.aerospike.client.AerospikeClient;

AerospikeClient client = new AerospikeClient("localhost", 3000);
System.out.println("Initialized the client and connected to the cluster.");Initialized the client and connected to the cluster.

## CREATE Lists in the Database and READ List Records

1. Create 10 **Keys** containing lists numbered 1 - 10.
2. Create a String list and an Integer list.
3. Upload the lists into **Bins** named *liststrbin*  and *listintbin*. Place them in the *test* namespace in the *listset1* set.
4. Print the data.

A **Namespace** is like a database in Aerospike. A **Set** is like a database table in Aerospike. A **Key** is a record in the Aerospike database. A **Bin** is a field in the database record._____no_output_____

import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

import com.aerospike.client.AerospikeClient;
import com.aerospike.client.policy.WritePolicy;
import com.aerospike.client.policy.ClientPolicy;
import com.aerospike.client.Bin;
import com.aerospike.client.Key;

String listNamespace = "test";
String listSet = "listset1";
String listStrBin = "liststrbin";
String listIntBin = "listintbin";

int numRecords = 10;

String names[] = {"Annette", "Bharat", "Chenguang", "Darrell", "Eva",
                "Francois", "Geri", "Helen", "Ian", "Javier"};

String fruits[] = {"Pineapple", "Nectarine", "Avocado", "Pear", "Mangosteen",
                "Date", "Banana", "Orange", "Tomato", "Durian"};

String colors[] = {"Aquamarine", "Mauve", "Snowflake", "Salmon", "Khaki",
                "Black", "Darkslategray", "Royalblue", "White", "Rainbow"};

ClientPolicy clientPolicy = new ClientPolicy();
for (int keyNum = 0; keyNum < numRecords; keyNum++) {

    ArrayList<String> listStr = new ArrayList<String>();
    listStr.add(names[keyNum]);
    listStr.add(fruits[keyNum]);
    listStr.add(colors[keyNum]);

    ArrayList<Integer> listInt = new ArrayList<Integer>();
    listInt.add(keyNum);
    listInt.add(numRecords-keyNum);
    listInt.add(1+keyNum);
    listInt.add(2*keyNum);
    listInt.add(keyNum*keyNum);

    Key key = new Key(listNamespace, listSet, keyNum);
    Bin bin1 = new Bin(listStrBin, listStr);
    Bin bin2 = new Bin(listIntBin, listInt);
    client.put(clientPolicy.writePolicyDefault, key, bin1, bin2);

    System.out.println("Key: " + keyNum + ", listStr: " + listStr + ", listInt: " + listInt );
}Key: 0, listStr: [Annette, Pineapple, Aquamarine], listInt: [0, 10, 1, 0, 0]
Key: 1, listStr: [Bharat, Nectarine, Mauve], listInt: [1, 9, 2, 2, 1]
Key: 2, listStr: [Chenguang, Avocado, Snowflake], listInt: [2, 8, 3, 4, 4]
Key: 3, listStr: [Darrell, Pear, Salmon], listInt: [3, 7, 4, 6, 9]
Key: 4, listStr: [Eva, Mangosteen, Khaki], listInt: [4, 6, 5, 8, 16]
Key: 5, listStr: [Francois, Date, Black], listInt: [5, 5, 6, 10, 25]
Key: 6, listStr: [Geri, Banana, Darkslategray], listInt: [6, 4, 7, 12, 36]
Key: 7, listStr: [Helen, Orange, Royalblue], listInt: [7, 3, 8, 14, 49]
Key: 8, listStr: [Ian, Tomato, White], listInt: [8, 2, 9, 16, 64]
Key: 9, listStr: [Javier, Durian, Rainbow], listInt: [9, 1, 10, 18, 81]

## READING Lists and List Metrics From the Server

Now that the lists are in Aerospike, the client can return full or partial lists.
1. Pull each record from Aerospike.
   * For string lists:
      * Pull the last element from the list.
      * Pull the highest rank item.
   * For the integer lists:
      * Pull numbers between 4 and 20.
      * Pull the 2nd and 3rd ranked items.
4. Print the data.

Note that none of the records were modified by these ops._____no_output_____

import com.aerospike.client.AerospikeClient;
import com.aerospike.client.policy.WritePolicy;
import com.aerospike.client.policy.ClientPolicy;
import com.aerospike.client.Bin;
import com.aerospike.client.Key;
import com.aerospike.client.Operation;
import com.aerospike.client.Record;
import com.aerospike.client.Value;
import com.aerospike.client.cdt.CTX;
import com.aerospike.client.cdt.ListOperation;

int last = -1;
int highestRank = 2;
int lowerBound = 4;
int upperBound = 21;
int secondRank = 1;  // Rank is 0th based, so 1 is the 2nd rank item.
int rangeRankSize = 2;
int returnTheValues = 7;

for (int keyNum = 0; keyNum < numRecords; keyNum++) {

    Key key = new Key(listNamespace, listSet, keyNum);
    Record record = client.get(null, key);
    Record lastString = client.operate(client.writePolicyDefault, key,
        ListOperation.get(listStrBin, last)
        );
    Record highestRankString = client.operate(client.writePolicyDefault, key,
        ListOperation.getByRank(listStrBin, highestRank, returnTheValues)
        );
    Record between4And20 = client.operate(client.writePolicyDefault, key,
        ListOperation.getByValueRange(listIntBin, Value.get(lowerBound), Value.get(upperBound), returnTheValues)
        );
    Record rank1And2 = client.operate(client.writePolicyDefault, key,
        ListOperation.getByRankRange(listIntBin, secondRank, rangeRankSize, returnTheValues)
        );

    System.out.println("Full record: " + record);
    System.out.println("The last string: " + lastString.getValue(listStrBin));
    System.out.println("The highest rank string: " + highestRankString.getValue(listStrBin));
    System.out.println("The integers between 4 and 20: " + between4And20.getValue(listIntBin));
    System.out.println("The 2nd and 3rd ranked integers: " + rank1And2.getValue(listIntBin));
    System.out.println();
}Full record: (gen:27),(exp:350713513),(bins:(liststrbin:[Annette, Pineapple, Aquamarine]),(listintbin:[0, 10, 1, 0, 0]))
The last string: Aquamarine
The highest rank string: Pineapple
The integers between 4 and 20: [10]
The 2nd and 3rd ranked integers: [0, 0]

Full record: (gen:27),(exp:350713513),(bins:(liststrbin:[Bharat, Nectarine, Mauve]),(listintbin:[1, 9, 2, 2, 1]))
The last string: Mauve
The highest rank string: Nectarine
The integers between 4 and 20: [9]
The 2nd and 3rd ranked integers: [2, 1]

Full record: (gen:27),(exp:350713513),(bins:(liststrbin:[Chenguang, Avocado, Snowflake]),(listintbin:[2, 8, 3, 4, 4]))
The last string: Snowflake
The highest rank string: Snowflake
The integers between 4 and 20: [8, 4, 4]
The 2nd and 3rd ranked integers: [4, 3]

Full record: (gen:27),(exp:350713513),(bins:(liststrbin:[Darrell, Pear, Salmon]),(listintbin:[3, 7, 4, 6, 9]))
The last string: Salmon
The highest rank string: Salmon
The integers between 4 and 20: [7, 4, 6, 9]
The 2nd and 3rd ranked integers: [6, 4]

Full record: (gen:27),(exp:350713513),(bins:(liststrbin:[Eva, Mangosteen, Khaki]),(listintbin:[4, 6, 5, 8, 16]))
The last string: Khaki
The highest rank string: Mangosteen
The integers between 4 and 20: [4, 6, 5, 8, 16]
The 2nd and 3rd ranked integers: [6, 5]

Full record: (gen:27),(exp:350713513),(bins:(liststrbin:[Francois, Date, Black]),(listintbin:[5, 5, 6, 10, 25]))
The last string: Black
The highest rank string: Francois
The integers between 4 and 20: [5, 5, 6, 10]
The 2nd and 3rd ranked integers: [6, 5]

Full record: (gen:27),(exp:350713513),(bins:(liststrbin:[Geri, Banana, Darkslategray]),(listintbin:[6, 4, 7, 12, 36]))
The last string: Darkslategray
The highest rank string: Geri
The integers between 4 and 20: [6, 4, 7, 12]
The 2nd and 3rd ranked integers: [7, 6]

Full record: (gen:27),(exp:350713513),(bins:(liststrbin:[Helen, Orange, Royalblue]),(listintbin:[7, 3, 8, 14, 49]))
The last string: Royalblue
The highest rank string: Royalblue
The integers between 4 and 20: [7, 8, 14]
The 2nd and 3rd ranked integers: [8, 7]

Full record: (gen:27),(exp:350713513),(bins:(liststrbin:[Ian, Tomato, White]),(listintbin:[8, 2, 9, 16, 64]))
The last string: White
The highest rank string: White
The integers between 4 and 20: [8, 9, 16]
The 2nd and 3rd ranked integers: [9, 8]

Full record: (gen:27),(exp:350713513),(bins:(liststrbin:[Javier, Durian, Rainbow]),(listintbin:[9, 1, 10, 18, 81]))
The last string: Rainbow
The highest rank string: Rainbow
The integers between 4 and 20: [9, 10, 18]
The 2nd and 3rd ranked integers: [10, 9]

## UPDATE Lists on the Server

Aerospike's [list operations](https://www.aerospike.com/apidocs/java/com/aerospike/client/cdt/ListOperation.html) modify data in the Aerospike database. Here are some examples:

1. Print the record.
2. Modify the string list:
   * Insert a Fish string into the second-from-last position.
   * Remove the Name.
3. Modify the integer list:
   * Append a power of 5 to the end the list.
   * Increment the 4th integer in each list by 11.
4. Print the final record._____no_output_____

import com.aerospike.client.AerospikeClient;
import com.aerospike.client.policy.WritePolicy;
import com.aerospike.client.policy.ClientPolicy;
import com.aerospike.client.Bin;
import com.aerospike.client.Key;
import com.aerospike.client.Operation;
import com.aerospike.client.Record;
import com.aerospike.client.Value;
import com.aerospike.client.cdt.CTX;
import com.aerospike.client.cdt.ListOperation;

String Fishes[] = {"Koi", "Tuna", "Stingray", "Arowana", "Mackerel",
                "Needlefish", "Sardine", "Angelfish", "Cod", "Mako"};
int fishIndex = -1;
int nameIndex = 0;
int newMult = 5;
int incNum = 11;
int incIndex = 3;

for (int keyNum = 0; keyNum < numRecords; keyNum++) {

    Key key = new Key(listNamespace, listSet, keyNum);
    Record origRecord = client.get(null, key);
    System.out.println("Key:" + keyNum + " Before – " + " listStr: " + origRecord.getValue(listStrBin) + ", listInt: " + origRecord.getValue(listIntBin) );
    origRecord = client.operate(client.writePolicyDefault, key,
        ListOperation.insert(listStrBin, fishIndex, Value.get(Fishes[keyNum])),
        ListOperation.remove(listStrBin, nameIndex),
        ListOperation.append(listIntBin, Value.get(incNum*keyNum)),
        ListOperation.increment(listIntBin, incIndex, Value.get(incNum))
        );
    Record finalRecord = client.get(null, key);
    System.out.println("      After  – " + " listStr: " + finalRecord.getValue(listStrBin) + ", listInt: " + finalRecord.getValue(listIntBin) );
}Key:0 Before –  listStr: [Annette, Pineapple, Aquamarine], listInt: [0, 10, 1, 0, 0]
      After  –  listStr: [Pineapple, Koi, Aquamarine], listInt: [0, 10, 1, 11, 0, 0]
Key:1 Before –  listStr: [Bharat, Nectarine, Mauve], listInt: [1, 9, 2, 2, 1]
      After  –  listStr: [Nectarine, Tuna, Mauve], listInt: [1, 9, 2, 13, 1, 11]
Key:2 Before –  listStr: [Chenguang, Avocado, Snowflake], listInt: [2, 8, 3, 4, 4]
      After  –  listStr: [Avocado, Stingray, Snowflake], listInt: [2, 8, 3, 15, 4, 22]
Key:3 Before –  listStr: [Darrell, Pear, Salmon], listInt: [3, 7, 4, 6, 9]
      After  –  listStr: [Pear, Arowana, Salmon], listInt: [3, 7, 4, 17, 9, 33]
Key:4 Before –  listStr: [Eva, Mangosteen, Khaki], listInt: [4, 6, 5, 8, 16]
      After  –  listStr: [Mangosteen, Mackerel, Khaki], listInt: [4, 6, 5, 19, 16, 44]
Key:5 Before –  listStr: [Francois, Date, Black], listInt: [5, 5, 6, 10, 25]
      After  –  listStr: [Date, Needlefish, Black], listInt: [5, 5, 6, 21, 25, 55]
Key:6 Before –  listStr: [Geri, Banana, Darkslategray], listInt: [6, 4, 7, 12, 36]
      After  –  listStr: [Banana, Sardine, Darkslategray], listInt: [6, 4, 7, 23, 36, 66]
Key:7 Before –  listStr: [Helen, Orange, Royalblue], listInt: [7, 3, 8, 14, 49]
      After  –  listStr: [Orange, Angelfish, Royalblue], listInt: [7, 3, 8, 25, 49, 77]
Key:8 Before –  listStr: [Ian, Tomato, White], listInt: [8, 2, 9, 16, 64]
      After  –  listStr: [Tomato, Cod, White], listInt: [8, 2, 9, 27, 64, 88]
Key:9 Before –  listStr: [Javier, Durian, Rainbow], listInt: [9, 1, 10, 18, 81]
      After  –  listStr: [Durian, Mako, Rainbow], listInt: [9, 1, 10, 29, 81, 99]

## Sorting the Lists

Aerospike also sorts and rermoves duplicates from lists.
1. Print both lists.
2. Sort both lists and remove the duplicate integers.
3. Print the sorted lists._____no_output_____

import com.aerospike.client.AerospikeClient;
import com.aerospike.client.policy.WritePolicy;
import com.aerospike.client.policy.ClientPolicy;
import com.aerospike.client.Bin;
import com.aerospike.client.Key;
import com.aerospike.client.Operation;
import com.aerospike.client.Record;
import com.aerospike.client.Value;
import com.aerospike.client.cdt.CTX;
import com.aerospike.client.cdt.ListOperation;

int dropDuplicates = 2;

for (int keyNum = 0; keyNum < numRecords; keyNum++) {

    Key key = new Key(listNamespace, listSet, keyNum);
    Record origRecord = client.get(null, key);
    System.out.println("Key:" + keyNum + " Unsorted – " + " listStr: " + origRecord.getValue(listStrBin) + ", listInt: " + origRecord.getValue(listIntBin) );
    origRecord = client.operate(client.writePolicyDefault, key,
        ListOperation.sort(listStrBin, dropDuplicates),
        ListOperation.sort(listIntBin, dropDuplicates)
        );
    Record finalRecord = client.get(null, key);
    System.out.println("        Sorted – " + " listStr: " + finalRecord.getValue(listStrBin) + ", listInt: " + finalRecord.getValue(listIntBin) );
}Key:0 Unsorted –  listStr: [Aquamarine, Koi, Pineapple], listInt: [0, 1, 10, 11]
        Sorted –  listStr: [Aquamarine, Koi, Pineapple], listInt: [0, 1, 10, 11]
Key:1 Unsorted –  listStr: [Mauve, Nectarine, Tuna], listInt: [1, 2, 9, 11, 13]
        Sorted –  listStr: [Mauve, Nectarine, Tuna], listInt: [1, 2, 9, 11, 13]
Key:2 Unsorted –  listStr: [Avocado, Snowflake, Stingray], listInt: [2, 3, 4, 8, 15, 22]
        Sorted –  listStr: [Avocado, Snowflake, Stingray], listInt: [2, 3, 4, 8, 15, 22]
Key:3 Unsorted –  listStr: [Arowana, Pear, Salmon], listInt: [3, 4, 7, 9, 17, 33]
        Sorted –  listStr: [Arowana, Pear, Salmon], listInt: [3, 4, 7, 9, 17, 33]
Key:4 Unsorted –  listStr: [Khaki, Mackerel, Mangosteen], listInt: [4, 5, 6, 16, 19, 44]
        Sorted –  listStr: [Khaki, Mackerel, Mangosteen], listInt: [4, 5, 6, 16, 19, 44]
Key:5 Unsorted –  listStr: [Black, Date, Needlefish], listInt: [5, 6, 21, 25, 55]
        Sorted –  listStr: [Black, Date, Needlefish], listInt: [5, 6, 21, 25, 55]
Key:6 Unsorted –  listStr: [Banana, Darkslategray, Sardine], listInt: [4, 6, 7, 23, 36, 66]
        Sorted –  listStr: [Banana, Darkslategray, Sardine], listInt: [4, 6, 7, 23, 36, 66]
Key:7 Unsorted –  listStr: [Angelfish, Orange, Royalblue], listInt: [3, 7, 8, 25, 49, 77]
        Sorted –  listStr: [Angelfish, Orange, Royalblue], listInt: [3, 7, 8, 25, 49, 77]
Key:8 Unsorted –  listStr: [Cod, Tomato, White], listInt: [2, 8, 9, 27, 64, 88]
        Sorted –  listStr: [Cod, Tomato, White], listInt: [2, 8, 9, 27, 64, 88]
Key:9 Unsorted –  listStr: [Durian, Mako, Rainbow], listInt: [1, 9, 10, 29, 81, 99]
        Sorted –  listStr: [Durian, Mako, Rainbow], listInt: [1, 9, 10, 29, 81, 99]

## DELETING the Records and Closing Server Connection

1. Use the **asinfo** administration tool to dump the set containing our list data.
2. Close the client's connection to the Aerospike cluster._____no_output_____

%sh asinfo -v "truncate:namespace=test;set=listset1;"
client.close();
System.out.println("Index dropped and server connection closed.");_____no_output_____

## Aerospike Does Lists

Aerospike database and its Java Client are up to the task of working with your lists._____no_output_____## What's Next?

Want to learn more?

_____no_output_____### Next Steps

Have questions? Don't hesitate to reach out if you have additional questions about working with lists at https://discuss.aerospike.com/.

Want to check out other Java notebooks?
1. [Hello, World](hello_world.ipynb)
2. [Aerospike Query and UDF](query_udf.ipynb)
3. [Simple Put Get Example](SimplePutGetExample.ipynb)
4. [Working with Twitter Data](tweetaspike.ipynb)_____no_output_____### Additional Resources

Here are some links to learn more about this demo:
* [The Aerospike Java Client](https://github.com/aerospike/aerospike-client-java)
* [Aerospike's List Operations](https://www.aerospike.com/apidocs/java/com/aerospike/client/cdt/ListOperation.html)
* [Aerospike Database Architecture](https://www.aerospike.com/docs/architecture/index.html)_____no_output_____


# Notebook from mojones/dfd_jupyter_test
Path: _notebooks/numerical precision.ipynb

# "Does reducing numerical precision affect real world datasets?"
> "Reducing numerical precision is a way to save memory in pandas, but does it make a difference to the conclusions that we might draw from real world datasets?"

- toc: true
- badges: false
- comments: false
- author: Martin
- categories: [pandas, numpy, kaggle]_____no_output_____### Executive summary

Switching from 64 bit to 32 bit precision for floating point data in pandas saves memory and has virtually no effect on the data, as long as the values aren't greater than around 10300

### Slightly longer summary

- I wrote code to download ~1,000 real world datasets from kaggle and look at summary statistics and linear regressions of floating point columns in both 64 and 32 bit precision
- The final analysis involved ~1,000 datasets, ~8,000 data files, ~500,000,000 individual floating point values and ~500,000 linear regressions.
- Changing to 32 bit precision introduced a mean proportional error of around 10-8 in the summary statistics, with the worst affected columns having a proportional error of around 10-6
- For linear regressions, the mean proportional change in slope and intercept was on the order of 10-6
- A small number of outliers with near-zero slope (representing uninteresting relationships) had larger proportional errors
- In ~400,000 valid linear regressions I was unable to find any relationships whose significance at the 5% level was changed by moving from 64 to 32 bit precision
- For real world datasets, switching from 64 to 32 bit floating point precision is overwhelmingly unlikely to alter our conclusions about the data, and is an easy way to reduce memory usage.

## Introduction

As you've probably noticed, this is a longish article, so grab a cup of coffee if you're planning to read it all. If you're already familiar with floating point precision and error in pandas you can probably skip part one. Most of the methods stuff is in part two, so if you just want to see the results, skip to part three. Also, just to head off the obvious criticism, I think that most of the conclusions of this article could have been arrived at by pure reasoning, without looking at any data at all. But hopefully some folks will, like me, be more satisfied by empirical data :-)

Obviously, I think that pandas is wonderful :-) But it has a number of quirks, among them a reputation for being memory hungry. There are no shortage of articles online discussing how to fit large dataframes in memory, and there are a number of well established techniques (there's a whole chapter on the subject in the [Drawing from Data book](https://drawingfromdata.com/book/), and also an extended discussion in the car accidents dataset video).

One technique that always gets mentioned is storing numbers in a reduced precision data type. This is usually proposed as one of the last things to try after other options have been exhausted. I think there are a couple of reasons for this. One is that messing about with data types is a bit harder to explain than simple things like ignoring unneeded columns. Another is that deliberately throwing away information - which is what reducing precision will do - feels very uncomfortable from a data analysis point of view!

One thing that I've rarely seen in such discussions is a measurement of how much of an effect reduced precision will have in real life datasets. This is surprising, since pandas makes it very easy to measure. So let's go ahead and try it...

## Part one: changing data types and measuring differences

Before we dive in, a quick reminder of the pandas tools we'll use. All of the methods etc. are described in detail in the [Drawing from Data book](https://drawingfromdata.com/book/), so we'll limit ourselves here to an overview.

As an example, let's load the planets dataset:_____no_output_____

import pandas as pd
pd.options.display.max_columns=4

planets = pd.read_csv('https://raw.githubusercontent.com/mwaskom/seaborn-data/master/planets.csv')
planets_____no_output_____

which stores information about planets that have been discovered around other stars. We are interested in the orbital period column, which we'll store in a variable for convenience:_____no_output_____

op = planets['orbital_period']
op_____no_output_____

which tells us how long each planet takes to revolve around its host star. Because it's comprised of floating point numbers, it get the `float64` data type by default. To change the data type, we can just use the `as_type` method. Let's try `float32`:_____no_output_____

op.astype('float32')_____no_output_____

All we need to know about these two data types is that, as the names suggest, a single `float64` value uses 64 bits of memory, whereas a `float32` uses, you guessed it, 32 bits. So it comes as no great surprise that the memory usage of the `float32` version is very close to half that of the `float64` version:_____no_output_____

op.memory_usage(), op.astype('float32').memory_usage()_____no_output_____

As well as a difference in memory usage, we can see from the output above that there are slight differences in the values themselves. This is perhaps easiest to see if we put the two series next to each other:_____no_output_____

pd.concat([op, op.astype('float32')], axis=1)_____no_output_____

To figure out how different the two series are, pandas makes it easy to just take the difference:_____no_output_____

op.astype('float32') - op_____no_output_____

It's probably more useful to view these as absolute difference, i.e. to ignore the sign:_____no_output_____

(op.astype('float32') - op).abs()_____no_output_____

Now we can look at a summary:_____no_output_____

(op.astype('float32') - op).abs().describe()_____no_output_____

The general picture here is that the change from `float64` to `float32` has made very little difference to the values. On average, it introduces an absolute error of around 4x10-6, or 0.000004, and the single worst-case value has an absolute error of about 0.0002. These errors are very small relative to the range of the original values:_____no_output_____

op.describe()_____no_output_____

### Proportional differences

If we wanted to compare the errors meaningfully across multiple different datasets, it might be useful to look at the error as a proportion of the original values:_____no_output_____

((op.astype('float32') - op).abs() / op).describe()_____no_output_____

This calculation is a bit more involved, but easier to interpret. It is telling us that the switch from `float64` to `float32` changed the values on average by about 10-8 - a truly miniscule chanage.

Another way to look at the difference is to compare the summaries of the two series. For any given aggregation we can figure out the proportional change by taking the absolute difference and dividing by the original:_____no_output_____

import numpy as np

np.abs(op.mean() - op.astype('float32').mean()) / op.mean()_____no_output_____

So changing the series to `float32` alters the mean by a factor of around 10-8. We can do the same calculation with any other summary statistic - for instance, the standard deviation:_____no_output_____

np.abs(op.std() - op.astype('float32').std()) / op.std()_____no_output_____

### Comparing correlations

Another way to measure the effect of reduced precision is to see if it changes the correlation between floating point columns. In our planets dataset we have two other floating point columns: the mass and the distance. Using pandas' built in `corr` method we can calculate the pairwise pearson correlation of all the floating point columns:_____no_output_____

planets.select_dtypes('float64').corr()_____no_output_____

Then do the same with a `float32` version and take the difference:_____no_output_____

(
    planets.select_dtypes('float64').corr()
    - planets.select_dtypes('float64').astype('float32').corr()
)_____no_output_____

As we can see, the differences are tiny, and would definitely not alter our conclusions about the relationships in the dataset. For a more sophisticated look at correlations, we could bring in a linear regression:_____no_output_____

from scipy.stats import linregress

linregress(
    planets.dropna()['orbital_period'],
    planets.dropna()['mass']
)_____no_output_____

and again take the difference between the 64 and 32 bit versions:_____no_output_____

l_64 = linregress(
    planets.dropna()['orbital_period'],
    planets.dropna()['mass']
)

l_32 = linregress(
    planets.dropna()['orbital_period'].astype('float32'),
    planets.dropna()['mass'].astype('float32')
)

print(f'slope difference : {l_64[0] - l_32[0]} ')
print(f'intercept difference : {l_64[1] - l_32[1]} ')
print(f'R value difference : {l_64[2] - l_32[2]} ')slope difference : -1.6647770943907514e-12
intercept difference : 2.81312630967534e-08
R value difference : -1.7317597356125702e-10

Once again, the difference is incredibly minor. Given that the original values might be very small, a proprtional change might be easier to interpret:_____no_output_____

print(f'proportional slope difference : {(l_64[0] - l_32[0]) / l_64[0]} ')
print(f'proportional intercept difference : {(l_64[1] - l_32[1]) / l_64[1]} ')
print(f'proportional R value difference : {(l_64[2] - l_32[2]) / l_64[2]} ')proportional slope difference : -3.6375395536379053e-09
proportional intercept difference : 1.3226957723928434e-08
proportional R value difference : -9.365612763031196e-10

## Part two: let's try it on a larger scale

Attentive readers will have noticed that all of the calculations above were done by Python code :-) That means that it should be possible to automate this process and try it across a large number of different datasets. For a convenient source of real world dataset we turn to [kaggle](http://kaggle.com).

The collection of datasets hosted at Kaggle is perfect for our purposes. It contains real world data from a massive range of different fields and sources, mostly stored in files that will be easy to get in to pandas. It also has a convenient API with a Python client library, which will make it easy to download a large collection of different data files.

### Downloading the data

To download our collection of datasets using the Kaggle API we have to go through a couple of steps. First we will connect to the API and authenticate:

```python
from kaggle.api.kaggle_api_extended import KaggleApi
api = KaggleApi()
api.authenticate()
```

Next we can get a list of datasets by using the `dataset_list` method. We will search for CSV files, since they will be the easiest to load into pandas, and use pagination to get a large collection of dataset objects. Here I'm using the `tqdm` package for a progress bar, since this code takes a long time to run. I'm not sure if it's necessary, but I'm also putting in a 1-second delay between requests in an attempt to be a good internet citizen:

```python
from tqdm.notebook import trange, tqdm
import time

dataset = []

for page in trange(1,1000):
    datasets.append(api.dataset_list(file_type='csv', page=page))
    time.sleep(1)

```

Now we have a list of dataset objects, we can get on to the actual downloading. To keep things organized I'm using a separate folder for each dataset (which might end up containing multiple files). Once I've created the folder it's just a case of calling `dataset_download_files` with each dataset reference from the previous step and the new folder name as the destination. I'm also using `unzip=True` to make sure that we end up with extracted files. This will obviously take more disk space than leaving them compressed.

```python

for dat in tqdm(all_datasets):

    folder_name = dat.ref.replace('/', '-')

    if not os.path.exists(folder_name):
        os.mkdir(folder_name)
    print(dat.ref)
    api.dataset_download_files(str(dat.ref),path=folder_name, unzip=True)
```_____no_output_____### Processing the downloaded files

Now that we've done the download step, we can get on to reading our data files and measuring the effect of reducing precision. Let's take a look at the function to take a single file and just give us the columns we're interested in:_____no_output_____

def get_float_columns(filepath):

    try:
        df = pd.read_csv(filepath, encoding='latin-1').select_dtypes(['float64'])
    except:
        print(f'cannot read {filepath}')
        return None

    # replace infinite values with missing data
    df.replace([np.inf, -np.inf], np.nan, inplace=True)

    # get rid of columns that are only floats because they contain missing data
    # i.e. they should actually be integers
    keep = []
    for c in df.columns:
        if not all(df[c].dropna().astype(int) == df[c].dropna()):
            keep.append(c)
    if len(keep) == 0:
        return None
    return df[keep]
    _____no_output_____

Some of the files will need special arguments to `read_csv` in order to open. Of course, since we have so many files, we can't go and manually figure out the correct arguments for every one. So to open the file we'll just call `read_csv` in a `try` block so that files requiring complex arguments won't crash our program. We are only interested in floating point columns, so we'll use `select_dtypes` to get them.

There's a slight complication in selecting the floating point columns: due to the way that pandas handles missing data, integer columns that contain missing data will end up as `float64` by default. So there's a chunk of code to identify floating point columns that are really integers, and to drop them from the dataset.

We can test out the function by running it on our planets dataset:_____no_output_____

get_float_columns('planets.csv')_____no_output_____

As expected, we just get the floating point columns.

### Comparing summary statistics

Now we can move on to the first question, calculating the summary statistics for each column in both 64 bit and 32 bit versions. Most of the work will be done by `describe`. We will add a `_64` and a `_32` suffix to the column names, then concatenate the two descriptions to give a wide summary table with one row per column in the original dataframe:_____no_output_____

import numpy as np
def summarize_columns(df):
    summary_64 = df.describe().T
    summary_64.columns = [c + '_64' for c in summary_64.columns]

    summary_64_abs = df.abs().describe().T
    summary_64_abs.columns = [c + '_64_abs' for c in summary_64_abs.columns]

    summary_32 = df.astype('float32').describe().T
    summary_32.columns = [c + '_32' for c in summary_32.columns]

    result = pd.concat([summary_32, summary_64, summary_64_abs], axis=1)

    return result

planets = get_float_columns('planets.csv')
summarize_columns(planets)_____no_output_____

There's one slight complication compared to our original planet code. Think about the case where we have a column of mixed negative and positive values. We might end up with a situation where the mean is very close to zero, even though the values themselves are not. This will give us an inflated estimate of the precision error when we divide by the mean. By working with a mean of the absolute values we can avoid this. The same goes for the other summary statistics - median, min, max, etc.

This dataframe is a bit awkward to read on a web page, since it's so wide. But the column structure makes it very easy to check, for example, the difference in the means, either in absolute terms:_____no_output_____

summary = summarize_columns(planets)
(summary['mean_64'] - summary['mean_32']).abs()_____no_output_____

Or as a proportion of the original:_____no_output_____

summary = summarize_columns(planets)
(summary['mean_64'] - summary['mean_32']).abs() / summary['mean_64_abs']_____no_output_____

### Comparing linear regressions

Next we need a function that will take our dataframe of floating point columns and calculate pairwise linear regressions for both 32 and 64 bit versions. Here's what it looks like:_____no_output_____

import scipy.stats
import itertools

def summarize_regressions(df):

    df = df.dropna()

    # if fewer than two columns or fewer than ten rows, skip
    if len(df.columns) < 2 or len(df) < 10:
        return None

    df_32 = df.astype('float32')

    results = []

    for x, y in itertools.combinations(df.columns, r=2):

        regression_64 = scipy.stats.linregress(df[x], df[y])
        regression_32 = scipy.stats.linregress(df_32[x], df_32[y])

        results.append(
            (
                x,
                y,
                regression_64[0], # slope
                regression_32[0],
                regression_64[1], # intercept
                regression_32[1],
                regression_64[2], # R value
                regression_32[2],
                regression_64[3], # P value
                regression_32[3],
            )
        )

    return pd.DataFrame(
        results,
        columns = ['x', 'y',
               'slope_64', 'slope_32',
               'intercept_64', 'intercept_32',
               'rvalue_64', 'rvalue_32',
               'pvalue_64', 'pvalue_32']
        )

summarize_regressions(planets)_____no_output_____

The output is another summary table, this time showing us the slope, intercept, rvalue and pvalue for each pair of columns in both 64 bit and 32 bit form. As in the summary statistics, this makes it easy to find the differences in the various regression properties. Here's the difference in slope as a proportion of the original slope:_____no_output_____

summary = summarize_regressions(planets)
(summary['slope_64'] - summary['slope_32']).abs() / summary['slope_64']_____no_output_____

## Part three: looking at the results

Having written the functions above, actually analysing all of the downloaded data files is pretty straightforward. We will use `glob` to iterate over the folders, and just call the functions in order. _____no_output_____

import glob
from tqdm.notebook import tqdm

dfs = []

for filepath in tqdm(glob.glob('/home/martin/Downloads/kaggle_test/*-*/*.csv')):
    df = get_float_columns(filepath)
    if df is not None:
        summary = summarize_columns(df)
        summary['filepath'] = filepath
        dfs.append(summary)

summary_stats = pd.concat(dfs)_____no_output_____#hide

summary_stats.rename_axis('column_name').reset_index().to_csv('summary_stats.csv', index=False)_____no_output_____

At the end of the process we have a big dataframe that contains the summary for all the data files:_____no_output_____

summary_stats.head()_____no_output_____

If you want to play with the data, you can [download a compressed copy here](https://github.com/mojones/dfd_jupyter_test/raw/master/_notebooks/summary_stats.csv.gz).

Let's first take a look at the volume of data. We have processed 919 data files:_____no_output_____

summary_stats['filepath'].nunique()_____no_output_____

containing just over 8000 floating point columns:_____no_output_____

len(summary_stats.reset_index().groupby(['index', 'filepath']))_____no_output_____

and just over half a billion individual floating point values:
_____no_output_____

summary_stats['count_64'].sum()_____no_output_____

One slight complication is that some of our data files contain very large values that will be outside the range of 32 bit floating point numbers. It's easy to spot those in the file as the maximum of the 32 bit version will end up infinite:_____no_output_____

summary_stats[np.isinf(summary_stats['max_32'])]_____no_output_____

It's also pretty easy to remove them from the results:_____no_output_____

results = summary_stats[ ~ np.isinf(summary_stats['max_32'])]_____no_output_____

### Looking at changes in summary statistics

Now we can start to address the questions of how much effect changing to 32 bit precision had on the data. Since the `results` dataframe is in the same format as our earlier planets example, the code will be very similar. We can calculate the absolute error:_____no_output_____

(results['mean_64'] - results['mean_32']).abs()_____no_output_____

but since we have so many values, we need to summarize them. We can do this with summary statistics:_____no_output_____

(results['mean_64'] - results['mean_32']).abs().describe()_____no_output_____

Notice that we have a pretty wild distribution here. The mean absolute error is about three million, but the median (50% in the above table) is 0.0000004! This is why we need to normalize these by viewing them as a proportion of the original mean of absolute values:_____no_output_____

prop_errors = (results['mean_64'] - results['mean_32']).abs() / results['mean_64_abs']
prop_errors.describe()_____no_output_____

This is easier to look at. The mean and median proportional error are both on the order of 10-8, and the worst affected data column has its mean altered by around 10-6. In practical terms, these are incredibly minor effects. We can easily view them as a distribution to see the long tail:_____no_output_____

import seaborn as sns

sns.displot(
    prop_errors,
    aspect=2
)_____no_output_____

We can do the same thing for the other aggregations of the original data - let's add the median (50% in our table) and the max. Calculating the proportional error in the minimum is tricky, since for many data columns the minimum value will be zero, leading to an infinite proportional error, so we will skip it.

Adding these measurements as additional columns:_____no_output_____

results['mean prop error'] = (results['mean_64'] - results['mean_32']).abs() / results['mean_64_abs']
results['median prop error'] = (results['50%_64'] - results['50%_32']).abs() / results['50%_64_abs']
results['max prop error'] = (results['max_64'] - results['max_32']).abs() / results['max_64_abs']
_____no_output_____

will allow us to make a summary table:_____no_output_____

results[['mean prop error', 'median prop error', 'max prop error']].describe()_____no_output_____

A little bit of rearrangement using `melt` will allow us to look at the distribution of proportional errors for all three summary statistics:_____no_output_____

g = sns.displot(
    data = results[['mean prop error', 'median prop error', 'max prop error']].melt(),
    x = 'value',
    col = 'variable',
)

import matplotlib.pyplot as plt
plt.xlim((0, 2e-7))_____no_output_____

### Looking at linear regressions

Now we can build a similar dataframe for the linear regression results. The logic is very similar to the summary statistics; we are just using the `summarize_regressions` function instead:_____no_output_____

import glob
from tqdm.notebook import tqdm

dfs = []

for filepath in tqdm(glob.glob('/home/martin/Downloads/kaggle_test/*-*/*.csv')):
    df = get_float_columns(filepath)
    if df is not None:
        summary = summarize_regressions(df)
        if summary is not None:
            summary['filepath'] = filepath
            dfs.append(summary)

regressions = pd.concat(dfs)_____no_output_____#hide

regressions.to_csv('regressions.csv', index=False)_____no_output_____

If you want to play with the data, you can [download a compressed version here](https://github.com/mojones/dfd_jupyter_test/raw/master/_notebooks/regressions.csv.gz).

The resulting dataframe has the same wide shape as before, but this time each row represents a regression result between two columns from the same data file. We have just over half a million regression results in total:_____no_output_____

regressions_____no_output_____

Because the number of pairwise comparisons increases with the square of the number of columns, the results are dominated by a small number of data files with many floating point columns:_____no_output_____

regressions['filepath'].value_counts()_____no_output_____

Out of our 919 data files, only 289 had at least two floating point columns. The data file with the most floating point columns contributes nearly two hundred thousand regression results!

To get an overview of the difference for each property of the regression result, we can add new columns as we did before and summarize them:_____no_output_____

regressions['slope error'] = ((regressions['slope_64'] - regressions['slope_32']) / regressions['slope_64']).abs()
regressions['intercept error'] = ((regressions['intercept_64'] - regressions['intercept_32']) / regressions['intercept_64']).abs()
regressions['rvalue error'] = ((regressions['rvalue_64'] - regressions['rvalue_32']) / regressions['rvalue_64']).abs()
regressions['pvalue error'] = ((regressions['pvalue_64'] - regressions['pvalue_32']) / regressions['pvalue_64']).abs()

regressions[['slope error', 'intercept error', 'rvalue error', 'pvalue error']].dropna().describe()_____no_output_____

Notice that in the `count` row we can see how many of the comparisons gave meaningful (i.e. non-missing) output for the linear regressions on both the 64 and 32 bit versions.

Remember that these are proportional errors i.e. the proportion by which the values change. Although the mean errors are a couple of orders of magnitude greater than for the summary statistics, they are still very small, and very unlikely to change any conclusions about the data.

We have a very long tailed distribution for all four properties: the mean is several orders of magnitude higher than the median (50%) and the maximum values are several orders of magnitude higher again, suggesting a small number of extreme outliers. One way to show this is with a boxen plot:_____no_output_____

sns.catplot(
    data = regressions[['slope error', 'intercept error', 'rvalue error', 'pvalue error']].dropna().melt(),
    x = 'variable',
    y = 'value',
    kind='boxen',
    aspect = 2
)_____no_output_____

The presence of the outliers make it impossible to see the distribution of the remaining values. If we plot the slope against the slope error, we can see that these outliers all represent regressions where the original slope was very near zero, suggesting no interesting relationship between the variables:_____no_output_____

sns.relplot(
    data = regressions,
    x = 'slope_64',
    y = 'slope error'
)

plt.xlim(-1e-8,1e-8)_____no_output_____

The same is true for the intercept:_____no_output_____

sns.relplot(
    data = regressions,
    x = 'intercept_64',
    y = 'intercept error'
)

plt.xlim(-1e-5,1e-5)_____no_output_____

An alternative way of looking at the pvalue error is to ask whether the change in pvalue would result in a difference in whether the relationship was considered significant. Setting aside the problems with interpreting pvalues in this way, we can use pandas to find any regressions where one version of the regression was significant at the 5% level and the other wasn't:_____no_output_____

((regressions.dropna()['pvalue_64'] < 0.05) != (regressions.dropna()['pvalue_32'] < 0.05)).value_counts()_____no_output_____

In the ~400,000 out of our ~500,000 comparisions that had no missing data, there were no such cases.

## Conclusion

It looks like in real world datasets, switching from 64 bit to 32 bit precision for floating point numbers has a vanishingly small chance of affecting our conclusions, and we should probably do it without fear when trying to minimize memory usage.

If you've made it this far, you should definitely subscribe to the [Drawing from Data newsletter](https://drawingfromdata.com/newsletter/), [follow me on Twitter](https://twitter.com/DataDrawing), or buy the [Drawing from Data book](https://drawingfromdata.com/book/)!_____no_output_____


# Notebook from knightlab-analyses/ibd-surgery
Path: metabolomics/1.0.0-differential-abundance.ipynb

%matplotlib inline
import pandas as pd, numpy as np, seaborn as sns, matplotlib.pyplot as plt, qiime2 as q2
from biom import load_table, Table
from biom.util import biom_open
from skbio import OrdinationResults
from emperor import Emperor, nbinstall; nbinstall()

def load_mf(fn, index='#SampleID'):
    _df = pd.read_csv(fn, sep='\t', dtype='str', na_values=[], keep_default_na=False)
    _df.set_index(index, inplace=True)
    return _df_____no_output_____mf = load_mf('data/11546_20190819-143944.txt', 'sample_name')
# remove blanks
mf = mf.query('empo_1 == "Host-associated"').copy()_____no_output_____

Filter the feature table to exclude blanks._____no_output_____

table = q2.Artifact.import_data('FeatureTable[Frequency]', 'data/boland_biom_original.biom')
table.save('data/feature-table.qza')_____no_output_____!qiime feature-table filter-samples \
--i-table data/feature-table.qza \
--m-metadata-file data/11546_20190118-122918.txt \
--p-where "empo_1 == 'Host-associated' and time_point == 'baseline'" \
--o-filtered-table data/feature-table.baseline.noblanks.qza Saved FeatureTable[Frequency] to: data/feature-table.baseline.noblanks.qza
!qiime composition add-pseudocount \
--i-table data/feature-table.baseline.noblanks.qza \
--o-composition-table data/feature-table.baseline.noblanks.composition.qzaSaved FeatureTable[Composition] to: data/feature-table.baseline.noblanks.composition.qza
!qiime composition ancom --i-table data/feature-table.baseline.noblanks.composition.qza \
--m-metadata-file data/11546_20190118-122918.txt \
--m-metadata-column has_surgery \
--o-visualization ms.ancom.qzvSaved Visualization to: ms.ancom.qzv

Next step is to download the summary file from ANCOM and using it to look at the _____no_output_____

ancom_results = pd.read_table('ancom.tsv', index_col=0)/Users/yoshikivazquezbaeza/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/ipykernel_launcher.py:1: FutureWarning: read_table is deprecated, use read_csv instead, passing sep='\t'.
  """Entry point for launching an IPython kernel.
ms = q2.Artifact.load('data/feature-table.baseline.noblanks.composition.qza').view(pd.DataFrame)_____no_output_____spectra_metadata = load_table('data/boland_biom_original.biom').metadata_to_dataframe('observation')_____no_output_____spectra_metadata[spectra_metadata.Compound_Name == 'Spectral Match to 1-Amino-2-ethoxybenzene from NIST14']_____no_output_____differential = spectra_metadata.loc[ancom_results[ancom_results["Reject null hypothesis"]].index]_____no_output_____

Transform into CLR_____no_output_____

from skbio.stats.composition import clr

# ms_df = ms.to_dataframe(True)
ms_df = ms.T.apply(lambda row: clr(row), axis=0)
ms_df = ms_df.loc[differential.index].copy()

# ms_df = ms.filter(differential.index, axis='observation', inplace=False).to_dataframe(True)_____no_output_____mf = mf.loc[ms_df.columns].copy()

names = []
for molecule in ms_df.index:
    row = spectra_metadata.loc[molecule]

    if row.Compound_Name == 'nan':
        name = row.name
    else:
        name = row.Compound_Name

    # should be copying the CLR transformed data
    mf[name] = ms_df.loc[molecule].copy()
    names.append(name)

long = pd.melt(mf, value_vars=names, id_vars=['has_surgery'], var_name='Molecule', value_name='Intensity',)_____no_output_____long.Molecule.replace({
 'Spectral Match to Glycocholic acid from NIST14': 'Glycocholic acid',
 'Spectral Match to Cholic acid from NIST14': 'Cholic acid',
 'Spectral Match to L-Tyrosine from NIST14': 'L-Tyrosine',
 'Spectral Match to L-Glutamic acid from NIST14': 'L-Glutamic acid',
 'Spectral Match to 12-Ketodeoxycholic acid from NIST14': '12-Ketodeoxycholic acid',
 "Massbank:EA019604 Ranitidine|1,1-Ethenediamine, N-(2-(((5-((dimethylamino)methyl)-2-furanyl)methyl)thio)ethyl)-N'-methyl-2-nitro-|1-N'-[2-[[5-(dimethylaminomethyl)furan-2-yl]methylsulfanyl]ethyl]-1-N-methyl-2-nitroethe": 'Ranitidine',
 'Spectral Match to 1-Amino-2-ethoxybenzene from NIST14': '1-Amino-2-ethoxybenzene'
}, inplace=True)_____no_output_____fig, ax = plt.subplots(figsize=(8,6))
sns.boxplot(y='Molecule', x='Intensity', hue='has_surgery', data=long.query('Molecule != "nan"'),
            hue_order=['y', 'n'], palette={'n': '#66c2a5', 'y': '#34495e'})

ax.set_xlabel('CLR Intensity')

# legends will be added in Illustrator
ax.legend().set_visible(False)

plt.tight_layout()
plt.savefig('differential-metabolomics.pdf', transparent=True)_____no_output_____