# Notebook from changsin/FSDL
Path: notebooks/bayesian_mnist.ipynb

_____no_output_____# Modeling Uncertainty in Neural Networks

A typical neural network outputs a deterministic value for a given input. In this respect, a neural network is basically a function that computes the output by multiplying weights plus a biase for the input.

$$ H = W \cdot X + b $$

Then a non-linearity is applied through an activation function $ g $ to get the output that we want.

$$ \hat Y = g(H) $$

In reality, the deterministic nature of the neural network creates problems when we want to model uncertainties. For instance, when a neural network trained on MNIST dataset always predicts a single value even if the given input is a totally random image._____no_output_____![Screen Shot 2021-04-13 at 5.43.49 AM.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAIQAAABmCAYAAAAKyDkuAAAMZ2lDQ1BJQ0MgUHJvZmlsZQAASImVVwdYU8kWnluSkJDQAhGQEnoTpFcpIbQAAlIFGyEJJJQYEoKIvSwquHYRwYquithWV0DWgtiVRbD3xYKKsi7qoigqb0ICuu4r3zv55s6ff86clpncGQA0+7gSSS6qBUCeuEAaHx7MHJ+axiR1AhxggA4/JC5PJmHFxUUDKEP93+XdDYAo+quOClv/HP+vosMXyHgAIBMhzuDLeHkQNwGAb+BJpAUAEBW8xbQCiQLPhVhXCgOEeI0CZynxLgXOUOKjgzqJ8WyI2wBQo3K50iwANO5BnlnIy4J2ND5B7Czmi8QAaI6COIAn5PIhVsQ+Ki9vqgJXQGwL9SUQw3iAd8Y3NrP+Zj9j2D6XmzWMlXkNilqISCbJ5U7/P0vzvyUvVz7kwxo2qlAaEa/IH9bwVs7UKAWmQtwtzoiJVdQa4j4RX1l3AFCKUB6RpNRHjXgyNqwfYEDszOeGREFsBHGYODcmWsVnZIrCOBDD1YIWiQo4iRDrQ7xYIAtNUOlskU6NV/lC6zKlbJaKP8+VDvpV+Hogz0liqey/EQo4KvuYRrEwMQViCsSWhaLkGIg1IHaS5SREqXTGFAvZMUM6Unm8In5LiOMF4vBgpX2sMFMaFq/SL82TDeWLbRGKODEqfLBAmBihrA92mscdjB/mgrUJxKykITsC2fjooVz4gpBQZe7Yc4E4KUFlp09SEByvnItTJLlxKn3cXJAbruDNIXaXFSao5uLJBXBxKu3jmZKCuERlnHhxNjcyThkPvgJEAzYIAUwghy0DTAXZQNTaXd8NvylHwgAXSEEWEABHFTM0I2VwRAyfCaAY/AGRAMiG5wUPjgpAIeQ/D7PKpyPIHBwtHJyRA55CnAeiQC78Lh+cJR72lgyeQEb0D+9c2Hgw3lzYFOP/nh9ivzIsyESrGPmQR6bmkCYxlBhCjCCGEe1wQzwA98Oj4TMINlfcG/cZyuOrPuEpoZ3wiHCd0EG4PUU0X/pdlGNBB7QfpqpFxre1wK2hTQ88GPeH1qFlnIEbAkfcHfph4YHQswdk2aq4FVVhfmf7bxl882uo9MjOZJQ8ghxEtv1+poa9hsewFUWtv62PMtaM4Xqzh0e+98/+pvp82Ed9r4ktxg5h57CT2AXsKFYPmNgJrAFrwY4p8PDqejK4uoa8xQ/GkwPtiP7hj6vyqaikzLnWucv5k3KsQFBUoNh47KmS6VJRlrCAyYJvBwGTI+Y5jWK6Oru6AKB41yj/vt4yBt8hCOPiVy6/CQCfUkhmfeW4FgAceQoA/d1XzuIN3DYrADjWxpNLC5UcrngQ4L+EJtxpBsAEWABbmI8r8AR+IAiEgkgQCxJBKpgMqyyE61wKpoGZYB4oAWVgBVgLKsFmsA3sAnvBQVAPjoKT4Cy4BNrAdXAXrp5O8BL0gHegH0EQEkJD6IgBYopYIQ6IK+KNBCChSDQSj6Qi6UgWIkbkyExkAVKGrEIqka1IDfIzcgQ5iVxA2pHbyEOkC3mDfEQxlIrqosaoNToa9UZZaBSaiE5Cs9B8tBhdiC5DK9BqdA9ah55EL6HX0Q70JdqLAUwdY2BmmCPmjbGxWCwNy8Sk2GysFCvHqrF9WCP8na9iHVg39gEn4nSciTvCFRyBJ+E8PB+fjS/FK/FdeB1+Gr+KP8R78C8EGsGI4EDwJXAI4wlZhGmEEkI5YQfhMOEM3EudhHdEIpFBtCF6wb2YSswmziAuJW4k7ic2EduJj4m9JBLJgORA8ifFkrikAlIJaT1pD+kE6Qqpk9Snpq5mquaqFqaWpiZWm69WrrZb7bjaFbVnav1kLbIV2ZccS+aTp5OXk7eTG8mXyZ3kfoo2xYbiT0mkZFPmUSoo+yhnKPcob9XV1c3VfdTHqYvU56pXqB9QP6/+UP0DVYdqT2VTJ1Ll1GXUndQm6m3qWxqNZk0LoqXRCmjLaDW0U7QHtD4NuoaTBkeDrzFHo0qjTuOKxitNsqaVJktzsmaxZrnmIc3Lmt1aZC1rLbYWV2u2VpXWEa2bWr3adG0X7VjtPO2l2ru1L2g/1yHpWOuE6vB1Fups0zml85iO0S3obDqPvoC+nX6G3qlL1LXR5ehm65bp7tVt1e3R09Fz10vWK9Kr0jum18HAGNYMDiOXsZxxkHGD8XGE8QjWCMGIJSP2jbgy4r3+SP0gfYF+qf5+/ev6Hw2YBqEGOQYrDeoN7hvihvaG4wynGW4yPGPYPVJ3pN9I3sjSkQdH3jFCjeyN4o1mGG0zajHqNTYxDjeWGK83PmXcbcIwCTLJNlljctyky5RuGmAqMl1jesL0BVOPyWLmMiuYp5k9ZkZmEWZys61mrWb95jbmSebzzfeb37egWHhbZFqssWi26LE0tRxrOdOy1vKOFdnK20potc7qnNV7axvrFOtF1vXWz230bTg2xTa1NvdsabaBtvm21bbX7Ih23nY5dhvt2uxRew97oX2V/WUH1MHTQeSw0aF9FGGUzyjxqOpRNx2pjizHQsdax4dODKdop/lO9U6vRluOThu9cvS50V+cPZxznbc733XRcYl0me/S6PLG1d6V51rles2N5hbmNsetwe21u4O7wH2T+y0PusdYj0UezR6fPb08pZ77PLu8LL3SvTZ43fTW9Y7zXup93ofgE+wzx+eozwdfT98C34O+f/o5+uX47fZ7PsZmjGDM9jGP/c39uf5b/TsCmAHpAVsCOgLNArmB1YGPgiyC+EE7gp6x7FjZrD2sV8HOwdLgw8Hv2b7sWeymECwkPKQ0pDVUJzQptDL0QZh5WFZYbVhPuEf4jPCmCEJEVMTKiJscYw6PU8PpifSKnBV5OooalRBVGfUo2j5aGt04Fh0bOXb12HsxVjHimPpYEMuJXR17P84mLj/u13HEcXHjqsY9jXeJnxl/LoGeMCVhd8K7xODE5Yl3k2yT5EnNyZrJE5Nrkt+nhKSsSukYP3r8rPGXUg1TRakNaaS05LQdab0TQiesndA50WNiycQbk2wmFU26MNlwcu7kY1M0p3CnHEonpKek707/xI3lVnN7MzgZGzJ6eGzeOt5LfhB/Db9L4C9YJXiW6Z+5KvN5ln/W6qwuYaCwXNgtYosqRa+zI7I3Z7/Pic3ZmTOQm5K7P08tLz3viFhHnCM+PdVkatHUdomDpETSke+bvza/Rxol3SFDZJNkDQW68FDfIreV/yB/WBhQWFXYNy152qEi7SJxUct0++lLpj8rDiv+aQY+gzejeabZzHkzH85izdo6G5mdMbt5jsWchXM654bP3TWPMi9n3m/zneevmv/XgpQFjQuNF85d+PiH8B9qSzRKpCU3F/kt2rwYXyxa3LrEbcn6JV9K+aUXy5zLyss+LeUtvfijy48VPw4sy1zWutxz+aYVxBXiFTdWBq7ctUp7VfGqx6vHrq5bw1xTuuavtVPWXih3L9+8jrJOvq6jIrqiYb3l+hXrP1UKK69XBVft32C0YcmG9xv5G69sCtq0b7Px5rLNH7eIttzaGr61rtq6unwbcVvhtqfbk7ef+8n7p5odhjvKdnzeKd7ZsSt+1+kar5qa3Ua7l9eitfLarj0T97TtDdnbsM9x39b9jP1lB8AB+YEXP6f/fONg1MHmQ96H9v1i9cuGw/TDpXVI3fS6nnphfUdDakP7kcgjzY1+jYd/dfp151Gzo1XH9I4tP045vvD4wIniE71Nkqbuk1knHzdPab57avypa6fHnW49E3Xm/Nmws6fOsc6dOO9//ugF3wtHLnpfrL/keamuxaPl8G8evx1u9Wytu+x1uaHNp62xfUz78SuBV05eDbl69hrn2qXrMdfbbyTduHVz4s2OW/xbz2/n3n59p/BO/9259wj3Su9r3S9/YPSg+ne73/d3eHYcexjysOVRwqO7j3mPXz6RPfnUufAp7Wn5M9NnNc9dnx/tCutqezHhRedLycv+7pI/tP/Y8Mr21S9/Bv3Z0jO+p/O19PXAm6VvDd7u/Mv9r+beuN4H7/Le9b8v7TPo2/XB+8O5jykfn/VP+0T6VPHZ7nPjl6gv9wbyBgYkXCl38CiAwYZmZgLwZicAtFR4doD3NsoE5V1wUBDl/XUQgf+ElffFQfEEYGcQAElzAYiGZ5RNsFlBTIW94gifGARQN7fhphJZppur0hYV3oQIfQMDb40BIDUC8Fk6MNC/cWDg83YY7G0AmvKVd1CFEOGdYYudArW2oEXgO1HeT7/J8fseKCJwB9/3/wLuro6WBfD55wAAAIplWElmTU0AKgAAAAgABAEaAAUAAAABAAAAPgEbAAUAAAABAAAARgEoAAMAAAABAAIAAIdpAAQAAAABAAAATgAAAAAAAACQAAAAAQAAAJAAAAABAAOShgAHAAAAEgAAAHigAgAEAAAAAQAAAISgAwAEAAAAAQAAAGYAAAAAQVNDSUkAAABTY3JlZW5zaG90rVDEWQAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAAdZpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IlhNUCBDb3JlIDYuMC4wIj4KICAgPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4KICAgICAgPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIKICAgICAgICAgICAgeG1sbnM6ZXhpZj0iaHR0cDovL25zLmFkb2JlLmNvbS9leGlmLzEuMC8iPgogICAgICAgICA8ZXhpZjpQaXhlbFlEaW1lbnNpb24+MTAyPC9leGlmOlBpeGVsWURpbWVuc2lvbj4KICAgICAgICAgPGV4aWY6UGl4ZWxYRGltZW5zaW9uPjEzMjwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgpq6KnxAAAAHGlET1QAAAACAAAAAAAAADMAAAAoAAAAMwAAADMAABYOW3jhhwAAFdpJREFUeAHsnPlzVVW2x1fmEQiBMA8BZRCHxzxoWwJlva7SEn7p90P3r1Z3afmD/kFWafUrX1c50KLg+KTLUhGkAREQlECCDCETGQiZh/f9rH13chISSEgCwrv71r3n3HPOPXfvtb57zftkWIb1271uo/1jhjrCe7Tz97qf/w//L+OeA0IMz8iE67e2zIxwvK+33/r706i4lUJTf2TqADECzzOzMi0nN8dycnItS/vDxUF2ZpZ1dXVZe0eH9Xb3pkEx9fy/5R/uDSAAh2Z/bm6uTZ8x3aZPm+7AyBIAXEWkoMFlLS03rLGx0dpa26y3t/eWDqcPTC0F7ikg8vLzbNasWVZaWipw5ElKZIGT0KQhMGeam1uspqbGWrTt7UkDYmrZf+vdpw4Q/BfMjqaANEReQb6VlZU5KPLy8iwzE7URGjZDht43JCGqr11zQPT09MTTQ7fxnkOPpr9NAgWmDhDM+GzdXiIA/slMtIKCQps3b66DAkBwLjYHRJ8AcSMAgm2vAJGRkWl9/X3W090TVEgaDJFkU7KdMkDgSeQXFViUBDB/xowZtmzZclu8eJEVFha6ykBKuCBBQujV3NRkl69c8S0SAv63t7dbfX291EmzdXd2Twkh0jcNFJgaQIj5OXk5NmfOHCspmWHZOTnO/Llz59r6dettzZrHbNp0GZbZ2X6OTgAK3g3Xr1tVVZU1NDS4x9HX12d1tbV25uxZO3/+vIzOFuuXW5puU0OByQeEuMpNC4sLbfny5a4eAES2mL940SL7w7N/sA3rN9jM0plyPwNQHBACEbKi/nqDMx6JgAuKlLh86ZIdOnzYjh49KnDUWV9PX6AGJkgaG5OKjMkHhLrn6mHmDHviiSds4cKFDgY8CvafefppW7dunZXMnBmO43qqYWNkZWRZU3OjXfztN7suSdEtQHQLEBcvXrSvv/7avvv2W6u5ViNbIgUI1zX+8/THJFFgSgBBAGr+gvm2c+dOW7VqlUsCbAVcTkDy6KOPWlFhkXFdpoxGGoCgM51dnQ6GmzdvulEJICoqKuyzTz+1//3qK7si+6K7u9v6ZYDqJ+k2yRSYEkBk52bbqpUr7c9/+Ytt2rTJCvLzFa7OtKKiIluwYIHHIQhKhT8f5Gr8TkCqt1cGpQzNHu3/Ivthzz//afv27bPKykrrUCTTXVLCFEiJdJs0CkwZINasWWMvv/yybdu61YEAIAoKChwMACMyn21swd8I0gJXk9Yro/KXX36xPR98YB999JEDoq2tTZKkSyd1weDP/fr0x8QoMKWA+Ntf/2qbN292FxO7wgEhtVFcXOx8lAAYEotgKAEg/Yo9cNIML+OsJMQHAsTHAsQFSQjc0M7OTp2c2ODTv76VApMLiNRszc3LtdWrV9urr7xiWyQh8glCSUKwJWwNIGgoi6SE8IOpD2wKGsGps5IQACJIiAsCRIcbnCkhkvpFejMZFJg4IAQCZj8GYrbnJjINQDwulfHaa6/ZNnkVhVIVusjjDgSkPEopKASWjzaMcBajEgmxZ88e27t374AN4YZlOh4xGvHu+vjEAOFgMIEhy5nskkCMJ6v5uLyJ119/3Z4WIFAVgIY3KW7PcqrLsDxKgpFH0O9up6uM99+3vVIZBK0wKtOAGJliEz06CYAQk7Oz3E6A8UiCXAWcAMQbb7wxCAgMArGfP0y2pJQIZ8J1fX6tuZdx5swZe/+99+xDSQgAQcCKd9qGSFJycvYnBxA52Q6IqBrykBCPP26vCxDbtm2TyigcU2+BAh3q08slR3+GvIxeOytAvAcgPvzQKgGEDMpuJbu8JRE1pn9JX3Q7CkwYEJkksRRnmK7E1TS8h5TKeOyxx+yVV1+1zYpDYDfA6ju1KD26urscCFxPTII4hBuVH3/sNgTSYaBWIg2IO5F1XOcnDAgMSGoclixd4sks7AOMy/Jly+y//vQnJbLWOGDGCgjAUNdQb21EKj1A1euRyk/277eviFReverqIh2pHMbn5HybwCS5e0CoAxnyLCiJW7VylW3avMlWrlih0rhcD0eXKlexRTGIufPmybvIGdb7wa9hHHgcein20NDYYKdP/2zV1YHxfcpbXFJy6+DBg3bs6DEvr+uTGiFMoZ883C3JZEaaHHPyHPvJ70m6JPfHQK27A0SqAzmSDrNnz1ayar398Y//aWvXrpOHkeNqI18lcuXl5R6ljPmK0foTwUBU8tLlS/bNN994xrNDASgkASV1p06etHPnKiQ52sLgGSj9GOeAR+vD7/K4xicN7C0MN/Ul2dnkIb9Y1CTPE9s4g3fjA0RkAh2V7ZCrGknUxcaNm+zFF19wQJDVpJweA3O+pEO+tqMBgvA0aiHOeABBpvPAga/E/HPWoQBUn87Xqh7iVwWnftO5jg5FKCFCHHPcRgI8LFtoLDrGN3Ee3Pm4VAFpGofuWweGDHLRq0sJQkBxN2r1rgFBBymaLVMRzJYtWwSIFz2tTd0DPS0SECiQoZiWQcXGQGLr6OywJlVY32y76aFqzuBWfvnllw6IduUs+qUyqJTCdmhQjYRXUQ3eYhAY8aYP+hZS6Y0dRgAvl7eAUFRUqIqzEnfpmYye+FMRMpOI8H6UsoT1KR242do6mBVGSpBUHoO0uGtAZCn2QGfL5pQ5IHbt2m0bNmxI1TgoTK1BkO6mCCZMaW1SjTEziOaWZrtw4YLVidG4l0i6qqpKBwTBKAABuPAqWjXA9rb2QQBwkyQw4s0f9K3GBcMBw3RVlVF2WCQvDdU8TxK3QPuUEkA/JgfvbmWGgwHep4qyZpekVJmRBOzq7LL+nhShoNkd2t0DIicFCKmMrcpX7N692zZJdaAyWIRTmF8QimDkdQzlW/iGoGhqbnJVUFunKijUhwZJmdwXX3xhBKPcXkgNvFOqokeLdwaMp4cZEJKoSN+ZJSU2Q+9iZYdni84UGAEO8kIOCNWFdOkdJlO/QNFnTapJray8IKO82qAZMRsKlF06TDogIrp042wFo6KEABC7du3y2gdUBjZDoWITDCY7Syok0YLuI2KpRTmqrCa1DZp7BQjyFrcAQsghTD0QexiKrsSdH8BdxhKZFLcc0j6AKBH9qEktKiwWIGbbokULPaaDa082GJpQBoB04FbDAdEt6UDVWVeXCpNRF4n/0LcR2/gkRLyFbgwg4joLbAgkxMaNG11lYPgUICFGAES8BX5jY1Ojnf75Z/ciQDnFMKiQAwcOeEKLWARup4tGoZyBDxM3A7d7oHaSoIaWUr/Qkm1o/R67KVXdaUnJTAcB6nf+/Pnal5GO1I0TRZMFuvVhROoYEqJKdhi1p4AB+4tFT16YPLWAyNEggpcxXkDQL5h7RbGG77//3i5qAK4HpR6uyng8rILaqsoqr3sAOIhHgMH2oQBEAr0Y57Nnz7KFixbJZpjm6hYjvLi4yKhSx3YoVLkh9sSsWaWyLVR9pvNIESQCYEDVikxOH6rSqUHFM2uVBD4vV71Cahj1O8QdTfQhuTsBCZEjlzLfRdl4AUEHKHA5V3HOPvnkE89VdPV0uxik/B6DsramVgjvTkk50C8s+Eey+w/ovphJyB9BgdFNuSEG+bz5CuJJ8mKHIV1REYAiX9KW8D/SIVtBvui1QQ9URww78B3j+8qVq1Ynu+y6jPVvVZhMXKe+ToY7Eja2pJSKx7SdMCDwMqiKGq/KaG9vU0TytH2gOocTJ064nYDzBMKpm2xpbA75ChGP2QAWHhbpgBeB4c2Q8CaoQt/+3HO2ZMli2WVhaUJp6Sx9X+KAcPdTXpsb7KpMZ5bgZtKgS7DIAnmgq4Oh4bpAUGf7FfKnFrX6SrWkSSohqN/7z8Mt/D7x474C4uSpU57F/PHHHwcAgc67fPmytTS1aP1FwqugxyMMIA7kgdmKGbGYiOHA7PXr13uFernyQUR6YfxMVZYtXbrUXc08RYSzZJxjrMMw2gAgfJ9j4XhHZ1jlxgp6pAJlhx+qQPmqAIGdRkPCjCZtJwUQWzZvsV27d43DqOyXj9xuJ0/+ZO+++64dFyAwHHsF92a5olcl8lj0OwQQDwoY4At9DfyB/gMNRsBsqtJZtcbMJui0bu1aB8TS8qVG6QDSA2MSiTFv7jxJDUUoZUjGWwamBYKET86Es+0AQkBo0IKnehmWHytDvFdlA9XV1wYA4R3ih+HHA/1jZ9IAgcpADxKwAskF0nf40dmZDDy0OCC+td5stePHjtn//OMfviIL/cabRb519XXWfrPdDcnUTx+YTUaWRsmAE4MlbgAYYiHRtGnTBjLA2BBrBYjtO7bbUqkIqs4IPBUXFducuXO8BhXXPdoNIxFiUGpoHWxHuxuUtXW1iuw2uLr4SIVF13yB0z2yIaJR6YDQDGBAVE+BcgARoRhoFLyGZtkKeBNvv/22HTlyxI3MfsUiyF+0kvruwreOUBqJDL/PY8x+H6c+cP1dIkgawHhWv5cpnoD7iNdAI26DDfHMM8/4Usd8eRE0v16xHJ62c3swxFkdqNUmG+KaglI1tTVaIN3gRjs2hK94QwXfoU1IQgy6nSFSuXGjJITQjMijaGYkQOA6sggHfxmX88033/QtMXjYT5U1rtQtLtIDgg08r5hrYMu6VpiKHQAIiDaWl5d7SDqokGxJiP+wrVu2ur2Qn5OXki5xwGEajcZHzkYJQaUZK97I+9QqQ1wvj+3zzz93w/KanrnBY5ru1MYPiNQzPkBwnphOBC1KCAJTSAdmBW4SyRiKamOj8/6sBzEdoycC4uB333ncHQIG1ZGKN0CTSI9In3iz+7kNXAg9QAwk+skkYfz0G8MNQAQvIeQmeBTC8uXLHRycx4Z46qmnnIbzZS9QO8LtbzfcQJJQQ8KVYZl0oG2rEoXEcigZABRfKFGIaz91EiJFjCQgYi4DleGh65TKGBEQkhBURQGIQ4cO2VtvveW+MokYmgehCLTgXN+OKn717+hDdMF7QFVmafzYDSoJdXDkSkLA+GkKPC1SAApATNNztmRW+PEnn3zS80Fzy+a6sTmWUcEGWlzQxHeilTdabzggooQgL+SAqGaR9FRICP0zA+FpckFChORWyHaud92HlEBljAQIfGFi8KRoD0llvCUbguAJKgMEMGsAg2OBD2YgLVIgfLuvnwSVAD7he3cH9R0AFEhdzJmjQJIMw4yUlEBaAAbKAIg+UhIAKFAfwdDMtkceecQXQc8qKQ3S5Q6jgxRBtLP2NaViZX8BCMoJyAfhuhPk+07Sl5Xz9XUNUwkIuUvSiaRiQz3EZk9uhVxGqmJKRGFGDF/USzqW4AmdBRB//++/2/cHD4XUdkYCCBDlfkiIJPD4//g9sc9kwFMgmsg6VWoXAAir0lh+MFt5B1QFDOdaJgdqA4ORdDbXQTvOc3uSgB6Wlv3AsdjYCyQIn+FMUBVMHFbKE8iDnh7Gliquvlptx48f8zrUJsV0fv31nGeO8dr4zZ3a+G0I9colgNxKluQRqWSF90svveRuJ6oEKuJPF08rHliUEztCWRzhVaJpqIx33nnHfvjhB3cz4zX3dTvIhQAG0TBTriSLkbyJqPFpesQIqDYnmYcUYGX7Vi07QAJAB5jLxGEZAkYlxiW2FSCKcQj+DrcSQzyCIQkC/jNME2wFrAWlvnlJIgAGbAVcdVQtWWGqypC4p06dVkynxV1QHr4yFoOS/xo/IPSjHLlWPBJoxswSF4FE2l54gRK6tW4U6a5OEAaPyORPGApuZcuNZmu83uiAOCIgvKv1FjwZZkjxCz27n039R7JlqeYDewDgF+nNPszJk/gvlRSYpzwDksIZL0BQwLJJYfz4kBQmDhMD1QpA3O6SpEBaACIYL6Go34dH4fj3FPsHYIAKFd26JQ2IMWTIMCHET9khdhjeA8/2DIDoGShIJi3A8aamZjfYx2qTjR8QIlahyrnmSDKUiSAQASA8//zzXnKP6KQ5MRi4iMKL1qmSOQZA8QbZOELW+xRJO6XOo0qcQn7l/fsIrmK2z2JmMgUpy2QEEkamThTmYzvw8FWeguOr1dRdZjx2AQ9D4bhPBK7VhMjSOb5DmxyXBkpfIxpoDoj4hRka5AEMdvtAST8WN9fUVCuLGTLAntCSgQizUb3+cBXPBvf6w9koLrp4MdSfEhFGcri6iPZY+OcRP8cNCAg2Q0UbrMMg+bJg4SK5TU/as88+68RgZtEYlkfYUvKht58oZItVKB17vqLCrknUnTnzs/3rwL+UzKoaXHgzYjfv3cHM7PBgE4w/ahB4b1W9B2pxZorREezYATAZYiM9YPwMAQXACDkaOQzWp+8obK3EVFQLiP1wReoaH2IAA9SDiQSZYPZ1SdSTJ0/4UoRaBZzwFjAgW1tvuLro1GQCQPyOda9c39qqWhJApWDUQEwHEXSHdleAKJGqYCY8qnUYixVvZzEOS/bKy5cNikINl1QMQ+RPIADBqNNKaPG+KilBZTXrLchdUO/we2iE3gE8dgCpZ6rKd+zYYTu2b7dZqk1gxsPUOONhPA0VEF+IdF6Mn1dgc9gPY5TI19HhjTtxNGkwYicgTQ8fPiT3cb9shIsOCK4hokudA0wPvwyuZ6gdSdw/sTv8P4d/D/0dfvR239VrZsqq1SttxapVTjiW7RGcKi8vF2GwFSBQ1Iv0JgwVQJz66aSdPHXSfeUKAYKny11T4oXk1qgtOSBuNbwlzw8/N5bviXuSbwAQixeH1DMFKjt37nRAAA6kAy4mwGCfBkDii64EAISJwPGhLQAkXBPOhCtSx7EZ9KJepEVeAoXIGOCHZYDv27/PK8qY+byJ3fB2KTCcBqEjQ/96DN/GDwjRAEAAghUrV0plLNT+6gCIpeUSnaGoNpIhDDwMFh/5J9U+sOjmqqqleJjYkR+OeCaup0uAiD8aQ8cHLomEuN1vuUbnXVxzXertTIzHiSXoFCoANxDDEPuoTIDYsX2HPad6hbKyOQEQAkAMPvEbbpjcxi7FWeuXpK5wdstIRLLwm2hsh6qnwEWYDSAoBcBwpCr9B02cTz/7TJVklQ4G1AYqhfiNFx8P/mkYH3+aPBY6ccdPejW+n0VA6OGjK1asdMKtTgECwysks4b+L3+CyriulCyG5IkfT7hhWXnhgv1bHgZG5ljdoqF31jd6H7hxyyk/kDon+vsyQw8myU6gDdgCMhZxG2ESgCBLW15e7oAgZsBzNdc+tdaNRZcQug4VwfVRLQx2IpAzzHP/m4EPuuLV5WKmSxfuIYnQJqbCfGog+TXqoE0haJJTSAeKkH/SJMJNJywdbAg8j1Bk6+p2fFwc6NPwnQkDIikhRgcEvnOv5+iP/fuoAifHfWBVqqU8JkBgYE7a02mhemwJsLgqUMyAtag5+Tmu0rADYDiqwANMAgPAQE0g/ZYsXuxBpQXzF/o1hKCTtw+zHBLyovE5yJlbQaFSebmLlAZ6Ak+MRxpQA8kKd2wFKs85Rj0kMQakA3EbJg32A/sYlAAyeBuSNoN/6b2YyMf/AQAA///4fcO+AAAWxklEQVTtnPlXVVeWxzfzDI4g+ICHBOeqZCXRqIkV0pYaU7Xql+6qVWv1Sv2fXcNa3ekhiUM0iXMUFTU4IAIyj/397PPO4/KAxyCx1ebo5U7nnnPu2d+z5/sKrMBmbTWl0Gzz5s22b/8+6+jYbU07d9q+fXvto48+snRr2ooKixa0VmCFNmPT9qzvmV04d94uXLhgP//8s3V3d9vFixft8ePHNjM9Y6scyYJ+/PmCxGWOeTvti0uKbevWrba9fpuVlZdZQUGBFRcXW0NDg79DXV2dlei8rKzM6+3Zs8daW1u9bl1NnVWUVegZvXxOCV2ETuZ3PX9iZzUQ/k1PT9no6JiNj41pXAU2PTVl12/csIvffWf379+3iclJm52Zsf6BAZ+jJ0+e2OjIiA2+eGGDujY1PT1/nlZHvZzRLzwtWFdApAWIgoWAKBQgppOAOH/eX/aeAHHp0qX1A0R8v0ClcJahEoDYvm2b1TfUW1lFmRUWFlqRALBDgEilUgYgAEhJSYlt377d9u7bZ60tLVZaVmq11bUCS6mmKjSWaTL2JrDPZu9lL+pgdjZQq1BACkOatcmpSRsRgcfHx/WU2YyIf+3qVTt77pzdu3fPpgCE/g2I+A8ePNTcPBIgRm14eNi3LP05yJ4ke32545cDxO7dtrNppyYvP4fIAuKZOITAEDkEEwAgnohDTE+JQ6ylMCnMdqRSZu+XRHQIDKEBAByivmG7lVeUW4HOudZQX29NTU1WW1ubBUS9rsEhmpubvW5NVY0VF5VkiR6BESmyJCBEMcYBMAI2Zm1iYsKGBQj2XJ/Rir9x86ad/fZbY4E4IHQdDvHw4UOLHAJAjI6O2gwN/UJgYPrXDgitoN0AIiVA7N1nhw8flsho1cQV025OKdAqmrFnz3oFiAv2nUTGQ4mMeYBYi8iIK0SzXlRcZMWlxVZUVOSsvaio0CorKmybVvumuk0SGSW2RaJue/12FwsFhXp1sezNmzYZAKiurtGzgWtwLZVq9rrlEiHlZeVWmOB8Gcw5XZzgibedAwsXBYDJCXsxNORcYVIgGBJhnz9/7hwC7jAtQNy7e9d++PFH55pTEiEAZUT1nvb2Wl9fn4sXAARXcUCsce0khrnk4ZoAwcTu3bvXdmsVAYh9GUAgc5OACJMDAwyrpBdACAxJQHz//ffOIaamcmTjkkNO3KBhiXWIy6qv0SqvKBfxpMeUlJb46t+3f7+1pdusQuBA99kmsVEmMcDYAATXa2qqrbQ06BWFaqtMx1XV1aEtAQy9KAmCxAjyHs7OzFpff5/duXPHiT04OGhPnz71lc+KhztMCxS9IvzDBw+cKwAS+oqiZUTiAtBwnfowCJ/QvD2v/eaqAYFexcQ6IJxDpKRUBg6xGkCgVN7NiIynUpxWBYhIHd5b4ykU0WpqalxBrBYhEQXlAkaLdAA418GDB/3+Jq38zZs2W6m4RaHAQIGjoEtEOc81gALIKMy/g8fPlqdF1BuoPjMz7UC4IIXxhhTHZxKZiIGuri4b6O8PhBaFAceQuMjkxKT3wtDgBCjaAMGXFAOhxH04W/e/rx4Q0iG+0wQlAeE6xApFBoQKRJTyyszpP+esfAAA0TnHWkBZPHTokB3Yf0Dco8ZBUV1VLS4WVnyc20DwCIBwFSJwxc9EHOp4yQCJ49lZEYsKuhX0gRkbnxh3QnODawDg66+/tstXrlgfgNBCgGOgNGJhwCHQGyYRFeIo80o8jQOJ+3mV1vdk7YCIOoTMTrgFZudKOMT5BCDm6RDLKZWaDGgBN6iqqrJqbcUSCyiHAADi7xfhG2RFBEUSa2GblMO9lk63umhAFygtKc1yB6YyED5AIp7HKfaVSR2nuh849wABmI+YiHAByrQAPSalD5GA0ugrW8/19PTYt998Y9euXbMBiQxAgZk9oroAYkYgcDAJGPNKBMO8i7/8ydoAsWWL7RcgOiQy0NBXIzLWBAhWBmAQ8UtLS22L+kcRBBgABJPynXfesSNHjgqUMhVVB5Bwv6G+wUUcnCWIhexa99kNgKD50EkEATfjMQSLG7UgNgSF1QdrQatcOhCr/vr16640IvfxJzwS8fG1/PTTT24l4H/AWqC+Ay3DYXwwodPs4f/FwdoBIWWto6NjHQBxUUrlk/xmp6jgBM3oBvgJWmQSVktvgENgVmL6fvrpp9bW1uYcAkCgR9TW1LqSKIkskuNYWrj0mAQInXsnAgLCwNohMAUWj7WAkghxER3TInCvLIIfZEZjHQAINvSjH2VB3O66rbpjAVhJ0UCnbAyNkjuIcPWV/X0pQOze3WGNjU0uMo4gMtJpX8Vhen1RZ9+P1YCVMZ9D3M04pp7YTK7ISE6SpgPtH+UP3QAHU4s4Ab4D2i2SqXvgwAHr7Oy0CAi4CRYESiYiJI4pd2YBQigBEkl6cAWVjoKMhzNQIDTcAY4wMjKcuTbjJiL6EWKDOlgFERAokmPjAgRNJDvxp1+fP6sGBEsJB8+BgwfcD9HU1OiAOHTosCt1rMzFJj8vIB5lADFHnTBDgUbOHUokFgCDu5YFCHwemIZUgeD7xbE+/uQTa0un3ecAIPBDVFRW+v3gKwzNrpQeM6JecUGxTc5MOTGnM/oCvoIXciX3y1KAQ1AAAOeIBwABF/F31vG1q9fsjnwNeBxfZzDwHqsGBKy7YUeDvfvuu9Le97sfor39HT9vEreAfS9WVg2IDNUwcyurKl1nwMlUIxBgUbTt2uUWBSYiMYiURAigIDYRPZOYlyXSJ9AdkoCIq36xceZeQ3l80ivfgayDMcl/uMSkiI24wGmUBMSwTMefbt0KOgR+FRXiD90yrx89euz6xtsBCC3DrN0ujyABrY9k3wMKjtNalVgaEIrJZxkkV6GjLooM+ewXmJ2Ss4uKDPVbIu/jjh07XCSgxNJHY2OjuFOHlMutchoV+kpENGyWsoneQH/CiTbBQAc6nFeSY5t3I+cEDoHH8KL0grPnz8mB9Mw5wZTiEXCDZzp3B5PejXcGKHAHgKNB+TjwLsJNxqQ/eAAvp4/X7dRptdygCorQ0EP0Dhue1Xj06FF7//33LSVANMv+R8HEYcVaBA5JSHgnCUB4LKMnuq6/l5yVzM2sqOxYmGMAUVbiCiT+hF/9+tcejMKy2SMAIrogNlXhEm5SAhD9W9GLZTtb/IB2WOH/9T//bX/7+9+1yh85ICA87mfO4RRwP9+kLA5Lp8CcZFCAEY4SHUxce93LiuYNQPhK1NsADII+xz7+2D744AO3//EBtLe3uzkIIJiN8O5hvyZAZGYOF3RLS7MdltL63nvviTvsMERTBASEYOLRXYoJveuYEoGSaWbNu8HBAfv3r/7D/u2vf7Wen3tkbUw7J0Bc4GOAU2QBobHgi5idzlA+OYg3AAxh3uaH7RdOnF6qkKCPtoh6OMQnUuAOHfrQg0Cs2LZ0OrBwsWnePUKCBjlm0nDKnJPIgEMwmfe679kPP/zoET3nEKpDX8QVUAp5Egti584m++DDD+1dOITEBSIDv4NziAwAAIQrtDrnX/4y53lkXIgG9l4yj+Iwwun04sWgffXVf9rf/vEPe9TzSP6DKQFiwjlEjzjEQP+AnpXpoMdpYXR0JAAi01z+cbx+d5fnEJogJ5IUNN4RToGI6Oz81L2Tzc0trsghOmpr67L5EEQ35wqTHgBxPhPcAhAoavj4Ce7AWrHzsRyi04mVDzA4Jx6xd88e2yIxgWOK/ohf4IeAFAS0XH/RePMDgreYA8To2KjHEcY9HB1iBxCdoNL4+KgAMSRT+YJ9I2/j06e9SlABEJP2YmBQCT99rkNEMDHe8THlOeBnoJvlcDk3Qa/N0bKAcHZcIiePTD53E4sA6XTafnvypH0isUH8YKsUva1S8DDzIjFyNXlWEcEdAj2XyJJ68tiVNLKEXA4DCE0Lqz9aC3gc6R/dhD7hRCiNjAUfREmxQMqkC2zUWxkF5gAxLXMSTyIudEQAhEZhBAQoh8+fQ/ARBaPuWNftLndGQXx0BHwKOJoADwPnOmNwX0XoYmXDeW2gEAayIkCUlpe6k4cXRqncJX3hzJkz1inPIIAg26hGHkHuxcKcJIsDQpOOnU7IG64AEJh4UsTiKqPtI0eOeLsQHw6BCxpnFOHtIEpCQCuCzzGR7CxzvPB6EF3cRlKSq4CZSJLO/fvdsg7GHRB9fc8dJCiNmJXoCQP9g+6tDMQPnGTRZJXYKRPA8RtW8gNCL1UohbKiskK+gCp/NeQ0CuTvf/c7++yzz9ziwDeAAwhxEksSEHSCnGZiL1++7BvcApNuQErbxPhEEBlaZW3yL+D1RFEtEyDoD2DUKIHFM52cE2iu436JWY9gQQ+AzXPu7F4cwGW+BooucP36DddruqXPTCj8HCyIPrsrR1KPdAbS3eAE3IMT6b9zhPieb9t+aUBk0F0svwP2PVyAmcC23yWifSEOQexgpwiHixgPIjJ8boHMQcI70UxikhH67erqctsc7x4bBQ7BhmNpt0xY9AScSrRXLNFQrsRYFEzGAFtGfNHbXC+0kiyznpAKQfEVwObJXOrvf668A6Wv6R9Op5s3bzrXIn8xiIwp1yl65YyCU0wonD01SSKLekItii9IV8njZNdv8PHSgOB95ZUkyZRE1Hpt0ReRTqft9OnTdvz4cWuU0whZjzyPjTHZ/MstTHivRMQTbRAFEAEm1xVEYAjNOTkNeBnnCn6QoDzSKtwmgA9ALOyH5+ACRCRh++QkAAzCziizbgnoPmCEE1y7dtU9iegDgDJkRo8KDAGsWbrHruI+e2NupG/6UaThou8BIEhZxxuJtxD5Dacm39ABIdOT6ziE8AFE4sT5io1ynbmL/n4UONLJSXeD8wAMRAAbASx0EQge2+HZ5EBjP1znDhYN93MLHIlUf4iOWxkzkXMCUoiSyckpP8fSIZmVcy/q2HWaOIBkw3Sz2PVknTf4ODnPC17DAaFcxUZp9wSxotzGMXXq5Ck7LkA0ySoIHsI5QCxoKDODOHX6Zbfj5YNbYJXUiRtEQMCByG8AeIEjhJYiIDiDFhEQ4e7c3+R1CIqoAAxsKIeepaVjnEdwGXISsHLmAIGeMNfevOPE5bf5cFlAlFcqN7E17c4hJgIdork5ZSf+6YQdk/sakVEiDkEiapIguZPGPUQClgXKJezZzccMhwAMlOUAQZ1ckza8BKyepJPg/8ACIIaAvhIB0SO/BybmKEEq1WUMiJNbt2+7+9xNSDqIJQmOeO0t3+cFBKntm7ZstoO/OuiWhcttiRF8BUePHHXPId5CwBBYfP4ZZNWOy35HtgMO4g/l4hKIiPBVVFBaIydicLHEIwdWcumqS+6RpMqXYaSvAQw2RNMNZTChyI4pyMQ5eoSns6t/HGF94laPJUrwPcwLPsUO4wDyv1qs9cbvlwaEJoTUtEaJiuO/+Y1HNmNoGxDgRt7VtsuqK6uyomQlc4bsBgzQFJGEh9H1B5E1yWEY2EKahBr8BVxstEXuAeKAD17QBUhrY7WjwF5RciscACUWUAyJa3j/kE4d4HtAnEyhP6zkBd54kud/gaUBoecILKUkHk6ePuXp7FgDEMHT8OVGTjWl/COWOJOrmU+Ina9+LiAcBJl3Qf5DVAeEOAHJrnfFBfCCPpBOgBsaQAAO0tfIZ5wQGIhPeCQSJPBfW4hZBDHjzcdBMcBkideT197C4xUB4tTnp917SGYSZZO+bdizZ7elFHXkoxbK2ucLUoeS5AnQI54nwUBPwqRHHRELnJDehq5w9uxZuy8rAoWVa4gCAHFTgCAfwUvsjJNconMt3s+9F69T5y0uKwLE6TOfe/4DIgPPIaYin/E1NuwQIEo1Pbmztz4zFlsNtOBvECsohM4FRHgKwAAQRFK7xSEQD3AN9IWr+pCWe6PDIdXNH9j4s+QMLAuIVn3T8Ps//EHh7o89NxFAEGVMp9NWv217woEUybdkXy9xI+gKzvJF/OHhIVcG/ZN6tQogsCTIbHrwQCJDrnD0CJRIwIAlMTE28RL9//95NC8g0BnaO9rtT3/+s+IWncE9LR8B3kQcUnXzAlrrBwhERBQXkAJ9gS+iSE1DAcT7CAD6ZSHAN+AYcIbLfDArTyRgACTUJWbyQqbuzJRqrt8Q31qE5AUEDqOOPR325V/+onD3b7MOJIJNHu4moJXxH6x9toMocN2AacZy0BYcU+EeCuKgElVIVsGP0X2v265IFDyL3z8IMFgZ/M4CrmnECSBi41nMy9mE3vjWUnMdXiwvIIhjdCiZ9V+//NJOnTqZ5RAokpieVcqGLvIYAyNZ6/KDG8zFKWiJqCQWRGwRRxIrPYbMEQFXZU6SoALRUSL5PvTWrVue3+CeSIJRNMBO+w1AMLPLl/yAkAcy3Z62f/7jv1hnZ2f4RE5+g0pxBpxTW+S0KlVQK5RIvuU7TdYI4gHahedJPOEbSIgK2+eLKTybmJX8oAYfxxCJvHvnrkcvAQ71uI6ZiXjIfkkOgxHWNgCRnPH8x3kBgZnZ3JKyM198YUePHXPPIrOLldHevstBwQe0sUSixvPl9u5HyDiqnCOo7eciPt8x4EGMDia+fiKPgg9muT409MKeE5oWZ8gwAfcvoEwiIrKm43ID2Li/YAaWBURKgCCyeUyAIDeCUqdfZCFJJrUzpQBVpV9jpa+2hNUdPol354IAgViAG/B9JOYjBEZRJNPqikCB6ECEEKdAmfRe9Ye22NYwjNUO+62uvzggMtyfHAcA8bkAAYdwQGjOa8UhOpT1TBi8orwiO0GrBUUEBEqkK35qCYJ3ydVMzgQuacQBOgMf9/AbCw4IiRNiF/NiD4zC0ZEdzsbBGmYgPyAkMlKpnXbixAn7UB/KeJhanZDgSqobmc+kzmERhFR9RTwzq5S9xyiiaqHVTzYyIXCvo3Yg6KRS2iC6V1MdfAd35DvoFTBcZAgUWBCXlId5XSIDQMRP8Rd87bWGCdh4ZP4MLA0IUQgnVAxk8XtS7roWoUl6JQOaLCpyGgCK/yaTjikQPJnbALFZvLB/lEWijSiLEJxgE3svAgSigDA15iXWA2DBmsDB9EC/w0R9glTcy34QE57e+LsOM7A0INQ4K5yMKX6PgRxHzjHzcFdzjujASVUnjgE4SH0LXCHzJVXG1c01QIJjieQYiE0YfHho2DkB6W2IDPIkqcN9QBLFCADC0hhSiBpQMQbnMhsiYh0gML+JxQGRqAMx+bm/GOmEGOQv8GMd8XN7UuTJouL7DDKv/fsN1SEvkvgH5xCQDCZMQ2f7nnE96P4DuIJzFT3Dyg9gCNR2EMqJwJddC3SGxDg3DtdnBpYFhHcjno+O4KtSFyAyvgg8luz58c8W/V4Dv+kUvqAKqXCACDETk3MBBG5n8hQ4xnfAhzJwBAeEwIeI4PeaFvwA1/q870Yry8zAygChRgqLM4DQSudHQAECugP7eokUvvdE3+AbUBolWRZAoIeQIwlnQTQ4h5BJGX+Shw91IiDQNagXRIJONkTCMuRb/9srB4Q+2GEVw8IhsK9+ERwuge7A95dYHwDA66heqQATz1n5BJsISOGJ5BiAAAZMyEB8rJCQTbUBhvUn9kpaXDEggl2oJuOq1XIOKXCFWU4BSAALgHBdQnsvmT0rn5A1iiNp+IAEncHT16hI9dh+3HsDG39e1QysHBB5RuTAkKiIuoJXFTcRMsJTGc7CCVwmbv5tpF8MVT0AlXkkC4zQwsbfVzQD6wIIX9nJ1Z0cfO5KjwSPdZLnhKjjee5zsf7G/hedgfUBxC86xI3GX+UMbADiVc72G9DX/wJt0m/PPjL/3wAAAABJRU5ErkJggg==)_____no_output_____In this case, the correct answer should be 'I don't know' or 'maybe 2 or 4' with low confidence. However, there is no way of knowing how confident the model is about its prediction. The ability for the AI systems to output their prediciton confidence becomes necessary as they take on more life-critical functions. The purpose of the notebook is to survey available methods in modeling uncertainty._____no_output_____## Types of uncertainties
There are two type of uncertainties.
1. aleatoric uncertainty: the uncertainty comes from the natura of the data itself and thus it is called 'data uncertainty'. The network's performance cannot be improved simply by adding more data.
2. epistemic uncertainty: the network doesn't have the confidence in its prediction. Out of distribution predictions (for instance, trying to predict on a dataset that is quite different from the training dataset) belong to this category of uncertainty._____no_output_____## Approaches to model uncertainty

There seems to be two approach to model uncertainty. First is to learn the probability distributions through a neural network itself. Second is to assemble an ensemble of competing models to get a sense of how confident we should be about the predictions. Since the probabilistic approach provides the probability and the confidence level, we will focus on this._____no_output_____## Bayesian Neural Network Intuition

For probability modeling, Bayesian neural network is often used. The intuition behind Bayesian neural network is to learn weight distributions of the model given the prior and the evidence.

$$ P(w|D) = \frac{P(D|w) \cdot P(w)}{P(D)} $$

The data here refers to the input and the expected ground truth label so the equation is sometimes written as:

$$ P(W|X, Y) = \frac{P(Y | X, W) \cdot P(W)}{P(Y | X)} $$

_____no_output_____Knowing and calculating the probability of the data is practically impossible so function approximations are used instead to output the probability distribution of weights and biases. In the following example, we are using MNIST dataset to illustrate how this can be done._____no_output_____# Implementation

Several frameworks exist to help the probability modelling. In our case, we are using [TensorFlow Probability](https://www.tensorflow.org/probability).
_____no_output_____## Weights and Biases_____no_output_____

!pip install wandb -qqq
import wandb
wandb.login()     |████████████████████████████████| 2.1MB 12.5MB/s
     |████████████████████████████████| 133kB 53.1MB/s
     |████████████████████████████████| 163kB 47.1MB/s
     |████████████████████████████████| 102kB 11.1MB/s
     |████████████████████████████████| 71kB 9.8MB/s
  Building wheel for pathtools (setup.py) ... done
  Building wheel for subprocess32 (setup.py) ... done
wandb.init(project="jupyter-projo",
           config={
               "batch_size": 12,
               "learning_rate": 0.01,
               "dataset": "MNIST",
           })_____no_output_____

## Import libraries_____no_output_____

import numpy as np
import tensorflow as tf
import tensorflow_probability as tfp
tfd = tfp.distributions

from tensorflow import keras
from tensorflow.keras import Sequential
from tensorflow.keras.datasets import mnist
from tensorflow.keras.layers import Conv2D, Dense, Dropout, Flatten, MaxPooling2D
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.utils import to_categorical
from tensorflow.keras import backend as K
from matplotlib import pyplot as plt

print('TensorFlow version (expected = 2.4.1):', tf.__version__)
print('TensorFlow Probability version (expected = 0.12.1):', tfp.__version__)
TensorFlow version (expected = 2.4.1): 2.4.1
TensorFlow Probability version (expected = 0.12.1): 0.12.1

## Define hyperparameters_____no_output_____

NUM_OF_CLASSES = 10
BATCH_SIZE = 128
# BATCH_SIZE = 32
EPOCHS = 10
# input image dimensions
IMG_ROWS, IMG_COLS = 28, 28_____no_output_____

## Prepare MNINST data
Load the dataset. [MNIST](https://en.wikipedia.org/wiki/MNIST_database) is a built-in dataset in TensorFlow._____no_output_____

(X_train, Y_train), (X_test, Y_test) = mnist.load_data()

X_train, X_test = X_train / 255.0, X_test / 255.0
Y_train = to_categorical(Y_train, NUM_OF_CLASSES)
Y_test = to_categorical(Y_test, NUM_OF_CLASSES)

from sklearn.model_selection import train_test_split
# X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size=0.2, random_state=1)Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11493376/11490434 [==============================] - 0s 0us/step
print(X_train.shape)
print(Y_train.shape)
print(X_val.shape)
print(Y_val.shape)
print(X_test.shape)
print(Y_test.shape)(48000, 28, 28)
(48000, 10)
(12000, 28, 28)
(12000, 10)
(10000, 28, 28)
(10000, 10)
data_format = K.image_data_format()
print(data_format)channels_last
if data_format == 'channels_first':
    X_train = X_train.reshape(X_train.shape[0], 1, IMG_ROWS, IMG_COLS)
    X_val = X_val.reshape(X_val.shape[0], 1, IMG_ROWS, IMG_COLS)
    X_test = X_test.reshape(X_test.shape[0], 1, IMG_ROWS, IMG_COLS)
    input_shape = (1, IMG_ROWS, IMG_COLS)
else:
    X_train = X_train.reshape(X_train.shape[0], IMG_ROWS, IMG_COLS, 1)
    X_val = X_val.reshape(X_val.shape[0], IMG_ROWS, IMG_COLS, 1)
    X_test = X_test.reshape(X_test.shape[0], IMG_ROWS, IMG_COLS, 1)
    input_shape = (IMG_ROWS, IMG_COLS, 1)_____no_output_____print(X_train.shape)
print(Y_train.shape)
print(X_val.shape)
print(Y_val.shape)
print(X_test.shape)
print(Y_test.shape)(48000, 28, 28, 1)
(48000, 10)
(12000, 28, 28, 1)
(12000, 10)
(10000, 28, 28, 1)
(10000, 10)

# Convolutional Neural Network
First, let's build a digit recognition model using CNN._____no_output_____
## Build CNN Model_____no_output_____

model = Sequential([
    Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=input_shape),
    Conv2D(64, (3, 3), activation='relu'),
    MaxPooling2D(pool_size=(2, 2)),
    Dropout(0.25),
    Flatten(),
    Dense(128, activation='relu'),
    Dropout(0.5),
    Dense(NUM_OF_CLASSES, activation='softmax')
])
model.summary()Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d (Conv2D)              (None, 26, 26, 32)        320
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 24, 24, 64)        18496
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 12, 12, 64)        0
_________________________________________________________________
dropout (Dropout)            (None, 12, 12, 64)        0
_________________________________________________________________
flatten (Flatten)            (None, 9216)              0
_________________________________________________________________
dense (Dense)                (None, 128)               1179776
_________________________________________________________________
dropout_1 (Dropout)          (None, 128)               0
_________________________________________________________________
dense_1 (Dense)              (None, 10)                1290
=================================================================
Total params: 1,199,882
Trainable params: 1,199,882
Non-trainable params: 0
_________________________________________________________________

Specify the optimizer for backpropagation. Note that the loss (cost) funciton is cross entropy that tends to go hand-in-hand with softmax activation function._____no_output_____

optimizer = keras.optimizers.Adadelta()
loss = keras.losses.categorical_crossentropy

model.compile(optimizer=optimizer, loss=loss, metrics=['accuracy'])_____no_output_____

## Train CNN_____no_output_____

Y_test_____no_output_____

Ready for training finally. Let's roll._____no_output_____

%%wandb
class_names = []

history = model.fit(X_train, Y_train,
                    batch_size=BATCH_SIZE,
                    epochs=EPOCHS,
                    verbose=1,
                    validation_data=(X_val, Y_val),
                    callbacks=[wandb.keras.WandbCallback(data_type="image",
                      save_model=False)])_____no_output_____

## Evaluate the model_____no_output_____

model.evaluate(X_test, Y_test)313/313 [==============================] - 1s 3ms/step - loss: 0.4183 - accuracy: 0.8892
CLASS_NAMES = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
print(CLASS_NAMES)

preds = model.predict(X_test)

preds_single = CLASS_NAMES[np.argmax(preds, axis = -1)]
actual_single = CLASS_NAMES[np.argmax(Y_test, axis = -1)][0 1 2 3 4 5 6 7 8 9]
# plot loss during training
plt.subplot(211)
plt.title('Loss')
plt.plot(history.history['loss'], label='train')
plt.legend()

# plot mse during training
plt.subplot(212)
plt.title('Accuracy')
plt.plot(history.history['accuracy'], label='train')
plt.legend()
plt.show()_____no_output_____

## Show Results_____no_output_____

n_to_show = 10
indices = np.random.choice(range(len(X_test)), n_to_show)

fig = plt.figure(figsize=(15, 3))
fig.subplots_adjust(hspace=0.4, wspace=0.4)

for i, idx in enumerate(indices):
    img = X_test[idx]
    ax = fig.add_subplot(1, n_to_show, i+1)
    ax.axis('off')
    ax.text(0.5, -0.35, 'pred = ' + str(preds_single[idx]), fontsize=10, ha='center', transform=ax.transAxes)
    ax.text(0.5, -0.7, 'act = ' + str(actual_single[idx]), fontsize=10, ha='center', transform=ax.transAxes)
    ax.imshow(np.squeeze(img))_____no_output_____

# Bayesian Neural Network_____no_output_____# Bayesian #2
_____no_output_____

# from tensorflow.keras.layers import Conv2D, Dense, Dropout, Flatten, MaxPooling2D
tfk = tf.keras

# Define prior for regularization.
prior = tfd.Independent(
    tfd.Normal(loc=tf.zeros(len(Y_train), dtype=tf.float32),
               scale=1.0),
               reinterpreted_batch_ndims=1)

regularizer = tfp.layers.KLDivergenceRegularizer(prior, weight=1/BATCH_SIZE)

# Define model instance.
model_bayes2 = Sequential([
      tfk.layers.InputLayer(input_shape=(len(X_train),), name="input"),
      tfk.layers.Dense(10, activation="relu", name="dense_1"),
      tfk.layers.Dense(
          tfp.layers.MultivariateNormalTriL.params_size(len(Y_train)),
          activation=None, name="distribution_weights"),
      tfp.layers.MultivariateNormalTriL(len(Y_train),
        activity_regularizer=regularizer, name="output")
    ],
    name="model")
# Compile model.
model_bayes2.compile(optimizer="adam", loss=neg_log_likelihood)
# Run training session.
model_bayes2.fit(X_train, epochs=EPOCHS, validation_data=X_test, verbose=False)
# Describe model.
model_bayes2.summary()
_____no_output_____

## Build BNN Model_____no_output_____

import tensorflow as tf
import tensorflow_probability as tfp
tfd = tfp.distributions

model_bayesian = Sequential([
    Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=input_shape),
    Conv2D(64, (3, 3), activation='relu'),
    MaxPooling2D(pool_size=(2, 2)),
    Dropout(0.25),
    Flatten(),
    Dense(128, activation='relu'),
    Dropout(0.5),
    tfp.layers.DenseFlipout(NUM_OF_CLASSES, activation='softmax')
])
model_bayesian.summary()Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_2 (Conv2D)            (None, 26, 26, 32)        320
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 24, 24, 64)        18496
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 12, 12, 64)        0
_________________________________________________________________
dropout_2 (Dropout)          (None, 12, 12, 64)        0
_________________________________________________________________
flatten_2 (Flatten)          (None, 9216)              0
_________________________________________________________________
dense_2 (Dense)              (None, 128)               1179776
_________________________________________________________________
dropout_3 (Dropout)          (None, 128)               0
_________________________________________________________________
dense_flipout_2 (DenseFlipou (None, 10)                2570
=================================================================
Total params: 1,201,162
Trainable params: 1,201,162
Non-trainable params: 0
_________________________________________________________________
optimizer = keras.optimizers.Adadelta()
loss = keras.losses.categorical_crossentropy

model_bayesian.compile(optimizer=optimizer, loss=loss, metrics=['accuracy'])_____no_output_____%%wandb

history = model_bayesian.fit(X_train, Y_train,
                    batch_size=BATCH_SIZE,
                    epochs=EPOCHS,
                    verbose=1,
                    validation_data=(X_val, Y_val),
                    callbacks=[wandb.keras.WandbCallback(data_type="image",
                      save_model=False)])_____no_output_____model_bayesian.evaluate(x_test, y_test)_____no_output_____
preds_bayesian = model_bayesian.predict(x_test)

preds_single_bayesian = CLASS_NAMES[np.argmax(preds_bayesian, axis = -1)]
actual_single = CLASS_NAMES[np.argmax(y_test, axis = -1)]_____no_output_____

Predict means inferencing._____no_output_____Show the training loss and accuracy._____no_output_____Show the results in UI._____no_output_____

n_to_show = 10
indices = np.random.choice(range(len(x_test)), n_to_show)

fig = plt.figure(figsize=(15, 3))
fig.subplots_adjust(hspace=0.4, wspace=0.4)

for i, idx in enumerate(indices):
    img = x_test[idx]
    ax = fig.add_subplot(1, n_to_show, i+1)
    ax.axis('off')
    ax.text(0.5, -0.35, 'pred = ' + str(preds_single_bayesian[idx]), fontsize=10, ha='center', transform=ax.transAxes)
    ax.text(0.5, -0.7, 'act = ' + str(actual_single[idx]), fontsize=10, ha='center', transform=ax.transAxes)
    ax.imshow(np.squeeze(img))_____no_output__________no_output_____

# Bayesian Neural Network
The BNN implementation is adopted from

https://www.kaggle.com/piesposito/bayesian-nerual-networks-with-tensorflow-2-0
_____no_output_____## 4.2 Implementing the Bayesian model with TFP Layers_____no_output_____

def build_bayesian_bcnn_model(input_shape):

    """
    Here we use tf.keras.Model to use our graph as a Neural Network:
    We select our input node as the net input, and the last node as our output (predict node).
    Note that our model won't be compiled, as we are usign TF2.0 and will optimize it with
    a custom @tf.function for loss and a @tf.function for train_step
    Our input parameter is just the input shape, a tuple, for the input layer
    """

    model_in = tf.keras.layers.Input(shape=input_shape)
    conv_1 = tfp.python.layers.Convolution2DFlipout(32, kernel_size=(3, 3), padding="same", strides=2)
    x = conv_1(model_in)
    x = tf.keras.layers.BatchNormalization()(x)
    x = tf.keras.layers.Activation('relu')(x)
    conv_2 = tfp.python.layers.Convolution2DFlipout(64, kernel_size=(3, 3), padding="same", strides=2)
    x = conv_2(x)
    x = tf.keras.layers.BatchNormalization()(x)
    x = tf.keras.layers.Activation('relu')(x)
    x = tf.keras.layers.Flatten()(x)
    dense_1 = tfp.python.layers.DenseFlipout(512, activation='relu')
    x = dense_1(x)
    dense_2 = tfp.python.layers.DenseFlipout(10, activation=None)
    model_out = dense_2(x)  # logits
    model = tf.keras.Model(model_in, model_out)
    return model_____no_output_____

## 4.3 Implementing the ELBO Loss function_____no_output_____

"""
this is our loss function: a sum of KL Divergence and Softmax crossentropy
We use the @tf.function annotation becuase of TF2.0, and need no placeholders
we get each loss and return its mean
"""

@tf.function
def elbo_loss(labels, logits):
    loss_en = tf.nn.softmax_cross_entropy_with_logits(labels, logits)
    loss_kl = tf.keras.losses.KLD(labels, logits)
    loss = tf.reduce_mean(tf.add(loss_en, loss_kl))
    return loss_____no_output_____

## 4.4 Implementing the train function_____no_output_____

"""
this is our train step with tf2.0, very elegant:
We do our flow of the tensors over the model recording its gradientes
Then, our gradient tape to give us a list of the gradients of each parameter in relation of the loss
we dan ask our previously instanced optimizer to apply those gradients to the variable
It is cool to see that it works even with TensorFlow probability- probabilistic layers parameters
"""
@tf.function
def train_step(images, labels):
    with tf.GradientTape() as tape:
        logits = bcnn(X_train)
        loss = elbo_loss(labels, logits)
    gradients = tape.gradient(loss, bcnn.trainable_variables)
    optimizer.apply_gradients(zip(gradients, bcnn.trainable_variables))
    return loss

def accuracy(preds, labels):
    return np.mean(np.argmax(preds, axis=1) == np.argmax(labels, axis=1))_____no_output_____

## 4.5 Wrap-up and train the BNN

We will se that our BNN takes mroe time to start converging_____no_output_____

bcnn = build_bayesian_bcnn_model(X_train.shape[1:])
optimizer = tf.keras.optimizers.Adam(lr=0.01)/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/base_layer.py:2281: UserWarning: `layer.add_variable` is deprecated and will be removed in a future version. Please use `layer.add_weight` method instead.
  warnings.warn('`layer.add_variable` is deprecated and '
assert(Y_train.shape[0] == X_train.shape[0])_____no_output_____%%wandb

import time
"""
in our train step we can see that it lasts more tha na normal CNN to converge
on the other side, we can have the confidence interval for our predictions, which are
wonderful in terms of taking sensitive predictions
"""
times = []
accs = []
val_accs = []
losses = []
val_losses = []
for i in range(20):
    tic = time.time()
    loss = train_step(X_train, Y_train)
    preds = bcnn(X_train)
    acc = accuracy(preds, Y_train)
    accs.append(acc)
    losses.append(loss)

    val_preds = bcnn(X_val)
    val_loss = elbo_loss(Y_val, val_preds)
    val_acc = accuracy(Y_val, val_preds)

    val_accs.append(val_acc)
    val_losses.append(val_loss)
    tac = time.time()
    train_time = tac-tic
    times.append(train_time)

    print("Epoch: {}: loss = {:7.3f} , accuracy = {:7.3f}, val_loss = {:7.3f}, val_acc={:7.3f} time: {:7.3f}".format(i, loss, acc, val_loss, val_acc, train_time))_____no_output_____

### Accuracy and loss evolution:

Some metrics to illustrate its evolution. We can see from the logs above that our BNN is generalizing very well on the validation set. On the other side, it lasts much more to train._____no_output_____

#plotting accuracy
plt.plot(np.array(accs), label="acc")
plt.plot(np.array(val_accs), label="val_acc")
plt.title('Model accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.show()

# Plot training & validation loss values
plt.plot(np.array(losses), label="loss")
plt.plot(np.array(val_losses), label="val_loss")
plt.title('Model loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'], loc='upper left')
plt.show()_____no_output_____

## 5. Predicting with Monte Carlo methods and epistemic uncertainity gathering_____no_output_____## 5.1 Notes on the prediction method

We will now use our trained Bayesian Neural Network to predict on labeled real values and noisy ones. Our approach will consist in sampling 30 predictions for each value, then see the prediction distribution. The neural network will be set to "I don't know" if it cant predict, with confidence, any value. To predict a value, using the montecarlo method, the median of the labels predicted probability must surpass 0.35.

We want to show that, the more the sampled networks predict on the mode prediction, the more reliable the prediction will be. On the other side, "confuse" and uncertain predictions will have the predictions distributed on more labels, approximating to a uniform distribution.

In order to to that, we are implementing functions to gather and plot those distributions.
    _____no_output_____### Plot prediction histogram function:

The function below plots the histograms for each label probabilities prediction and will be used to visualize our data. This function was mostly from Zhullingchen GitHub (with minor changes).

Here is the link of the repo, which I strongly recomend you give a look: [TFP tutorial from Zhulingchen](https://github.com/zhulingchen/tfp-tutorial/)_____no_output_____

def plot_pred_hist(y_pred, n_class, n_mc_run, n_bins=30, med_prob_thres=0.2, n_subplot_rows=2, figsize=(25, 10)):
    bins = np.logspace(-n_bins, 0, n_bins+1)
    fig, ax = plt.subplots(n_subplot_rows, n_class // n_subplot_rows + 1, figsize=figsize)
    for i in range(n_subplot_rows):
        for j in range(n_class // n_subplot_rows + 1):
            idx = i * (n_class // n_subplot_rows + 1) + j
            if idx < n_class:
                ax[i, j].hist(y_pred[idx], bins)
                ax[i, j].set_xscale('log')
                ax[i, j].set_ylim([0, n_mc_run])
                ax[i, j].title.set_text("{} (median probability: {:.2f}) ({})".format(str(idx),
                                                                               np.median(y_pred[idx]),
                                                                               str(np.median(y_pred[idx]) >= med_prob_thres)))
            else:
                ax[i, j].axis('off')
    plt.show()_____no_output_____

## 5.2 Gathering the prediction distribution

On the snippet below, we just apply the Monte Carlo technique to sample and predict labels 30 times for each image on the validation set. We then store those predictions on lists to plot the distribution and gather insights._____no_output_____

n_mc_run = 50
med_prob_thres = 0.35

y_pred_logits_list = [bcnn(X_val) for _ in range(n_mc_run)]  # a list of predicted logits
y_pred_prob_all = np.concatenate([tf.nn.softmax(y, axis=-1)[:, :, np.newaxis] for y in y_pred_logits_list], axis=-1)
y_pred = [[int(np.median(y) >= med_prob_thres) for y in y_pred_prob] for y_pred_prob in y_pred_prob_all]
y_pred = np.array(y_pred)

idx_valid = [any(y) for y in y_pred]
print('Number of recognizable samples:', sum(idx_valid))

idx_invalid = [not any(y) for y in y_pred]
print('Unrecognizable samples:', np.where(idx_invalid)[0])

print('Test accuracy on MNIST (recognizable samples):',
      sum(np.equal(np.argmax(Y_val[idx_valid], axis=-1), np.argmax(y_pred[idx_valid], axis=-1))) / len(Y_val[idx_valid]))

print('Test accuracy on MNIST (unrecognizable samples):',
      sum(np.equal(np.argmax(Y_val[idx_invalid], axis=-1), np.argmax(y_pred[idx_invalid], axis=-1))) / len(Y_val[idx_invalid]))Number of recognizable samples: 11679
Unrecognizable samples: [    9    20    58    83   106   121   184   209   223   227   237   279
   294   297   323   334   379   394   400   427   474   489   530   538
   560   562   603   721   727   854   858   909   917   918   921   935
   940   988  1051  1074  1086  1221  1240  1298  1301  1304  1401  1410
  1483  1515  1531  1563  1583  1610  1627  1659  1795  1823  1840  1847
  1857  1879  2154  2161  2308  2421  2445  2463  2480  2510  2546  2573
  2591  2616  2671  2708  2724  2730  2747  2756  2772  2808  2814  2839
  2842  2868  2910  2915  2973  2976  3157  3212  3288  3296  3306  3310
  3359  3374  3405  3414  3427  3563  3577  3722  3724  3756  3831  3836
  3914  3922  3945  3962  3965  3982  4054  4087  4126  4176  4207  4248
  4297  4404  4443  4450  4452  4494  4503  4506  4550  4564  4565  4582
  4647  4698  4700  4707  4766  4795  4798  4849  4896  5006  5377  5455
  5473  5501  5630  5642  5670  5684  5687  5698  5771  5801  5856  5874
  5961  6005  6023  6058  6121  6158  6239  6309  6317  6399  6420  6497
  6507  6510  6641  6675  6946  6959  7000  7002  7006  7056  7057  7065
  7066  7081  7143  7164  7175  7250  7272  7375  7388  7406  7412  7439
  7450  7451  7503  7507  7582  7600  7642  7685  7696  7722  7741  7811
  7817  7829  7850  8053  8073  8114  8165  8170  8227  8284  8326  8336
  8347  8389  8420  8421  8433  8438  8443  8467  8546  8643  8658  8695
  8872  8897  8902  8926  8949  8978  8991  9051  9058  9093  9105  9112
  9146  9182  9222  9297  9357  9475  9491  9536  9563  9589  9630  9650
  9734  9747  9865  9877  9884  9948 10015 10042 10109 10113 10117 10227
 10231 10247 10348 10363 10365 10382 10396 10409 10457 10458 10459 10526
 10542 10548 10575 10618 10620 10627 10658 10701 10710 10743 10772 10809
 10870 10941 10970 10998 11090 11115 11137 11214 11247 11257 11263 11290
 11298 11379 11392 11406 11414 11464 11486 11487 11501 11582 11627 11678
 11706 11728 11736 11760 11788 11918 11920 11925 11963]
Test accuracy on MNIST (recognizable samples): 0.8546964637383337
Test accuracy on MNIST (unrecognizable samples): 0.018691588785046728

## 5.3 Visualizing valid predictions
    _____no_output_____Here we just visualize some well predicted values._____no_output_____

X_val.shape_____no_output_____class_nmr = 10
plt.imshow(X_val[0, :, :, 0], cmap='gist_gray')
print("True label of the test sample {}: {}".format(0, np.argmax(Y_val[0], axis=-1)))

plot_pred_hist(y_pred_prob_all[0], class_nmr, n_mc_run, med_prob_thres=med_prob_thres)True label of the test sample 0: 3

## 5.4 Visualizing non-predicted values
    _____no_output_____On this snippet we do visualize the noisy images which the network is not able to predict on properly._____no_output_____

class_nmr = 10
invalids = 0
for idx in np.where(idx_invalid)[0]:
    plt.imshow(X_val[idx, :, :, 0], cmap='gist_gray')
    print("True label of the test sample {}: {}".format(idx, np.argmax(Y_val[idx], axis=-1)))

    plot_pred_hist(y_pred_prob_all[idx], class_nmr, n_mc_run, med_prob_thres=med_prob_thres)

    if any(y_pred[idx]):
        print("Predicted label of the test sample {}: {}".format(idx, np.argmax(y_pred[idx], axis=-1)))
    else:
        print("I don't know!")
    invalids += 1
    if invalids > 5:
        breakTrue label of the test sample 9: 9
end = time.time()
runtime = (end - start) / 60
print("This notebook ran in {:7.3f} minutes".format(runtime))_____no_output_____