# Notebook from wangyingsm/deep-learning-with-python-cn
Path: Chapter1_What_is_deep_learning.ipynb

[目录](index.md) || [第二章：开始之前：神经网络的数学知识](Chapter2_Mathematical_blocks_of_neural_networks.ipynb) >>_____no_output_____# 第一章：什么是深度学习_____no_output_____# 1.1 人工智能、机器学习和深度学习_____no_output_____> In the past few years, Artificial Intelligence (AI) has been a subject of intense media
hype. Machine learning, deep learning, and AI come up in countless articles, often
outside of technology-minded publications. We are being promised a future of intelligent
chatbots, self-driving cars, and virtual assistants—a future sometimes painted in a grim
light, and sometimes as an utopia, where human jobs would be scarce and most economic
activity would be handled by robots or AI agents.

最近几年，人工智能（AI）已经成为了媒体集中炒作的话题。机器学习、深度学习和AI出现在数不清的文章中，且经常并不是在科技出版物当中。普通人被灌输了一个美好的未来景象，有着智能的聊天机器人，自动驾驶车辆以及虚拟助手等，这个未来有时只是看到了一丝曙光，有时又像一个乌托邦，在这个未来中人类不怎么需要工作，大部分的经济活动都会由机器人和AI代理来完成。_____no_output_____> As a future or current practitioner of machine learning, it is important to be able to
recognize the signal in the noise, to tell apart world-changing developments from what
are merely over-hyped press releases. What is at stake is our future, and it is a future in
which you have an active role to play: after reading this book, you will be part of those
who develop the AIs. So let’s tackle these questions—what has deep learning really
achieved so far? How significant is it? Where are we headed next? Should you believe
the hype?

作为机器学习未来或目前的实践者，很重要的是要在这些噪音中分辨出有用的信号，能够区分其中真正改变世界的发展和过度炒作话题的不同。未来才是我们真正需要关注的东西，在其中你会扮演一个积极的角色：通过阅读本书，你会参与到AI的发展历程当中。因此我们需要先解决下面这些问题：深度学习目前为止真正取得的进展是什么？它到底有多重要？我们下一步将朝着哪个方向发展？你是否应该相信那些炒作内容？_____no_output_____> First of all, we need to define clearly what we are talking about when we talk about
AI. What is artificial intelligence, machine learning, and deep learning? How do they
relate to each other?

首先，我们需要明确的定义是当提到AI时，我们到底在讨论什么。什么是人工智能、机器学习和深度学习？它们有什么关联？

![AI, ML & DL](imgs/f1.1.jpg)

图1.1 人工智能、机器学习和深度学习_____no_output_____### 1.1.1 人工智能

> Artificial intelligence was born in the 1950s, as a handful of pioneers from the nascent
field of computer science started asking if computers could be made to "think"—a
question whose ramifications we are still exploring today. A concise definition of the
field would be: the effort to automate intellectual tasks normally performed by humans .
As such, AI is a very general field which encompasses machine learning and deep
learning, but also includes many more approaches that do not involve any learning. Early
chess programs, for instance, only involved hard-coded rules crafted by programmers,
and did not qualify as "machine learning". In fact, for a fairly long time many experts
believed that human-level artificial intelligence could be achieved simply by having
programmers handcraft a sufficiently large set of explicit rules for manipulating
knowledge. This approach is known as "symbolic AI", and it was the dominant paradigm
in AI from the 1950s to the late 1980s. It reached its peak popularity during the "expert
systems" boom of the 1980s.

人工智能诞生于1950年代，一些计算机科学新生领域的先锋们开始思考一个问题，计算机是否能够“思考”，这是一个至今我们仍然在探索答案的问题。我们可以为这个领域作一个简洁的定义：实现那些通常由人类完成的自动化的智能任务的努力。因此，AI是一个非常广泛的领域，大大超越了机器学习和深度学习，还包含着很多其他不涉及任何学习过程的方法。例如早期的下棋对弈程序，其中仅仅包括程序员硬编码的规则，因此不能称之为“机器学习”。实际上，在很长的一段时期内，许多专家相信人类级别的人工智能可以简单的通过程序员硬编码的规则集实现，只要这个用来处理知识的规则集足够大。这种方法被称为“符号人工智能”，它是1950到1980年代占统治地位的AI范式。这个方法在1980年代随着“专家系统”的爆发达到了流行的顶峰。_____no_output_____> Although symbolic AI proved suitable to solve well-defined, logical problems, such
as playing chess, it turned out to be intractable to figure out explicit rules for solving
more complex, fuzzy problems, such as image classification, speech recognition, or
language translation. A new approach to AI arose to take its place: machine learning.

虽然符号人工智能被证明能够解决良好定义的逻辑问题，例如下棋，但它在碰到更加复杂、模糊的问题时却难以定义明确的规则，例如图像分类、语音识别或语言翻译等。所以一种新的AI方法出现并取代了它的位置：机器学习。_____no_output_____### 1.1.2 机器学习

> In Victorian England, Lady Ada Lovelace was a friend and collaborator of Charles
Babbage, the inventor of the "Analytical Engine", the first known design of a
general-purpose computer—a mechanical computer. Although visionary and far ahead of
its time, the Analytical Engine wasn’t actually meant as a general-purpose computer
when it was designed in the 1830s and 1840s, since the concept of general-purpose
computation was yet to be invented. It was merely meant as a way to use mechanical
operations to automate certain computations from the field of mathematical
analysis—hence the name "analytical engine". In 1843, Ada Lovelace remarked on the
invention:
"The Analytical Engine has no pretensions whatever to originate anything. It can do
whatever we know how to order it to perform... Its province is to assist us in making
available what we are already acquainted with."

英国维多利亚时代，埃达·洛夫莱斯伯爵夫人是查尔斯·巴贝奇的朋友和同事，她是“分析引擎”的发明者，设计了已知最早的通用计算机原型 - 一台机械式的计算机。虽然她的远见远超了时代，但是她在1830和1840年代设计的分析引擎并不是真正意义上的通用计算机，因为当时通用计算机的概念还未出现。它仅仅是一种使用机械操作自动完成数学分析领域计算的方法而已，因此得名“分析引擎”。在1843年，埃达为她的发明写道：

“分析引擎没有对任何来源进行假设。它能够完成任何我们知道如何要求它操作的任务...它擅长在我们已经熟识的领域中帮助我们。”_____no_output_____> This remark was later quoted by AI pioneer Alan Turing as "Lady Lovelace’s
objection" in his landmark 1950 paper "Computing Machinery and Intelligence", which
introduced the "Turing test" as well as key concepts that would come to shape AI. Turing
was quoting Ada Lovelace while pondering whether general-purpose computers could be
capable of learning and originality, and he came to the conclusion that they could.

这段话后来被AI先锋阿兰图灵在他1950年标志性的论文“Computing Machinery and Intelligence”中引用，称为“埃达·洛夫莱斯伯爵夫人的目标”。在这篇论文中介绍了著名的“图灵测试”，也就是用来描述AI的关键概念。图灵在思考通用计算机是否能够进行学习和创新时引用了埃达的话，然后得出了结论是这是可能的。_____no_output_____> Machine learning arises from this very question: could a computer go beyond "what
we know how to order it to perform", and actually "learn" on its own how to perform a
specified task? Could a computer surprise us? Rather than crafting data-processing rules
by hand, could it be possible to automatically learn these rules by looking at data?

机器学习诞生于这个特定的问题：计算机能够在“我们知道如何命令它操作”之外走的更远，并且真正能够自己“学习”到操作某个特定任务的技能？计算机能给我们惊喜吗？除了手工进行数据处理外，计算机是否可能在观察数据之后自动学习到这些规则呢？_____no_output_____> This question opens up the door to a new programming paradigm. In classical
programming, the paradigm of symbolic AI, humans would input rules (a program), data
to be processed according to these rules, and out would come answers. With machine
learning, humans would input data as well as the answers expected from the data, and out
would come the rules. These rules could then be applied to new data to produce original
answers.

这个问题打开了一个新的程序设计领域的大门。在经典程序设计中，如符号人工智能范式，人类会输入规则集（或者称为程序），数据根据这些规则集进行相应处理，然后输出答案。使用机器学习，人类会输入数据和相应的答案，然后输出规则集。这些输出的规则能够应用到新的数据之上并产生相应的结果。

![ML: paradigm](imgs/f1.2.jpg)

图1.2 机器学习：一种新的程序设计范式_____no_output_____> A machine learning system is "trained" rather than explicitly programmed. It is
presented with many "examples" relevant to a task, and it finds statistical structure in
these examples which eventually allows the system to come up with rules for automating
the task. For instance, if you wish to automate the task of tagging your vacation pictures,
you could present a machine learning system with many examples of pictures already
tagged by humans, and the system would learn statistical rules for associating specific
pictures to specific tags.

一个机器学习系统与其说是明确编码的，不如说是被“训练”出来的。关联某项任务的许多“例子”会被展现给它，然后它会探索这些例子之间的统计学结构，最终得到规则能够让系统自动化完成某个任务。举个例子，如果你希望将标记你假期照片的任务自动化，你可以为机器学习系统提供许多已经人工标记过的照片作为例子，然后系统会从中学习到特定照片与特定标记之间的统计学规则。_____no_output_____> Although machine learning only started to flourish in the 1990s, it has quickly
become the most popular and most successful subfield of AI, a trend driven by the
availability of faster hardware and larger datasets. Machine learning is tightly related to
mathematical statistics, but it differs from statistics in several important ways. Unlike
statistics, machine learning tends to deal with large, complex datasets (e.g. a dataset of
millions of images, each consisting of tens of thousands of pixels) for which "classical"
statistical analysis such as bayesian analysis would simply be too impractical to be
possible. As a result, machine learning, and especially deep learning, exhibits
comparatively little mathematical theory—maybe too little—and is very
engineering-oriented. It is a hands-on discipline where ideas get proven empirically much
more often than theoretically.

虽然机器学习在1990年代才开始繁荣，但是很快它就成为了AI子领域中最流行和最成功的方法，同时也受到更快的硬件和更大的数据集的驱动。机器学习与统计学紧密关联，但是它们之间也有一些重要的不同。不像统计学，机器学习倾向于处理庞大而复杂的数据集（如一个有几百万张图像组成的数据集，其中每张图像都包含数万像素点），这种情况下“经典的”统计分析例如贝叶斯分析基本上没有实践意义。因此造成的结果是，机器学习，特别是深度学习，实际上展示了相对少量的数学理论（可能非常少），它们更加倾向于工程学。有一条不成文的规则就是往往它们的结果仅能从经验上证明而不能从理论上证明。_____no_output_____### 1.1.3 从数据中学习其表示

> To define deep learning, and understand the difference between deep learning and other
machine learning approaches, first we need to get some idea of what machine learning
algorithms really do . We just stated that machine learning discovers rules to execute a
data-processing task, given examples of what is expected. So, to do machine learning, we
need three things:

> - Input data points. For instance, if the task is speech recognition, these data points could be sound files of people speaking. If the task is image tagging, they could be picture files.
- Examples of the expected output. In a speech recognition task, these could be human-generated transcripts of our sound files. In an image task, expected outputs could tags such as "dog", "cat", and so on.
- A way to measure if the algorithm is doing a good job, to measure the distance between its current output and its expected output. This is used as a feedback signal to adjust the way the algorithm works. This adjustment step is what we call "learning".

要定义深度学习和理解深度学习以及其他机器学习方法的区别。首先我们需要掌握机器学习算法的一些基本原理。前面说过机器学习可以发现执行一个数据处理任务的规则集，只需要提供例子及其对应的结果。因此我们需要下面三个方面来进行机器学习：

- 输入的数据点。例如在语音识别任务中，这些数据点可能是人们说话的语音文件。在图像标记任务中，它们可能是图像文件。
- 例子的期望输出：在语音识别任务中，可能是人类转录的语音文件内容。在图像任务中，期望的输出可能是诸如“狗”、“猫”等等的标记。
- 一个用来判断算法好坏的衡量方法，用来度量算法目前输出与期望输出之间的距离。这将被用于调整算法工作情况的反馈信号。这个调整的步骤就是我们称为“学习”的部分。_____no_output_____> A machine learning model transforms its input data into a meaningful output, a
process which is "learned" from exposure to known examples of inputs and outputs.
Therefore, the central problem in machine learning and deep learning is to meaningfully
transform data , or in other words, to learn useful "representations" of the input data at
hand, representations that get us closer to the expected output. Before we go any further:
what’s a representation? At its core, it’s a different way to look at your data—to
"represent", or "encode" your data. For instance, a color image can be encoded in the
RGB format ("red-green-blue") or in the HSV format ("hue-saturation-value"): these are
two different representations of the same data. Some tasks that may be difficult with one
representation can become easy with another. For example, the task "select all red pixels
in the image" is simpler in the RBG format, while "make the image less saturated" is
simpler in the HSV format. Machine learning models are all about finding appropriate
representations for their input data, transformations of the data that make it more
amenable to the task at hand, such as a classification task.

一个机器学习模型能够将它的输入转换为有意义的输出，也就是所谓的通过已知的输入输出样例进行“学习”的过程。因此，机器学习和深度学习的中心问题是有意义地转换数据，或者换种说法，针对输入数据学习到有用的“表现形式”，这种表现形式能更加接近我们期望的输出。在我们更加深入之前还有几个问题：什么是表现形式？从核心来说它是一种我们观察数据的不同方法，指对数据进行“表示”或者“编码”。举例来说，一张彩色图像可以被编码成为RGB格式（“红-绿-蓝”）或者HSV格式（“色调-饱和度-值”）：这是对于相同数据的两种不同的表现形式。某些任务中一种表现形式可能是困难的而在另一种表现形式中却是比较简单的，正如“减少图像饱和度”任务在HSV格式中会简单一些那样。机器学习模型的本质就是找到输入数据的正确表现形式，然后对数据进行转换令其在某种任务重更加易于实现，例如分类任务。_____no_output_____> Let’s make this concrete. Let’s consider an x axis, and y axis, and some points
represented by their coordinates in the (x, y) system: our data, as illustrated in figure 3

让我们更加具体的来介绍一下。考虑一些坐落在$(x, y)$坐标系中的一些数据点：正如下图所示，代表我们的数据

![样例数据](imgs/f1.3.jpg)

图1.3 一些样例数据_____no_output_____> As you can see we have a few white points and a few black points. Let’s say we want
to develop an algorithm that could take the coordinates (x, y) of a point, and output
whether the point considered is likely to be black or to be white. In this case:

> - The inputs are the coordinates of our points.
- The expected outputs are the colors of our points.
- A way to measure if our algorithm is doing a good job could be, for instance, the
percentage of points that are being correctly classified.

正如上图所示，我们有一些白色点和一些黑色点。我们希望开发一个算法能够接受一对坐标$(x, y)$，然后输出这个坐标点是属于白色还是黑色。在这个情况中：

- 输入是我们数据点的坐标。
- 期望的输出是我们数据点的颜色。
- 用来衡量我们算法好坏的标准可以是，例如被正确分类颜色的数据点的百分比。_____no_output_____> What we need here is a new representation of our data that cleanly separates the
white points from the black points. One transformation we could use, among many other
possibilities, would be a coordinate change, illustrated in figure 1.4.

这里我们需要一种对我们数据新的表现形式，能够清晰地区分白点和黑点。其中一种可行的转换是坐标变换，在下图1.4中展示。

![coordinate change](imgs/f1.4.jpg)

图1.4 坐标变换_____no_output_____> In this new coordinate system, the coordinates of our points can be said to be a new
"representation" of our data. And it’s a good one! With this representation, the
black/white classification problem can be expressed as a simple rule: black points are
such that x > 0 or "white points are such that x < 0". Our new representation basically
solves the classification problem.

在这个新的坐标系统中，我们数据点的坐标值可以被认为是数据的一种新的“表现形式”。而且这种形式非常棒！使用这种表现，黑白分类问题就能够表达成为一条很简单的规则：黑点是那些$x > 0$的坐标点而白点是那些$x < 0$的坐标点。这个新的表现形式基本上解决了这个分类问题。

> In this case, we defined our coordinate change by hand. But if instead we tried
systematically searching for different possible coordinate changes, and used as feedback
the percentage of points being correctly classified, then we would be doing machine
learning. "Learning", in the context of machine learning, describes an automatic search
process for better representations.

在这个例子中，我们手动完成了坐标系的变换。但是如果我们尝试系统化的寻找不同可能的坐标系变换，然后使用被正确分类的数据点所占的百分比作为反馈的话，我们就在使用机器学习方法。“学习”在机器学习语境中，描述的是一个用于寻找更好的表现形式的自动化搜索过程。

> All machine learning algorithms consist of automatically finding such
transformations that turn data into more useful representations for a given task. These
operations could sometimes be coordinate changes, as we just saw, or could be linear
projections (which may destroy information), translations, non-linear operations (such as
select all points such that x 0), etc. Machine learning algorithms are not usually very
creative in finding these transformations, they are merely searching through a predefined
set of operations, called an "hypothesis space".

所有机器学习算法都包括这样的自动寻找过程，寻找对于特定任务更加实用的数据表现形式。这些操作可能是坐标变换（正如上例所示），或者是线性投射（会丢失信息），转译，非线性操作（例如选择所有$x > 0$的数据点）等等。机器学习算法在寻找这些变换时通常来说并不具有创造性，它们仅仅是通过一系列预定义的操作来进行搜索，被称为“假设空间”。

> So that’s what machine learning is, technically: searching for useful representations
of some input data, within a pre-defined space of possibilities, using guidance from some
feedback signal. This simple idea allows for solving a remarkably broad range of
intellectual tasks, from speech recognition to autonomous car driving.

这就是机器学习的内部原理，从技术上来说：在预定义的假设空间中对一些输入数据进行搜索更有效的表现形式，通过一些反馈型号指导搜索的过程。这个简单的思想可以用来解决相当广泛的智能任务，从语音识别到自动驾驶。

> Now that you understand what we mean by learning , let’s take a look at what makes
deep learning special.

现在你理解了什么叫做学习了，然后让我们来看一下是什么令深度学习如此特别。_____no_output_____### 1.1.4 深度学习中的“深度”

> Deep learning is a specific subfield of machine learning, a new take on learning
representations from data which puts an emphasis on learning successive "layers" of
increasingly meaningful representations. The "deep" in "deep learning" is not a reference
to any kind of "deeper" understanding achieved by the approach, rather, it simply stands
for this idea of successive layers of representations—how many layers contribute to a
model of the data is called the "depth" of the model. Other appropriate names for the field
could have been "layered representations learning" or "hierarchical representations
learning". Modern deep learning often involves tens or even hundreds of successive
layers of representation—and they are all learned automatically from exposure to training
data. Meanwhile, other approaches to machine learning tend to focus on learning only
one or two layers of representation of the data. Hence they are sometimes called "shallow
learning".

深度学习是机器学习中的一个特定子领域，一种从数据中学习的新的表现形式，它强调在连续的层上进行学习并因此获得更有意义的表现形式。深度学习中的“深度”不是指的任何形式的“更深”的理解，而是仅仅代表使用连续层来表达数据的方法，在一个数据模型中由多少层组成，被称为模型的深度。深度学习其他可能的名称包括“层级表现学习”或“层次化表现学习”。现代的深度学习通常包括数十个甚至数百个连续的数据表现层 - 它们都会使用训练数据自动进行学习。而之前提到的机器学习方法只会使用一层或两层的数据表现层。因此它们有时被称为“浅学习”。

> In deep learning, these layered representations are (almost always) learned via models
called "neural networks", structured in literal layers stacked one after the other. The term
"neural network" is a reference to neurobiology, but although some of the central
concepts in deep learning were developed in part by drawing inspiration from our
understanding of the brain, deep learning models are not models of the brain. There is no
evidence that the brain implements anything like the learning mechanisms in use in
modern deep learning models. One might sometimes come across pop-science articles
proclaiming that deep learning works "like the brain", or was "modeled after the brain",
but that is simply not the case. In fact, it would be confusing and counter-productive for
new-comers to the field to think of deep learning as being in any way related to the
neurobiology. You don’t need that shroud of "just like our minds" mystique and mystery.
So you might as well forget anything you may have read so far about hypothetical links
between deep learning and biology. For our purposes, deep learning is merely a
mathematical framework for learning representations from data.

在深度学习中，这些层次化的表现层（基本上全部）是通过一种称为“神经网络”的模型进行学习的，这些表现层以层次化的方式堆叠起来。“神经网络”是神经生物学的术语，是在我们理解大脑激活机制的基础上发展而来的，然而深度学习模型不是大脑的模型。没有任何证据表明我们的大脑使用了现代深度学习模型的方式实现的机制。你有时可能会看到一些科普文章宣称深度学习“像大脑”一样工作，或者“像大脑一样建模”，这是不正确的。实际上，对于初学者来说将深度学习和神经生物学联系到一起是容易引起混淆和适得其反的。你并不需要这种“就像我们的思维”建立出来的神秘感。因此你可以放心的忘记任何你已经读到的关于深度学习和生物学之间的联系的那些假说。对我们来说深度学习仅仅是一个数学的框架，我们用它来从数据中学习到表现形式而已。

![deep neural network for digit classification](imgs/f1.5.jpg)

图1.5 手写数字分类的深度神经网络

> What do the representations learned by a deep learning algorithm look like? Let’s
look at how a 3-layer deep network transforms an image of a digit in order to recognize
what digit it is:

一个深度学习算法学习得到的表现形式是怎样的呢？让我们看一个3-层的深度神经网络将一张手写数字图像进行转换的图例：

![deep representations](imgs/f1.6.jpg)

图1.6 手写数字分类模型的深度学习层次数据表现形式_____no_output_____> As you can see, the network transforms the digit image into representations that are
increasingly different from the original image, and increasingly informative about the
final result. You can think of a deep network as a multi-stage information distillation
operation, where information goes through successive filters and comes out increasingly
"purified" (i.e. useful with regard to some task).

正如上图所示，这个网络将原始的手写数字图像转变成不同的表现形式，每一层次的变化都会增加，同时每一个层次都会更加接近最终的结果。你可以将深度神经网络想象成一个多步骤的信息蒸馏器，在其中信息沿着连续的过滤器通过最终得到“高纯度”的结果（在某些任务中非常有效的类比）。

> So that is what deep learning is, technically: a multi-stage way to learn data
representations. A simple idea—but as it turns out, very simple mechanisms, sufficiently
scaled, can end up looking like magic.

深度学习的概念在技术上来说就是：一个多阶段的学习数据表现形式的方法。这是一个非常简单的想法，但是通过将这种简单机制进行足够的扩展，就能最终获得不错的结果。_____no_output_____### 1.1.5 使用三幅图理解深度学习的原理

> At this point, you know that machine learning is about mapping inputs (e.g. images) to
targets (e.g. the label "cat"), which is done by observing many examples of input and
targets. You also know that deep neural networks do this input-to-target mapping via a
deep sequence of simple data transformations (called "layers"), and that these data
transformations are learned by exposure to examples. Now let’s take a look at how this
learning happens, concretely.

目前为止，你已经知道机器学习是将输入（例如图像）映射到输出（例如标签“猫”）的方法，其中使用了大量输入和输出的样例进行学习。你也知道了深度神经网络通过一个深的一系列简单数据转换（被称为“层”）来实现这种输入到输出的映射，同样这些数据转换也是通过样例来进行学习的。现在让我们具体的来看一下这个学习是如何发生的。

> The specification of what a layer does to its input data is stored in the layer’s
"weights", which in essence are a bunch of numbers. In technical terms, you would say
that the transformation implemented by a layer is "parametrized" by its weights. In fact,
weights are also sometimes called the "parameters" of a layer. In this context, "learning"
will mean finding a set of values for the weights of all layers in a network, such that the
network will correctly map your example inputs to their associated targets. But here’s the
thing: a deep neural network can contain tens of millions of parameters. Finding the correct value for all of them may seem like a daunting task, especially since modifying
the value of one parameter will affect the behavior of all others!

一个层对输入数据进行的转换的定义被保存在该层的“权重”(weights)当中，它们实际上就是一组数字。用技术术语来说，你可以认为一层对数据进行的转换是通过它的权重参数实现的。实际上权重有时被称为层的参数。在这个上下文中，“学习”就意味着找到网络中所有层的全部参数值，然后网络就能正确的将你的样例输入映射到它们对应的目标上。但这里有个问题：一个深度神经网络可以包含几千万个参数值。找到所有参数的正确数值看起来是一个令人生畏的任务，特别是当修改其中一个参数会影响到所有其他参数行为的时候！

![NN parametrized by weights](imgs/f1.7.jpg)

图1.7 一个使用其权重参数化了的神经网络_____no_output_____> To control something, first, you need to be able to observe it. To control the output of
a neural network, you need to be able to measure how far this output is from what you
expected. This is the job of the "loss function" of the network, also called "objective
function". The loss function takes the predictions of the network and the true target (what
you wanted the network to output), and computes a distance score, capturing how well
the network has done on this specific example.

要控制某样事物，首先你需要能够观察它。要控制一个神经网络的输出，你需要能够衡量网络的输出和期望的输出之间的距离。这是网络中“损失函数”(loss function)的工作，也可以被称为“目标函数”(objective function)。损失函数接受网络的预测和真实的目标（你期望网络的输出值），然后计算距离分值，用来衡量网络在这个特定样例上的工作质量。

![loss function](imgs/f1.8.jpg)

图1.8 使用损失函数度量网络输出结果的质量_____no_output_____> The fundamental trick in deep learning is to use this score as a feedback signal to
adjust the value of the weights by a little bit, in a direction that would lower the loss
score for the current example. This adjustment is the job of the "optimizer", which
implements what is called the "backpropagation" algorithm, the central algorithm in deep
learning. In the next chapter we will explain in more detail how backpropagation works.

深度学习最基本的技巧在于使用上述的分值作为反馈信号对权重值进行微调，调整朝着就是能够降低当前样例损失分值的方向进行。这是“优化器”(optimizer)的工作，它实现了被称之为“反向传播”(backpropagation)的算法，也是深度学习的核心算法。在下一章中我们会更加详细的介绍反向传播算法的工作原理。

![feedback signal](imgs/f1.9.jpg)

图1.9 使用损失分值作为反馈信号来调整权重值_____no_output_____> Initially, the weights of the network are assigned random values, so the network
merely implements a series of random transformations --naturally its output is very far
from what it should ideally be, and the loss score is accordingly very high. But with
every example that the network processes, the weights get adjusted just a little in the right
direction, and the loss score decreases. This is the "training loop", which, repeated a
sufficient number of times (typically tens of iterations overs thousands of examples),
yields weight values that minimize the loss function. A network with a minimal loss is
one for which the outputs are as close as they can be to the targets: a trained network.

初始化时，网络的权重被随机赋值，因此一开始网络仅仅是实现了一系列的随机转换，显然输出会与期望输出距离很远，损失分值因此会非常高。但当每次网络处理过输入样例后，权重值都会得到一个微调，调整朝着减少损失的方向。这就是所谓的“训练循环”，通常会重复一个足够多的次数（很典型的情况是在上千个样例中重复迭代几十次），获得的权重值能最小化损失函数的分值。这样获得的最小化损失网络就能实现输出尽可能接近实际的目标值：称为训练好的网络。

> Once again: a very simple mechanism, which once scaled ends up looking like magic.

重新强调一遍：一个非常简单的机制，当扩展到足够规模时就像变魔术一样。_____no_output_____### 1.1.6 深度学习目前取得的成就

> Although deep learning is a fairly old subfield of machine learning, it only rose to
prominence in the early 2010s. In the few years since, it has achieved nothing short of a
revolution in the field, with remarkable results on all perceptual problems, such as
"seeing" and "hearing"—problems which involve skills that seem very natural and
intuitive to humans but have long been elusive for machines.

虽然深度学习是机器学习中一个有历史的子领域了，但是它直到2010年代初期才繁荣起来。在几年的发展中，取得了革命性的成就，在所有认知问题中都获得了惊人的结果，例如“视觉”和“听觉”，这些对于人类来说非常自然的技能和直觉，但是原本对于机器来说都是难以认知的领域。

> In particular, deep learning has achieved the following breakthroughs, all in
historically difficult areas of machine learning:

> - Near-human level image classification.
- Near-human level speech recognition.
- Near-human level handwriting transcription.
- Improved machine translation.
- Improved text-to-speech conversion.
- Digital assistants such as Google Now or Amazon Alexa.
- Near-human level autonomous driving.
- Improved ad targeting, as used by Google, Baidu, and Bing.
- Improved search results on the web.
- Answering natural language questions.
- Superhuman Go playing.

特别是深度学习取得了下面一些突破，都是一些机器学习领域的历史性难题：

- 接近人类水平的图像分类。
- 接近人类水平的语音识别。
- 接近人类水平的手写转录。
- 改进的机器翻译。
- 改进的文字-语音转换。
- 类似Google Now或者Amazon Alexa的电子助手。
- 接近人类水平的自动驾驶。
- 改进的精准广告，已经被Google，百度和Bing应用。
- 改进的互联网搜索结果。
- 回答自然语言的提问。
- 超越人类的围棋对弈。

> In fact, we are still just exploring the full extent of what deep learning can do. We have started applying it to an even wider variety of problems outside of machine
perception and natural language understanding, such as formal reasoning. If successful,
this might herald an age where deep learning assists humans in doing science, developing
software, and more.

实际上我们仍处于探索阶段，我们还在探求深度学习能力的完整边界。我们已经开始将它应用到机器认知和自然语言理解等之外的更广泛的领域，例如形式推理。如果成功了，这将会成为深度学习辅助人类进行科学研究、软件开发等工作的先驱。_____no_output_____### 1.1.7 不要相信短期的炒作

> Although deep learning has led to remarkable achievements in recent years, expectations
for what the field will be able to achieve in the next decade tend to run much higher than
what will actually turn out to be possible. While some world-changing applications like
autonomous cars are already within reach, many more are likely to remain elusive for a
long time, such as believable dialogue systems, human-level machine translation across
arbitrary languages, and human-level natural language understanding. In particular, talk
of "human-level general intelligence" should not be taken too seriously. The risk with
high expectations for the short term is that, as technology fails to deliver, research
investment will dry up, slowing down progress for a long time.

虽然深度学习近年来已经带来了令人瞩目的成就，在接下来的十年中我们会预期在该领域能获得更高的成就和进展。虽然某些改变世界的应用已经唾手可得，比方说自动驾驶车辆，但是还存在更多的问题仍然处于停滞的状态中，包括可信对话系统，任意语种间人类水平机器翻译，人类水平自然语言理解等。特别是，有关“人类水平通用智能”的话题不应该被严肃看待。对于短期成果的过高期望风险在于，当技术不能满足期望时，研究投入将会枯竭，从而损害这个领域长期的进展。

> This has happened before. Twice in the past, AI went through a cycle of intense
optimism followed by disappointment and skepticism, and a dearth of funding as a result.
It started with symbolic AI in the 1960s. In these early days, projections about AI were
flying high. One of the best known pioneers and proponents of the symbolic AI approach
was Marvin Minsky, who claimed in 1967: "Within a generation [...] the problem of
creating 'artificial intelligence' will substantially be solved". Three years later, in 1970, he
also made a more precisely quantified prediction: "in from three to eight years we will
have a machine with the general intelligence of an average human being". In 2016, such
an achievement still appears to be far in the future, so far in fact that we have no way to
predict how long it will take, but in the 1960s and early 1970s, several experts believed it
to be right around the corner (and so do many people today). A few years later, as these
high expectations failed to materialize, researchers and government funds turned away
from the field, marking the start of the first "AI winter" (a reference to a nuclear winter,
as this was shortly after the height of the Cold War).

这种情况之前就出现过。还不止一次。AI技术走过两次得到强烈关注然后走向失望及怀疑的周期，都以最终缩减技术投入结束。第一次周期开始于1960年代，当时还是符号人工智能时期。一开始，在AI中的投入非常高。其中符号AI最著名的先锋和拥护者马文闵斯基在1967年宣称：在一代人内...创造‘人工智能’的问题会得到解决。三年后的1970年他还做出了更加精确的量化估计：“在3至8年内，我们会创造出一台具有平均人类水平的通用智能机器”。然而到了2016年，这个目标依然非常遥远，实际上遥远得我们无法估计还需要多长的时间能够实现。但是在1960年代和1970年代初期，一些专家相信答案就躲在某个角落处等待我们去发现（至今仍有许多人相信这一点）。几年后，因为这些过高的期望无法实现，研究者和政府基金都从这个领域离开了，标志着第一个“AI寒冬”的降临（这个名词参考了“核冬天”，一个冷战巅峰时期创造出来的名词）。

> It wouldn’t be the last one. In the 1980s, a new take on symbolic AI, "expert
systems", started gathering steam among large companies. A few initial success stories
triggered a wave of investment, with corporations around the world starting their own
in-house AI departments to develop expert systems. Around 1985, companies were
spending over a billion dollar a year on the technology, but by the early 1990s, these
systems had proven expensive to maintain, difficult to scale, and limited in scope, and
interest died down. Thus began the second AI winter.

但这不是最后一次。在1980年代，一种新的符号化AI“专家系统”开始吸引了很多大企业的关注。一开始少量的成功故事引发了一波投资热潮，全世界的大企业都创建了自己的AI部门来研发专家系统。1985年左右，这些企业每年花费了超过10亿美元在这个技术的研发上，但是到了1990年代初，这些系统被证明需要高昂的维护费用，难以扩展的，并且仅在某个领域范围内有效，因此投资终止了。第二个AI寒冬开始了。

> It might be that we are currently witnessing the third cycle of AI hype and
disappointment—and we are still in the phase of intense optimism. The best attitude to
adopt is to moderate our expectations for the short term, and make sure that people less
familiar with the technical side of the field still have a clear idea of what deep learning
can and cannot deliver.

我们可能正在见证着第三个AI炒作到失望的周期当中，当前我们仍然处于高度乐观的阶段。为此我们采取的最佳态度应该是降低我们短期的期望值，并且令那些不熟悉该领域技术的人们对于深度学习能够获得和不能获得的成就保持清醒的认识。_____no_output_____### 1.1.8 AI的未来

> Although we might have unrealistic short-term expectations for AI, the long-term picture
is looking bright. We are only just getting started in applying deep learning to many
important problems in which it could prove transformative, from medical diagnoses to
digital assistants. While AI research has been moving forward amazingly fast in the past
five years, in large part due to a wave of funding never seen before in the short history of
A.I, so far relatively little of this progress has made its way into the products and
processes that make up our world. Most of the research findings of deep learning are not
yet applied, or at least not applied to the full range of problems that they can solve across
all industries. Your doctor doesn’t yet use AI, your accountant doesn’t yet use AI.
Yourself, you probably don’t use AI technologies in your day-to-day life. Of course, you
can ask simple questions to your smartphone and get reasonable answers. You can get
fairly useful product recommendations on Amazon.com. You can search for "birthday"
on Google Photos and instantly find those pictures of your daughter’s birthday party from
last month. That’s a far cry from where such technologies used to stand. But such tools
are still just accessory to our daily lives. AI has yet to transition to become central to the
way we work, think and live.

虽然我们对于AI短期有着不现实的期望，但是这个领域长期的远景是光明的。我们才刚刚开始将深度学习应用到很多重要的问题之上，并且证明其应用是革命性的，从医疗诊断到数字助手。虽然AI研究在过去5年发展惊人，大方面来说由于在AI短短的历史过程中从未有过如此大量的投入，但是至今仍只是相对很少的进展被开发成为了产品或组成了我们的世界。许多深度学习的发现仍然未被应用，或者至少未被应用到它能解决问题的所有产业上。你的医生还未使用AI，你的助理还未使用AI。你自己日常生活中可能还未使用AI技术。当然你可以向你的智能手机提出简单的问题并获得合理的回答。你可以在亚马逊网站上获得相当有用的商品推荐。你可以在Google照片上搜索“生日”来查看你女儿上个月生日趴体的照片。这些技术获得了令人称叹的发展。但是它们仅仅是我们日常生活的附加服务。AI仍未成为我们工作、思考和生活的核心方式。

> Right now it may seem hard to believe that AI could have a large impact on our
world, because at this point AI is not yet widely deployed—much like it would have been
difficult to believe in the future impact of the Internet back in 1995. Back then most
people did not see how the Internet was relevant to them, how it was going to change
their lives. The same is true for deep learning and AI today. But make no mistake: AI is
coming. In a not so distant future, AI will be your assistant, even your friend; it will
answer your questions, it will help educate your kids, and it will watch over your health.
It will deliver your groceries to your door and it will drive you from point A to point B. It
will be your interface to an increasingly complex and increasingly information-intensive
world. And even more importantly, AI will help humanity as a whole move forwards, by
assisting human scientists in new breakthrough discoveries across all scientific fields,
from genomics to mathematics.

当前我们难以相信AI对我们世界造成了巨大的影响，因为AI至今还没有广泛的部署应用，正如回到1995年的时候我们难以相信互联网会对我们的未来造成如此巨大的影响一样。那时大多数人们看不到互联网将如何与他们联系，将如何改变他们的生活。这对当前的AI技术也是一样的。但是你得相信：AI时代正在到来。在一个不远的未来，AI会是你的助手，甚至你的朋友；它会回答你的问题，它会帮助教育你的孩子，它会监控你的健康状态。它会将你的行李送到门口，它会载着你从A处去往B处。它会是你接入一个不断复杂和信息爆炸世界的接口，AI会帮助人类前进一大步，通过帮助人类科学家在各种科学领域中产生突破，从基因组学到数学。

> On the road to get there, we might face a few setbacks, and maybe a new AI
winter—in much the same way that the Internet industry got overhyped in 1998-1999 and
suffered from a crash that dried up investment throughout the early 2000s. But we will
get there eventually. AI will end up being applied to nearly every process that makes up
our society and our daily lives, much like the Internet today.

在通往目标的路上，我们可能会面对一些挫折，或者一个新的AI寒冬，正如互联网工业在1998-1999年间的过度炒作造成的2000年代初期投资急剧下滑一样。但是我们最终会到达终点。AI会最终应用到几乎所有我们社会的进程和我们日常生活当中，也正如今天的互联网一样。

> Don’t believe the short-term hype, but do believe in the long-term vision. It may take
a while for AI to get deployed to its true potential—a potential the full extent of which no
one has yet dared to dream—but AI is coming, and it will transform our world in a
fantastic way.

不要相信短期的炒作，但要相信长期的愿景。让AI能真正发挥出潜力可能需要一段时间。这些潜力和边界是目前没人敢于梦见的，但是AI时代正在到来，并且它会使用一种令我们惊讶的方式改变世界。_____no_output_____## 1.2 写在深度学习之前：机器学习简史

> Deep learning has reached a level of public attention and industry investment never seen
before in the history of AI, but it isn’t the first successful form of machine learning. In
fact, it’s a safe bet to say that most of the machine learning algorithms in use in the
industry today are still not deep learning algorithms. Deep learning isn’t always the right
tool for the job—sometimes there just isn’t enough data for deep learning to be
applicable, and sometimes the problem is simply better solved by a different algorithm. If
deep learning is your first contact with machine learning, then you may find yourself in a
situation where all you have is the deep learning hammer and every machine learning
problem starts looking like a nail for this hammer. The only way not to fall into this trap
is to be familiar with other approaches and practice them when appropriate.

深度学习获得了AI领域中史无前例的社会关注和工业投资，但是它并不是机器学习中第一个成功的案例。实际上，可以很肯定的说，今天工业中应用的机器学习算法大部分都不是深度学习算法。深度学习并不一直是该任务最合适的工具，有时数据量太少无法应用深度学习，有时问题比较简单更适合另外的算法。如果深度学习是你首先接触的机器学习算法，那么你的处境就变成了你的手里只有一把深度学习的锤子，因此看到任何问题都像是一个适合这个锤子的钉子了。要想不踩这个坑，唯一的办法就是要熟悉其他的方法并且在合适的时候实践它们。

> A detailed exposure of classical machine learning approaches is outside of the scope
of this book, but we will briefly go over them and describe the historical context in which
they were developed. This will allow us to place deep learning in the broader context of
machine learning, and better understand where deep learning comes from and why it
matters.

经典机器学习方法的详细说明超越了本书的范畴，但是我们将会简略的过一下，描述一下它们产生的历史背景。这能帮助我们将深度学习放置在机器学习更宽广的背景之中，帮助我们更好的理解深度学习的由来以及它为什么管用。_____no_output_____### 1.2.1 概率模型

> Probabilistic modeling is the application of the principles of statistics to data analysis. It
was one of the earliest forms of machine learning, yet it is still widely used to this day.
One of the best-known algorithms in this category is the Naive Bayes algorithm.

概率模型是数据分析统计原理的基本应用。它是机器学习最早的形式，时至今日它仍然广泛使用。其中最著名的算法之一是朴素贝叶斯算法。

> Naive Bayes is a type of machine learning classifier based on applying the Bayes
Theorem while assuming that the features in the input data are all independent (a strong,
or "naive" assumption, which is where the name comes from). This form of data analysis
actually predates computers, and was applied by hand decades before its first computer
implementation (most likely dating back to the 1950s). The Bayes Theorem and the
foundations of statistics themselves date back to the 18th century, and these are all you
need to start using Naive Bayes classifiers.

朴素贝叶斯(Naive Bayes)是一种机器学习分类器，它基于贝叶斯定理的应用，假设输入数据中的特征都是相互独立的（这是一个非常强，或者叫“朴素”的假设，也是算法名称的由来）。这种数据分析的形式实际上早于计算机出现，在第一台计算机实现（可以回溯到1950年代）的几十年前就有手工实现了。贝叶斯定理和统计学基础本身可以回溯到18世纪，它们是你开始使用朴素贝叶斯分类器的前提知识。

> A closely related model is the Logistic Regression (logreg for short), which is
sometimes considered to be the "hello world" of modern machine learning. Don’t be
misled by its name—logreg is in fact a classification algorithm rather than a regression
algorithm. Much like Naive Bayes, logreg predates computing by a long time, yet it is
still very useful to this day, thanks to its simple and versatile nature. It is often the first
thing a data scientist will try on a dataset to get a feel for the classification task at hand.

与之紧密联系的模型是逻辑回归(Logistic Regression，简写为logreg)，通常被认为是现代机器学习中的“hello world”。别被它的名称误导了，逻辑回归实际上是一个分类器算法而不是一个回归算法。像朴素贝叶斯一样，逻辑回归早于计算机很长时间就出现了，因其简单和广泛适用性至今仍非常有用。它通常作为数据科学家在一个数据集上进行分类任务时获得感性认识的第一个算法。_____no_output_____### 1.2.2 早期神经网络

> Early iterations of neural networks have been completely supplanted by the modern
variants that we cover in these pages; however, it is helpful to be aware of how deep
learning originated. Although the core ideas of neural networks were investigated in toy
forms as early as the 1950s, the approach took decades to really get started. For a long
time, the missing piece was a lack of an efficient way to train large neural networks. This
changed in the mid-1980s, as multiple people independently rediscovered the
"backpropagation" algorithm, a way to train chains of parametric operations using
gradient descent optimization (later in the book, we will go on to precisely define these
concepts), and started applying it to neural networks.

神经网络的早期版本已经被我们在本书中介绍的其现代的变种完全取代了；然而，它对于我们理解深度学习的来源非常有帮助。虽然神经网络的核心理论早在1950年代就已经被实验性的探讨过，但是这个方法花费了数十年才真正开始发展。在长时间内，拼图少了一块关键信息，那就是缺少有效方法来训练大型神经网络。这在1980年代中期发生了改变，多个学者独立发现发表了“反向传播”算法，这是一种使用梯度下降优化方式对链式参数进行调整的方法（本书后续会精确定义这些概念），然后将其应用到了神经网络中。

> The first successful practical application of neural nets came in 1989 from Bell Labs,
when Yann LeCun combined together the earlier ideas of convolutional neural networks
and backpropagation, and applied them to the problem of handwritten digits
classification. The resulting network, dubbed "LeNet", was used by the US Post Office in
the 1990s to automate the reading of ZIP codes on mail envelopes.

第一个成功的神经网络实际应用出现在贝尔实验室1989年，那时杨立昆(Yann LeCun，译者注：法国科学家)将早期卷积神经网络理论与反向传播结合，并将它们应用到手写数字识别问题当中。获得的网络被称为“LeNet”，在1990年代开始使用在美国邮政中对信封上的邮政编码进行自动识别。_____no_output_____### 1.2.3 核方法

> As neural networks started gaining some respect among researchers in the 1990s thanks
to this first success, a new approach to machine learning rose to fame and quickly sent
neural nets back to oblivion: kernel methods.

当神经网络在1990年代开始获得研究人员的一些关注时，一个新的机器学习方法迅速声名鹊起并将神经网络放回到了遗忘的角落，这就是核方法。

> Kernel methods are a group of classification algorithms, the best known of which is
the Support Vector Machine (SVM). The modern formulation of SVM was developed by
Vapnik and Cortes in the early 1990s at Bell Labs and published in 1995, although an
older linear formulation was published by Vapnik and Chervonenkis as early as 1963.

核方法是一组分类器算法，其中最著名的是支持向量机(SVM)。现代的SVM公式是由万普尼克和柯蒂斯于1990年代早期在贝尔实验室发展出来的，并在1995年出版，虽然其一个更早的线性公式版本是在1963年就由万普尼克和泽范兰杰斯发表了。

> SVM aims at solving classification problems by finding good "decision boundaries"
(Figure 1.10) between two sets of points belonging to two different categories. A
"decision boundary" can be thought of as a line or surface separating your training data
into two spaces corresponding to two categories. To classify new data points, you just
need to check which side of the decision boundary they fall on.

SVM专注于在两组数据点从属于两个不同的类别的情况下，寻找最佳的“决定边界”来解决分类问题（图1.10）。“决定边界”可以被认为是一根线或一个面用来将你的数据点划分为两个不同的空间，从而从属于不同的类别。要划分新的数据点，你仅需要检查它们落在了决定边界的哪一边即可。

![decision boundary](imgs/f1.10.jpg)

图1.10 决定边界_____no_output_____> SVMs proceed to find these boundaries in two steps:

> - First, the data is mapped to a new high-dimensional representation where the decision
boundary can be expressed as an hyperplane (if the data is two-dimensional like in our
example, an "hyperplane" would simply be a straight line).
- Then a good decision boundary (a separation hyperplane) is computed by trying to
maximize the distance between the hyperplane and the closest data points from each
class, a step called "maximizing the margin". This allows the boundary to generalize well
to new samples outside of the training dataset.

SVM使用下面两个步骤寻找这些边界：

- 首先映射在高维度空间中的数据表现其决定边界可以表达为一个超平面（如果数据是二维的像上图的例子那样，“超平面”就退化成一条直线）。
- 然后选择一个好的决定边界（一个划分数据的超平面），通过尝试最大化这个超平面和每个分类当中距其最近的数据点之间的距离来找到这个边界。这一步被称为“最大化边距”。这可以使得这个决定边界在分类训练集之外的新数据点时泛化的很好。

> The technique of mapping data to a high-dimensional representation where a
classification problem becomes simpler may look good on paper, but in practice it is
often computationally intractable. That’s where the "kernel trick" comes in, the key idea
that kernel methods are named after. Here’s the gist of it: for finding good decision
hyperplanes in the new representation space, you don’t have to explicitly compute the
coordinates of your points in the new space, you just need to compute the distance
between pairs of points in that space, which can be done very efficiently using what is
called a "kernel function". A kernel function is a computationally tractable operation that
maps any two points in your initial space to the distance between these points in your
target representation space, completely by-passing the explicit computation of the new
representation. Kernel functions are typically crafted by hand rather than learned from
data—in the case of SVM, only the separation hyperplane is learned.

将数据映射到高维度表示形式简化分类问题的技巧在论文上看起来很好，但是实践中这些计算通常都很棘手。这就是“核技巧”出现的由来，也是核方法这个名称的由来。下面是这个方法的要领：要在新的表示空间中找到好的决定边界超平面，你并不需要明确计算出数据点在新空间的坐标值，你仅需要计算在空间中每一对数据点之间的距离即可，这可以通过一种被称为“核函数”的方式非常高效的实现。核函数可以将原始空间中每一对数据点之间的距离映射到新的空间距离上，而完全绕开了明确计算数据点在新空间坐标的棘手过程。核函数通常是手工产生而不是从数据中学习到的 - 在SVM的情况中，只有决定边界超平面是学习到的。_____no_output_____> At the time they were developed, SVMs exhibited state of the art performance on
simple classification problems, and were one of the few machine learning methods
backed by extensive theory and amenable to serious mathematical analysis, making it
well-understood and easily interpretable. Because of these useful properties, it became
extremely popular in the field for a long time.

SVM等方法在它们发明的时代，展示了其在简单分类问题上良好性能，也是为数不多的能被扩展理论和严格数学分析方法支持的机器学习方法之一，这令它能被很好的理解和解释。正因为这些有用的特点，它成为该领域长期及其流行的算法。

> However, SVM proved hard to scale to large datasets and did not provide very good
results for "perceptual" problems such as image classification. Since SVM is a "shallow"
method, applying SVM to perceptual problems requires first extracting useful
representations manually (a step called "feature engineering"), which is difficult and
brittle.

然而，SVM被证明很难被扩展到大型数据集上，也无法对“认知”问题提供很好的结果，例如图像分类。因为SVM是一个“浅”方法，将SVM应用到认知问题上首先需要手动提取出数据的有效表现形式（该步骤被称为“特征工程”），这被认为是困难和脆弱的。_____no_output_____### 1.2.4 决策树、随机森林和梯度提升机

> Decision trees are flowchart-like structures that can allow to classify input data points or
predict output values given inputs. They are easy to visualize and interpret. Decisions
trees learned from data started getting significant research interest in the 2000s, and by
2010 they were often preferred to kernel methods.

决策树是一个类似流程图的结构，可以分类输入数据或者预测输出数据。它们很容易可视化和解释。决策树在2000年代开始获得了重点的研究关注，到了2010年它们通常被认为是核方法。

![decicion tree](imgs/f1.11.jpg)

图1.11 决策树：参数是通过对数据提出的问题来学习得到的。其中的问题可能是，例如“数据中的第二个系数是否大于3.5”。_____no_output_____> In particular, the "Random Forest" algorithm introduced a robust and practical take on
decision tree learning that involves building a large number of specialized decision trees
then ensembling their outputs. Random Forests are applicable to a very wide range of
problems --you could say that they are almost always the second-best algorithm for any
shallow machine learning task. When the popular machine learning competition website
Kaggle.com got started in 2010, Random Forests quickly became a favorite on the
platform—until 2014, when Gradient Boosting Machines took over. Gradient Boosting
Machines, much like Random Forests, is a machine learning technique based on
ensembling weak prediction models, generally decision trees. It leverages "gradient
boosting", a way to improve any machine learning model by iteratively training new
models that specialize in addressing the weak points of the previous models. Applied to
decision trees, the use of the "gradient boosting" technique results in models that strictly
outperform Random Forests most of the time, while having very similar properties. It
may be one of the best, if not the best, algorithm for dealing with non-perceptual data
today. Alongside deep learning, it is one of the most commonly used technique in Kaggle
competitions.

特别是“随机森林”算法提出了一种健壮和实际的决策树方法，通过构建大量的专门设计的决策树，然后将它们的输出合并起来。随机森林可以应用在非常广泛的问题之上，你可以认为它们几乎总是任何浅机器学习任务中第二好的算法选择。当著名的机器学习竞赛网站[Kaggle.com](https://kaggle.com)在2010年上线时，随机森林迅速变成了平台上一个受欢迎的算法。直到2014年，梯度提升机才取代了它的位置。梯度提升机，很像随机森林，是一个建立在合并弱预测模型结果基础上的机器学习技巧，通常来说这些弱预测模型都是决策树。使用“梯度提升”，通过专注于重复在上一个模型的弱数据点上进行重新训练并改进生成新模型的方式来提升性能。虽然都是应用决策树并有着很接近的属性，但是使用“梯度提升”技巧获得结果在大多数时候都能领先于随机森林产生的结果。梯度提升可能是今天最好的，至少也是最好之一的，用来处理非认知任务的方法。除了深度学习外，它是在Kaggle竞赛中最常使用的技术。_____no_output_____### 1.2.5 回到神经网络

> Around 2010, while neural networks were almost completely shunned by the scientific
community at large, a number of people still working on neural networks started making
important breakthroughs: the groups of Geoffrey Hinton at the University of Toronto,
Yoshua Bengio at the University of Montreal, Yann LeCun at New York University, and
IDSIA in Switzerland.

2010年左右，当神经网络基本上被科学社区大多数人完全遗忘了的时候，一些坚持研究神经网络的科学家们开始获得了重要的突破：包括多伦多大学杰弗里·辛顿的小组，蒙特利尔大学的约书亚·本吉奥，纽约大学的杨立昆和瑞士的IDSIA实验室。

> In 2011, Dan Ciresan from IDSIA started winning academic image classification
competitions with GPU-trained deep neural networks—the first practical success of
modern deep learning. But the watershed moment came in 2012, with the entry of
Hinton’s group in the yearly large-scale image classification challenge ImageNet. The
ImageNet challenge was notoriously difficult at the time, consisting in classifying
high-resolution color images into 1000 different categories after training on 1.4 million
images. In 2011, the top-5 accuracy of the winning model, based on classical approaches to computer vision, was only 74.3%. Then in 2012, a team led by Alex Krizhevsky and
advised by Geoffrey Hinton was able to achieve a top-5 accuracy of 83.6%—a significant
breakthrough. The competition has been dominated by deep convolutional neural
networks every year since. By 2015, we had reached an accuracy of 96.4%, and the
classification task on ImageNet was considered to be a completely solved problem.

在2011年，IDSIA的Dan Ciresan开始在图像分类竞赛中胜出，他使用了GPU来训练深度神经网络，这是现代深度学习第一个成功的实践。但是分水岭出现在2012年，当年辛顿小组出现在ImageNet年度大规模图像分类竞赛中。当时ImageNet挑战被公认为非常困难的，它需要在140万幅高分辨率彩色图像中训练出能够分为1000中不同种类的模型。在2011年，获奖模型前5位使用的都是经典的计算机视觉方法，准确率仅仅达到74.3%。但到了2012年，Alex Krizhevsky带领的团队在辛顿的指导下，将前5位准确率提升到了83.6%。这是一个巨大的突破。从此这个竞赛就被深度卷积神经网络统治了。到了2015年，我们已经达到了96.4%的准确率，从而ImageNet的分类任务已经被认为是一个完全解决了的问题。_____no_output_____> Since 2012, deep convolutional neural networks ("convnets") have become the go-to
algorithm for all computer vision tasks, and generally all perceptual tasks. At major
computer vision conferences in 2015 or 2016, it had become nearly impossible to find
presentations that did not involve convnets in some form. At the same time, deep learning
has also found applications in many other types of problems, such as natural language
processing. It has come to completely replace SVMs and decision trees in a wide range of
applications. For instance, for several years, the European Organization for Nuclear
Research, CERN, used decision tree-based methods for analysis of particle data from the
ATLAS detector at the Large Hadron Collider (LHC), but they eventually switched to
Keras-based deep neural networks due to their higher performance and ease of training
on large datasets.

从2012年开始，深度卷积神经网络（简写“convnets”）已经变成了计算机视觉任务的必选算法，甚至推广到了所有认知任务中。在2015年和2016年主要的计算机视觉学术会议中，基本不可能找到哪个演示中不涉及到深度卷积神经网络的了。与此同时，深度学习也在其他类型的问题中得到了应用，例如自然语言处理。它在广泛的领域中已经完全取代了SVM和决策树。比方说，多年来欧洲核研究中心CERN使用基于决策树的方法来对ATLAS探测器采集的大型强子对撞机中的例子数据来进行分析，但是最终它们转向了使用Keras的深度神经网络上，因为它能提供更高的性能和对大规模数据集的训练。_____no_output_____### 1.2.6 深度学习到底有什么不同

> The reason why deep learning took off so quickly is primarily that it offered better
performance on many problems. But that’s not the only reason. Deep learning is also
making problem-solving much easier, because it completely automates what used to be
the most crucial step in a machine learning workflow: "feature engineering".

深度学习发展如此快的最主要原因在于它能在很多问题上提供更好的性能。但这不是唯一的原因。深度学习也使得解决问题的方法更加简单，因为它完全自动化了在机器学习当中最为关键的步骤：“特征工程”。

> Previous machine learning techniques, "shallow" learning, only involved
transforming the input data into one or two successive representation spaces, usually via
very simple transformations such as high-dimensional non-linear projections (SVM) or
decision trees. But the refined representations required by complex problems generally
cannot be attained by such techniques. As such, humans had to go to great length to make
the initial input data more amenable to processing by these methods, i.e. they had to
manually engineer good layers of representations for their data. This is what is called
"feature engineering". Deep learning, on the other hand, completely automates this step:
with deep learning, you learn all features in one pass rather than having to engineer them
yourself. This has greatly simplified machine learning workflows, often replacing very
sophisticated multi-stage pipelines with a single, simple, end-to-end deep learning model.

之前的机器学习技术，“浅”学习，仅仅涉及通过一个或两个连续的表示空间对输入数据进行转换，比方说进行高维度非线性投射（SVM）或者决策树。但是解决复杂问题通常需要的更加精细的表现形式无法通过这些技巧来获得。正因为如此，人们常常不得不花费大量精力深入的研究数据初始处理过程，令其更加适合使用某种传统机器学习方法来使用，例如人们不得不手工设计他们的数据的表示层。这个过程被称为“特征工程”。深度学习，恰恰相反，完全自动化了这个步骤：使用深度学习，你可以在一次训练中学习到所有的特征而不需要手工设计它们。这能极大简化机器学习的流程，我们能够将一个非常复杂，多个流程阶段的管道操作替换成单个简单的端到端的深度学习模型。

> You may ask, if the crux of the issue is to have multiple successive layers of
representation, could shallow methods be applied repeatedly to emulate the effects of
deep learning? In practice, there are fast-diminishing returns to successive application of
shallow learning methods, because the optimal first representation layer in a 3-layer
model is not the optimal first layer in a 1-layer or 2-layer model . What is transformative
about deep learning is that it allows a model to learn all layers of representation jointly , at
the same time, rather than in succession ("greedily", as it is called). With joint feature
learning, whenever the model adjusts one of its internal features, all other features that
depend on it will automatically adapt to the change, without requiring human intervention. Everything is supervised by a single feedback signal: every change in the
model serves the end goal. This is much more powerful than greedily stacking shallow
models, as it allows for very complex and abstract representations to be learned by
breaking them down into long series of intermediate spaces (layers), each space only a
simple transformation away from the previous one.

你可能会问，如果这个问题的症结在于使用多个连续的表现层，能否将浅方法重复应用来模拟深度学习的效果呢？在实践中，连续应用浅学习方法会有一个快速递减的效果，因为在一个3层模型中的第一个最优化的表现层不等同于在一个1层或2层模型中的第一个最优化层。深度学习的革新点在于它能让一个模型所有的层次联合起来一起进行学习，而不是继续演进（被成为贪婪化）。使用联合特征学习，每当模型调整了它其中一个内部特征是，所有其他依赖的特征都会自动适应这种变化，不需要人工干预。所有过程都处于单个反馈信号的监控之中：模型的每一个变化都服务于最终的目标。这比起贪婪的将多个浅模型堆叠在一次要强大的多，因为它允许通过将一个非常复杂和抽象的表现形式拆散成为一系列中间空间（层）来进行学习，每个空间只是前面空间的简单转换。_____no_output_____> These are the two essential characteristics of how deep learning learns from data: the
incremental, layer-by-layer way in which increasingly complex representations are
developed , and the fact these intermediate incremental representations are learned
jointly , each layer being updated both to follow the representational needs of the layer
above and the needs of the layer below. Together, these two properties have made deep
learning vastly more successful than previous approaches to machine learning.

下面是深度学习从数据中进行学习的两大关键特征：渐进式的多层次方法，能够发展出逐渐提升的复杂表现形式；以及这些中间表现形式能够联合进行学习，每一层次都会按照其前面层次和后续层次的表现需要来更新。这两个特征使得深度学习较之前的机器学习方法获得了巨大的成功。_____no_output_____### 1.2.7 现代机器学习图景

> A great way to get a sense of the current landscape of machine learning algorithms and
tools is to look at machine learning competitions on Kaggle.com . Due to its highly
competitive environment (some contests have thousands of entrants and million-dollar
prizes) and to the wide variety of machine learning problems covered, Kaggle offers a
realistic way to assess what works and what doesn’t. So, what kind of algorithm is
reliably winning competitions? What tools do top entrants use?

获得目前机器学习算法和工具发展图景的一个好办法是浏览Kaggle上的机器学习竞赛。因为它具有高度的竞争环境（一些比赛有数千参赛者以及百万美元奖金额）以及上面的比赛覆盖了非常广泛的机器学习问题，Kaggle提供了一个真实客观的方式来评判算法的好坏。因此，哪种算法能够令人信服的赢得比赛？哪种工具是顶级参赛者使用的？

> In 2016, Kaggle is dominated by two approaches: gradient boosting machines, and
deep learning. Specifically, gradient boosting is used for problems where structured data
is available, while deep learning is used for perceptual problems such as image
classification. Practitioners of the former almost always use the excellent XGB library,
which offers support for the two most popular languages of data science: Python and R.
Meanwhile, most of the Kaggle entrants leveraging deep learning use the Keras library,
due to its easy of use, flexibility and support of Python.

在2016年，Kaggle被两种方法统治着，梯度提升机和深度学习。确切来说，梯度提升被用于当比赛提供了结构化的数据的情况下，而深度学习被用于认知问题例如图像分类。前者的参赛人员几乎总是使用优秀的XGB库，两门最流行的数据科学语言都提供了它们的实现：Python和R。同时，大部分的Kaggle参赛者都使用了Keras库作为深度学习的工具，因为它易于使用，灵活性和对Python的支持。

> These are the two techniques that you should be the most familiar with in order to be
successful in applied machine learning today: gradient boosting machines (for shallow
learning problems), and deep learning (for perceptual problems). In technical terms, this
means that you will need to be familiar with XGB and Keras—the two libraries that are
currently dominating Kaggle competitions. With this book in hand, you are already one
big step closer.

熟悉以下两种技巧对于当今要在应用机器学习领域获得成功至关重要：梯度提升机（应用于浅学习问题）和深度学习（应用于认知问题）。在技术术语中，这意味着你应该熟悉XGB和Keras - 目前在Kaggle竞赛中占统治地位的两个工具。通过阅读本书，你应该能够朝着目标前进一大步。_____no_output_____## 1.3 为什么是深度学习，为什么是现在？

> The two key ideas of deep learning for computer vision, namely convolutional neural
networks and backpropagation, were already well-understood in 1989. The LSTM
algorithm, fundamental to deep learning for time series, was developed in 1997 and has
barely changed since. So why did deep learning only take off after 2012? What changed
in these two decades?

对于计算机视觉来说，深度学习的两个关键理论被称之为卷积神经网络和反向传播，在1989年就已经广为认知了。对于时间序列来说，深度学习的长短期记忆（LSTM）算法在1997年就已经被发明并且之后基本没有变化。因此为什么深度学习知道2012年才开始起航高飞呢？这20年发生了什么变化呢？

> In general, there are three technical forces that are driving advances in machine
learning:

> - Hardware.
- Datasets and benchmarks.
- Algorithmic advances.

广义来说，有三个技术力量驱动着机器学习的进展：

- 硬件。
- 数据集和基准测试。
- 算法发展。

> Because the field is guided by experimental findings rather than by theory,
algorithmic advances only become possible when appropriate data and hardware is
available to try new ideas (or just scale up old ideas, as is often the case). Machine
learning is not mathematics or physics, where major advances can be done with a pen and
a piece of paper. It is an engineering science.

因为这个领域与其说是由理论引导的，倒不如说是由实验驱动的，算法发展仅在当出现了合适的数据以及硬件的情况下才有可能发展和尝试新理论（或者只是扩展原有理论，通常都是这样的情况）。机器学习不同于数学或者物理学，其发展可以通过一支笔和一张纸就能完成。这是一门工程科学。

> So the real bottleneck throughout the 1990s and 2000s was data and hardware. But
here is what happened during that time: the Internet took, and high-performance graphics
chips were developed for the needs of the gaming market.

因此贯穿1990年代和2000年代的瓶颈实际上是数据和硬件。但其间科技发展促进了这两个方面：互联网的爆发和为了满足游戏市场需求导致高性能显卡的诞生。_____no_output_____### 1.3.1 硬件

> Between 1990 and 2010, off-the shelf CPUs have gotten faster by a factor of
approximately 5,000. As a result, nowadays it’s possible to run small deep learning
models on your laptop, whereas this would have been intractable 25 years ago.

在1990和2010年代期间，CPU的速度提升了大约5000倍。效果就是，当今能在你的笔记本电脑上运行一些小型的深度学习模型了，然而这在25年前是十分困难的。

> However, typical deep learning models used in computer vision or speech recognition
require orders of magnitude more computational power than what your laptop can
deliver. Throughout the 2000s, companies like NVIDIA and AMD have been investing
billions of dollars into developing fast, massively parallel chips (graphical processing
units, GPUs) for powering the graphics of increasingly photorealistic video games.
Cheap, single-purpose supercomputers designed to render complex 3D scenes on your
screen, in real-time. This investment came to benefit the scientific community when, in
2007, NVIDIA launched CUDA, a programming interface for its line of GPUs. A small
number of GPUs started replacing massive clusters of CPUs in a number of various
highly-parallelizable applications, starting with physics modeling. Deep neural networks,
consisting mostly of many small matrix multiplications, are also highly parallelizable,
and around 2011, some researchers started writing CUDA implementations of neural
nets—Dan Ciresan and Alex Krizhevsky were some of the first among them.

然而，在计算机视觉或者语音识别应用中典型的深度学习模型还是需要超越你的笔记本电脑数量级的计算能力。在2000年代，NVIDIA和AMD这样的公司投入了数十亿美元研发快速，海量并行芯片（图形处理单元，GPU）用来为增长的视频游戏市场提供计算能力。它们能提供便宜的、单一目标的为超级计算机设计的复杂3D渲染能力。这项投资在2007年开始让科学社区受惠，当年NVIDIA发布了CUDA，其GPU计算的一个编程接口。少量的GPU开始取代原来的大型GPU阵列在许多高并行化应用中投入使用，首先是物理建模。深度神经网络，基本上由许多小的矩阵乘法组成，实际上是高度并行化的，接着在2011年，一些研究人员开始为神经网络编写CUDA的实现 - Dan Ciresan和Alex Krizhevsky是它们中的第一批先锋。

> So what happened is that the gaming market has subsidized supercomputing for the
next generation of artificial intelligence applications. Sometimes, big things start as
games. Today, the NVIDIA Titan X, a gaming GPU that cost $1000 at the end of 2015,
can deliver a peak of 6.6 TLOPS in single-precision, i.e. 6.6 trillion of float32
operations per second. That’s about 350 times more than what you can get out of a
modern laptop. On a Titan X, it only takes a couple of days to train an ImageNet model
of the sort that would have won the competition a few years ago. Meanwhile, large
companies train deep learning models on clusters of hundreds of GPUs of a type
developed specifically for the needs of deep learning, such as the NVIDIA K80. The
sheer computational power of such clusters is something that would never have been
possible without modern GPUs.

因此这里发生的情况就是游戏市场为下一代人工智能应用的计算能力提供的前期的准备。有时候，大事物诞生于游戏。时至今日，NVIDIA的Titan X，在2015年底的价格大概是1000美元，能够提供6.6TLOPS（译者注：6.6兆浮点数运算每秒）的单精度浮点数运算。这大约等于一台现代笔记本电脑算力的350倍。在Titan X上，仅需要几天时间就能训练一个ImageNet模型，也就是几年前能够在竞赛中获胜的那种模型。同时，大公司可以在数百个专门为深度学习设计的GPU的集群上进行深度学习模型的训练，例如NVIDIA K80。这样的集群纯粹的计算能力在现代GPU出现之前是不可能的。

> What’s more, the deep learning industry is even starting to go beyond GPUs, and is
investing into increasingly specialized and efficient chips for deep learning. In 2016, at
its annual I/O convention, Google revealed its "TPU" project (tensor processing unit), a new chip design developed from the ground-up to run deep neural networks, reportedly
10x faster and far more energy-efficient than top-of-line GPUs.

还有甚者，深度学习工业甚至在GPU基础上走的更远，在专为深度学习设计和实现的芯片上投入了更多的资源。在2016年，Google在其年度I/O大会上展示了“TPU”计划（张量处理单元），这是一种新的芯片，为了运行深度神经网络从0开始专门研发，宣称可以提供顶级GPU10倍的性能并且更加节省能源。_____no_output_____### 1.3.2 数据

> Artificial Intelligence is sometimes heralded as the new industrial revolution. If deep
learning is the steam engine of this revolution, then data is its coal. The raw material that
powers our intelligent machines, without which nothing would be possible. When it
comes to data, besides the exponential progress in storage hardware over the past twenty
years, following Moore’s law, the game-changer has been the rise of the Internet, making
it feasible to collect and distribute very large datasets for machine learning. Today, large
companies work with image datasets, video datasets, and natural language datasets that
could not have been collected without the Internet. User-generated image tags on Flickr,
for instance, have been a treasure trove of data for computer vision. So were YouTube
videos. And Wikipedia is a key dataset for natural language processing.

人工智能预示着新的工业革命。如果深度学习是这次革命的蒸汽机，那么数据就是它的煤炭。它们是为智能机器提供动力的原始材料，如果没有它们什么也做不了。当提起数据，除了过去20年来符合摩尔定律指数增长的存储硬件外，互联网的腾飞也是关键因素，它使得机器学习能够方便的收集和分发非常巨大的数据集。如今，大公司使用的图像数据集、视频数据集和自然语言数据集如果没有了互联网将无法获取。用户产生的图像标签，例如Flickr，已经成为了计算机视觉数据中的宝藏。同样的还有YouTube的视频。还有维基百科是自然语言处理的一个关键的数据集。

> If there is one dataset that has been a catalyst for the rise of deep learning, it is the
ImageNet dataset, consisting in 1.4 million images hand-annotated with 1000 images
categories (one category per image). But what makes ImageNet special is not just its
large size, but also the yearly competition associated with it. As Kaggle.com as been
demonstrating since 2010, public competitions are an excellent way to motivate
researchers and engineers to push the envelope. Having common benchmarks that
researchers compete to beat has greatly helped the recent rise of deep learning.

如果要选出一个数据集作为深度学习崛起的催化剂的话，那就是ImageNet数据集，包括了140万张人工标记的图像，涵盖了1000个图像分类（一张图像属于一个分类）。但是它巨大的数据量并不是ImageNet唯一的特殊性质，还有其每年举办的竞赛。正如Kaggle.com从2010年开始展示的情况，公开竞赛是激励研究人员和工程公开技术的非常好的方式。提供基准测试让研究人员参与并超越极大的帮助了深度学习的崛起。_____no_output_____### 1.3.3 算法

> Besides hardware and data, up until the late 2000s, we were still missing a reliable way to
train very deep neural networks. As a result, neural networks were still fairly shallow,
leveraging only one or two layers of representations, and so they were not able to shine
against more refined shallow methods such as SVMs or Random Forests. The key issue
was that of "gradient propagation" through deep stacks of layers. The feedback signal
used to train neural networks would fade away as the number of layers increased.

除了硬件和数据，直到2000年代晚期，我们仍然缺少一种可靠的方法来训练非常深的神经网络。因此导致神经网络仍旧相对而言比较浅，仅仅借助一层或两层表示，因此它们无法与那些改良后的浅学习方法进行比较，例如SVM或随机森林。这里关键的问题在于“梯度传播”在深层次结构之间的传递。用来训练网络的反馈信号随着层数的增加会消失。

> This changed around 2009-2010 with the development of several simple but
important algorithmic improvements that allowed for better gradient propagation:

> - Better "activation functions" for neural layers.
- Better "weight initialization schemes". It started with layer-wise pre-training, which was quickly abandoned.
- Better "optimization schemes", such as RMSprop and Adam .

这种情况在2009-2010年间随着一些简单但是重要的算法改进发生了变化，这使得更好的梯度传播变为可能：

- 更好的神经层“激活函数”。
- 更好的“权重初始化策略”。
- 更好的“优化策略”，例如RMSprop和Adam。

> It is only when these improvements started allowing for training models with ten or
more layers that deep learning really started to shine.

有了上述改进后，才使得训练一个10个层次或以上的模型变为可能，深度学习开始真正大放异彩。

> Finally, in 2014, 2015 and 2016, even more advanced ways to help gradient
propagation were discovered, such as batch normalization, residual connections, and
depthwise separable convolutions. Today we can train from scratch models that are
thousands of layers deep.

最后在2014、2015和2016年，更多先进的梯度传播方法被发明了出来，例如批量标准化、残差连接和深度可分卷积。今天我们已经可以训练一个具有数千层次深度的模型了。_____no_output_____### 1.3.4 投资的新浪潮

> As deep learning became the new state of the art for computer vision in 2012-2013, and
eventually for all perceptual tasks, industry leaders took note. What followed was a
gradual wave of industry investment far beyond anything previously seen in the history
of AI.

随着深度学习在2012-2013年成为了了计算机视觉技术的新标准，并且最终作为所有认知任务的事实标准，工业领袖们都看到了前景。紧接着会是一波渐进式的工业投资，并且会远超AI历史中的能看到的情形。

> In 2011, right before deep learning started taking the spotlight, the total venture
capital investment in AI was around $19M, going almost entirely to practical applications
of shallow machine learning approaches. By 2014, it had risen to a staggering $394M.
Dozens of startups launched in these 3 years, trying to capitalize on the deep learning
hype. Meanwhile, large tech companies such as Google, Facebook, Baidu and Microsoft
have invested in internal research departments in amounts that would most likely dwarf
the flow of venture capital money. Only a few numbers have surfaced. In 2013, Google
acquired the deep learning startup DeepMind for a reported $500M—the largest
acquisition of an AI company in history. In 2014, Baidu started a deep learning research
center in Silicon Valley, investing $300M in the project. The deep learning hardware
startup Nervana Systems was acquired by Intel in 2016 for over $400.

在2011年，就在深度学习站在聚光灯下之前，在AI中总的风险投资大约是1900万美元，这些投资基本都流向了使用浅学习方法的实际应用。到了2014年，这个数值已经达到了惊人的3.94亿美元。几百家的初创企业在这3年中开启，期望赶上深度学习这个风口获得投资。同时，大型科技公司例如Google，Facebook，百度和微软都投资了内部的研究部门，投入的资金令那些风险投资相形见绌。其中公开的投资数据只是部分。在2013年，报道称Google花了5亿美元收购了深度学习初创公司DeepMind，这是历史上AI公司收购的最高金额。在2014年百度在硅谷成立了深度学习研发中心，计划投资3亿美元。深度学习硬件初创企业Nervana Systems在2016年被Intel以超过4亿美元收购。

> In fact, machine learning and in particular deep learning have become central to the
product strategy of these tech giants. In late 2015, Sundar Pichai, Google CEO, stated:

> "Machine learning is a core, transformative way by which we’re rethinking how
we’re doing everything. We are thoughtfully applying it across all our products, be it
search, ads, YouTube, or Play. And we’re in early days, but you will see us?-- in a
systematic way—apply machine learning in all these areas."

事实上，机器学习确切说深度学习已经变成这些科技巨头的产品策略的中心。在2015年底，Google CEO Sundar Pichai说到：

“机器学习是我们重新考虑我们如何完成所有事情一个核心、革命性的方式。我们正在思考将其应用到我们所有的产品当中，包括搜索、广告、YouTube或Play。并且我们还处于初级阶段，但你会看到我们采取一种系统化的方式，将机器学习应用到所有这些领域当中。”

> As a result of this wave of investment, the number of people working on deep
learning went in just 5 years from a few hundreds, to tens of thousands, and research
progress has reached a frenetic pace. There are currently no signs that this trend is going
to slow anytime soon.

作为这波投资的结果，5年间在深度学习领域工作的人数从数百人升至几万人，研究的进程也达到了狂热化的程度。目前没有任何信号表明这个趋势会在短期内减慢下来。_____no_output_____### 1.3.5 深度学习中的民主化

> One the key factors driving this inflow of new faces in deep learning has been the
democratization of the toolsets used in the field. In the early days, doing deep learning
required significant C++ and CUDA expertise, which few people possessed. Nowadays,
basic Python scripting skills suffice to do advanced deep learning research. This has been
driven most notably by the development of Theano and then TensorFlow, two symbolic
tensor manipulation frameworks for Python that support auto-differentiation, greatly
simplifying the implementation of new models, and by the rise of user-friendly libraries
such as Keras, which makes deep learning as easy as manipulating Lego bricks. After its
release early 2015, Keras has quickly become the go-to deep learning solution for large
numbers of new startups, grad students, and for many researchers pivoting into the field.

驱动这些新面孔加入到深度学习的一个关键因素是这个领域中的工具集的民主化。在早期，使用深度学习工作需要C++和CUDA专家，这是仅有少数人才能掌握的技能。到了今天，基础的Python脚本技能就足够进行深度学习研究。这主要依赖于Theano和之后的TensorFlow的发展，这是两个支持Python的标志性张量操作框架，还能进行自动微分，极大简化了新模型的实现，然后随着用户友好程序库如Keras的诞生，使得深度学习应用变得像搭乐高积木一样简单。从2015年初Keras发布开始，它就迅速变成深度学习初创公司、研究生和许多研究人员必备的解决方案。_____no_output_____### 1.3.6 这个趋势会持续吗？

> Is there anything special about deep neural networks that makes them the "right"
approach for companies to be investing in and for researchers to flock to? Or is deep
learning just a fashion that might not last? Will we still be using deep neural networks in
20 years?

深度神经网络有什么特别之处令它成为这些公司正确的投资方向和研究者投入的领域？或者深度学习只是一个不会持久的潮流而已？20年后我们还会继续使用深度神经网络吗？

> The short answer is yes—deep learning does have several properties that justify its
status as an AI revolution, and it is here to stay. We may not still be using neural
networks two decades from now, but whatever we use will directly inherit from modern
deep learning and its core concepts.

简单回答的话是“对的” - 深度学习确实有一些特性能证明它是AI的革命，它会持续。我们也许在20年后不会使用神经网络了，但是我们依然会使用一种直接从现代深度学习和它的核心概念继承得到的新方法。

> These important properties can be broadly sorted into 3 categories:

> - Simplicity. Deep learning removes the need for feature engineering, replacing complex,
brittle and engineering-heavy pipelines with simple end-to-end trainable models typically
built using only 5 or 6 different tensor operations.
- Scalability. Deep learning is highly amenable to parallelization on GPUs or TPUs,
making it capable of taking full advantage of Moore’s law. Besides, deep learning models
are trained by iterating over small batches of data, allowing them to be trained on datasets
of arbitrary size (the only bottleneck being the amount of parallel computational power
available, which thanks to Moore’s law is a fast-moving barrier).
- Versatility and reusability. Contrarily to many prior machine learning approaches, deep
learning models can be trained on additional data without restarting from scratch, making
them viable for continuous online learning, an important property for very large
production models. Furthermore, trained deep learning models are repurposable and thus
reusable: for instance it is possible to take a deep learning model trained for image
classification and drop it into a video processing pipeline. This allows us to reinvest
previous work into increasingly complex and powerful models. This also makes deep
learning applicable to fairly small datasets.

这些重要的特征可以被粗略的分成三个类别：

- 简易性。深度学习不需要使用特征工程，将复杂的、棘手的和工程繁重的管道取代为简单的端对端的可训练模型，底层仅仅使用5或6个不同的张量操作进行构建。
- 高扩展性。深度学习高度适合在GPU或TPU上面进行并行化计算，令其能完全获得摩尔定律带来的优势。此外，深度学习模型通过在小批量数据上进行迭代训练，令其可以用来训练任意形状尺寸的数据集（这里唯一瓶颈就是可用的并行计算能力，而因为摩尔定律，我们可以确信可以很快不是障碍）。
- 高适应性和可重用。与前面说到的很多机器学习方法不同，深度学习模型可以用来在额外数据上进行训练而无需从头开始，令其特别适合进行持续的在线学习，这对于巨大的生产模型来说是非常重要的一个特性。并且训练好的深度学习模型是可重用的：例如可以将一个训练好的图像分类模型放入到一个视频处理管理中。这使得我们能重复利用之前已经训练好的越来越复杂模型，让它们变得更加强大。这同样也令深度学习适合于相对较小的数据集。

> Deep learning has only been in the spotlight for a few years, and we haven’t yet
established the full scope of what it can do. Every passing month we still come up with
new use cases, or with engineering improvements lifting previously known limitations.
Following a scientific revolution, progress generally follows a sigmoid curve: it starts
with a period of fast progress than gradually stabilizes, as researchers start hitting against
hard limitations and further improvements become more incremental. With deep learning
in 2017, it seems that we are still in the first half of that sigmoid, and there is a lot more
progress yet to come in the next few years.

深度学习目前仅仅站在舞台中央几年时间，我们还未探索完它的全部能力边界。每一个月我们都能发现新的应用场景，或者将原来的极限在工程学上进行了提升。根据科学革命的经验，技术的进展通常符合一条sigmoid曲线：它一开始会有一个快速的上升期，然后会逐渐稳定下来，因为研究者开始碰到了硬限制，更多的改进变得困难。深度学习领域在2017年，应该仍处于这条sigmoid曲线的前半部分，所以未来几年中还有很多的新进展会产生。_____no_output_____[目录](index.md) || [第二章：开始之前：神经网络的数学知识](Chapter2_Mathematical_blocks_of_neural_networks.ipynb) >>_____no_output_____