Auto Topic: relu
auto_relu | topic
Coverage Score
1
Mentioned Chunks
16
Mentioned Docs
1
Required Dimensions
definitionpros_cons
Covered Dimensions
definitionpros_cons
Keywords
relu
Relations
| Source | Type | Target | W |
|---|---|---|---|
| Auto Topic: activation | CO_OCCURS | Auto Topic: relu | 14 |
| Auto Topic: relu | CO_OCCURS | Auto Topic: residual | 5 |
| Auto Topic: convolutional | CO_OCCURS | Auto Topic: relu | 5 |
Evidence Chunks
| Source | Confidence | Mentions | Snippet |
|---|---|---|---|
textbook Artificial-Intelligence-A-Modern-Approach-4th-Edition.pdf | 0.67 | 11 | ... r to disable that layer. Then the residual f disappears and Equation (22.10) simplifies to z(i) = gr(z(i−1)). Now suppose thatgr consists of ReLU activation functions and thatz(i−1) also applies a ReLU function to its inputs: z(i−1) =ReLU(in(i−1)). In that case we have z(i) = gr(z ... |
textbook Artificial-Intelligence-A-Modern-Approach-4th-Edition.pdf | 0.61 | 4 | ... the following: •The logistic or sigmoid function, which is also used in logistic regression (see page 703):Sigmoid σ(x) = 1/(1 + e−x). •The ReLU function, whose name is an abbreviation for rectified linear unit: ReLU ReLU(x) = max(0,x). •The softplus function, a smooth version of ... |
textbook Artificial-Intelligence-A-Modern-Approach-4th-Edition.pdf | 0.61 | 4 | •The ReLU function, whose name is an abbreviation for rectified linear unit: ReLU ReLU(x) = max(0,x). •The softplus function, a smooth version of the ReLU function: Softplus softplus(x) = log(1 + ex). 804 Chapter 22 Deep Learning ŷ x1 w1,3 w2,3 w3,5 w4,5 x2 w2,4 w1,4 3 4 5 + ... |
textbook Artificial-Intelligence-A-Modern-Approach-4th-Edition.pdf | 0.57 | 2 | ... ward Networks 803 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 -6 -4 -2 0 2 4 6 0 1 2 3 4 5 6 7 8 -6 -4 -2 0 2 4 6 softplus ReLU -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 -6 -4 -2 0 2 4 6 (a) (b) (c) Figure 22.2 Activation functions commonly used in deep learning systems: (a) the log ... |
textbook Artificial-Intelligence-A-Modern-Approach-4th-Edition.pdf | 0.57 | 2 | ... elps to keep the activations near the linear part of the sigmoid, avoiding the flat operating region that leads to vanishing gradients. With ReLU activation functions, weight decay seems to be beneficial, but the explanation that makes sense for sigmoids no longer applies because t ... |
textbook Artificial-Intelligence-A-Modern-Approach-4th-Edition.pdf | 0.57 | 2 | ... patterns of those, and so on. This is what convolutional neural networks do well. You should think of a layer—a con- volution followed by a ReLU activation function—as a local pattern detector (Figure 27.12). The convolution measures how much each local window of the image looks ... |
textbook Artificial-Intelligence-A-Modern-Approach-4th-Edition.pdf | 0.55 | 1 | ... r networks (roughly 1985–2010), internal nodes used sigmoid and tanh activation functions almost exclusively. From around 2010 onwards, the ReLU and softplus become more popular, partly because they are believed to avoid the problem of vanishing gradients mentioned in Section 22. ... |
textbook Artificial-Intelligence-A-Modern-Approach-4th-Edition.pdf | 0.55 | 1 | ... more than 25%. AlexNet had five convolutional layers interspersed with max-pooling layers, followed by three fully connected layers. It used ReLU activation functions and took advantage of GPUs to speed up the process of training 60 million weights. Since 2012, with improvements i ... |
textbook Artificial-Intelligence-A-Modern-Approach-4th-Edition.pdf | 0.55 | 1 | ... in contributing factors in the emergence of deep learning. Architectural improve- ments were also important, including the adoption of the ReLU activation function instead of the logistic sigmoid (Jarrett et al., 2009; Nair and Hinton, 2010; Glorot et |
textbook Artificial-Intelligence-A-Modern-Approach-4th-Edition.pdf | 0.55 | 1 | arning. Architectural improve- ments were also important, including the adoption of the ReLU activation function instead of the logistic sigmoid (Jarrett et al., 2009; Nair and Hinton, 2010; Glorot et al., 2011) and later the development of residual networks (He et al., 2016). On ... |
textbook Artificial-Intelligence-A-Modern-Approach-4th-Edition.pdf | 0.55 | 1 | ... d layers, where the same feedforward weight matrices are applied independently at each position. A nonlinear activation function, typically ReLU, is applied after the first feedforward layer. In order to address the potential vanishing gradient problem, two residual connections ar ... |
textbook Artificial-Intelligence-A-Modern-Approach-4th-Edition.pdf | 0.55 | 1 | ... ut the output of the first convolutional layer. Each location receives inputs from pixels in a window about that location. The output of the ReLU, as we have seen, forms a simple pattern detector. Now if we put a second layer on top of this, each location in the second layer recei ... |
textbook Artificial-Intelligence-A-Modern-Approach-4th-Edition.pdf | 0.55 | 1 | ... omputer vision techniques. What was the secret sauce behind the success of AlexNet? Besides the technical innova- tions (such as the use of ReLU activation units) we must give a lot of credit to big data and big computation. By big data we mean the availability of large data sets ... |
textbook Artificial-Intelligence-A-Modern-Approach-4th-Edition.pdf | 0.55 | 1 | cal innova- tions (such as the use of ReLU activation units) we must give a lot of credit to big data and big computation. By big data we mean the availability of large data sets with category labels, 1030 Chapter 27 Computer Vision such as ImageNet, which provided the training d ... |
textbook Artificial-Intelligence-A-Modern-Approach-4th-Edition.pdf | 0.55 | 1 | ... are, and a few algorithmic tricks, such as generative adversarial networks (GANs), batch normal- ization, dropout, and the rectified linear (ReLU) activation function. The future should see continued emphasis on improving deep learning for the tasks it excels at, and also extendin ... |
textbook Artificial-Intelligence-A-Modern-Approach-4th-Edition.pdf | 0.55 | 1 | ... obability model (RPM), 643 relaxed problem, 117, 118, 371 relevance, 237 relevance-based learning (RBL), 749, 750, 767 relevant action, 368 ReLU, 803 Remolina, E., 47, 1086 Remote Agent, 327, 373, 402 Remote Agent (planning agent), 47 Ren, S., 837, 1098 renaming, 305 RENDER -NOIS ... |