Three ways to differentiate ReLU
John
When a function is not differentiable in the classical sense there are multiple ways to compute a generalized derivative. This post will look at three generalizations of the classical derivative, each applied to the ReLU (rectified linear unit) function. The ReLU function is a commonly used activation function for neural networks. It’s also called the […] The post Three ways to differentiate ReLU first appeared on John D. Cook .