Activation - Yuipas/Nature.js GitHub Wiki

Sigmoid:

• Equation: f(x) = 1 / (1 + exp(-x))
• Derivative: f(x)' = f(x) * (1 - f(x))
• Range: (0, 1)
• Wikipedia: Sigmoid Function
• Graph

ReLU:

• Equation: f(x) = max(0, x)
• Derivative: f(x)' = if f(x) > 0: 1 else: 0
• Range: [0, Infinity)
• Wikipedia: Rectifier
• Graph (slight approach)

Tanh:

• Equation: f(x) = (1 + exp(-2*x)) / (1 - exp(-2*x))
• Derivative: f(x)' = (4 * e ** (2 * x)) / (exp(2 * x) + 1) ** 2
• Range: (-1, 1)
• Wikipedia: Hyperbolic
• Graph