Thank you for this article! I too believe that there is still much room for improvement in activation functions, loss functions, architectures — and machines will find better, unexpected ways of doing things, potentially more efficiently than our own brains’ architecture. Yet, human insight can still produce improvements, as well: Alternative to ReLU.