Bengio’s main point seems to be that the feedback connections in our brains, while operating less frequently than back-prop in artificial neural networks, still settle on the same result. And yes, I agree that RNNs are imperfect and inaccurate — our brains definitely don’t ‘unroll’! I have a feeling that the temporal component is key; our own neurons spike frequently, eventually settling on synchony. “On Intelligence” is an excellent book about our neocortex, which points out that our brains have layers of prediction-circuits operating on every moment of input; higher layers predict when lower layers are erroneous! Also, our neuron connections are grown locally, according to some measure of correlation, which is not included in existing deep networks… so, multiple improvements are possible.
Thank you for your thoughtful critique! :)