Implement orthogonal weight initialisation
Seems to be key for RNNs to learn long-term dependencies.

Foundational paper:
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
    2013, Saxe et al, https://arxiv.org/abs/1312.6120

Papers
- Neural Photo Editing with Introspective Adversarial Networks, Brock et al, https://arxiv.org/abs/1609.07093
- Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs, Jing et al (Lecun supervision), https://arxiv.org/abs/1612.05231
- On orthogonality and learning recurrent networks with long term dependencies, Vorontsov et al, https://arxiv.org/abs/1702.00071
- Recurrent Orthogonal Networks and Long-Memory Tasks, Henaff et al (Lecun supervision), https://arxiv.org/abs/1602.06662
- Regularizing RNNs by Stabilizing Activations, Krueger et al, https://arxiv.org/abs/1511.08400
- On the difficulty of training Recurrent Neural Networks, Pascanu et al (Bengio supervision), https://arxiv.org/abs/1211.5063
- Initialization Matters: Orthogonal
Predictive State Recurrent Neural Networks, Choromanski et al, https://openreview.net/pdf?id=HJJ23bW0b
