CUDA - Custom memory management to avoid expensive alloc/dealloc
Memory allocations and release will probably become a bottleneck during the forward and backward propagation.

During the forward pass it will hold inputs tensor in cache. During backward pass it will create intermediate gradient tensors that will be consumed during gradient descent.

As evidenced by this [article](https://www.cs.virginia.edu/~mwb7w/cuda_support/memory_management_overhead.html) or by [this](https://devtalk.nvidia.com/default/topic/878664/custom-memory-allocator-for-cuda-desired/?offset=3), [this](https://devtalk.nvidia.com/default/topic/924453/-multiple-gpus-processes-cuda-memory-de-allocation-slow/) or https://devtalk.nvidia.com/default/topic/1011686/need-a-better-buffer-management-pool-for-improving-performance/ cuda forum threads, memory management is often an issue.

## Challenges:
- Avoid expensive calls. Replacing cudaMalloc / cudaFree by evenmore expensive calls is not an option.
- Memory is allocated and released very quickly.
- Allocations are very large as we often allocate the max available GPU memory per batch.
- Just reusing a tensor would work if input are the same size. This is often the case for images but completely impractical for text. Introducing padding on the user end would mean we should just use a static graph framework ;).
- It is not nice to outright reserve 70~90% of the GPU memory to run MNIST, what if people are actually using a screen?

## Other frameworks:
- Torch as [THCCachingAllocator](https://github.com/pytorch/pytorch/pull/57/files), aka Memory Pool
- Mxnet uses a memory pool
- Theano uses CNMem which just outright reserve a percentage (70% iirc) of GPU memory aka region-based memory management
- Tensorflow reserves everything
- Chainer uses a memory pool

## Overview of the field
- [GPU allocation](http://www.highperformancegraphics.org/2014/wp-content/uploads/sites/3/2014/07/Vinkler-Allocator.pdf) with benchmarks

## Dedicated GPU memory management technique
- [Baidu p15](https://arxiv.org/pdf/1512.02595v1.pdf) for speech DL obtained 2x improvements with a custom allocator using the buddy algorithm and inspired by jemalloc.
- [Buddy algorithm slides](http://iwcse.phys.ntu.edu.tw/parallel/Oct17/Jon-Yu_Lee_131017.pptx) by Nvidia.
- [CPU implementation](https://github.com/cloudwu/buddy) in C
- [Cuda caching allocator](https://github.com/kaldi-asr/kaldi/blob/master/src/cudamatrix/cu-allocator.h)
- Cuda allocator for massively parallel systems: [mallocMC](https://github.com/ComputationalRadiationPhysics/mallocMC)
- [CMalloc](http://decibel.fi.muni.cz/~xvinkl/CMalloc/), potentially the fastest most flexible Cuda allocator

## In other fields
- Memory pool, Arena, region-based
- [slab allocator](https://en.wikipedia.org/wiki/Slab_allocation) (used by Linux Kernel) to avoid fragmentation