Unify on NCHW and maybe support CHWN and HWCN tensor layout
N: batch size
C: Channel / convolution feature_map
H: Height
W: Width

## NCHW

NCHW is the most widespread format and is the default format in:

- CuDNN, Torch, PyTorch, Chainer

N is the first index which is the most familiar to data scientists. It however presents optimization challenges even on Nvidia side. See https://github.com/soumith/convnet-benchmarks/issues/93

## CHWN

CHWN is the format used by Neon and the (dead) pioneer cuda-convnet:
It is perfect for Winograd convolution. Main issue is that N on the right is unfamiliar. Also for RNNs it might be worse to have batch as the innermost dimension.

Some models also concatenates other the feature maps which can be done directly similar to `(C1 + C2 + C3)HWN`

- CuDNN  supports it according to doc 
![2017-10-29_10-03-54](https://user-images.githubusercontent.com/22738317/32142088-7f0c4486-bc90-11e7-9f88-824b56923200.png)
but for CuDNN v7 cudnn.h only has this:

```C
typedef enum
{
    CUDNN_TENSOR_NCHW = 0,          /* row major (wStride = 1, hStride = w) */
    CUDNN_TENSOR_NHWC = 1,          /* feature maps interleaved ( cStride = 1 )*/
    CUDNN_TENSOR_NCHW_VECT_C = 2    /* each image point is vector of element of C : the length of the vector is carried by the data type*/
} cudnnTensorFormat_t;
```

## HWCN

HWCN is the format used by Tensorflow. It is also better than NCHW for Winograd (but not as good as CHWN), it is also the best format for implementing "Memory Efficient Convolution" https://github.com/mratsim/Arraymancer/issues/131

## Format conversion

Converting between NCHW and CHWN can be done very efficiently by considering a transposition between [N, CHW] and [CHW, N]

Implementation by Neon: https://github.com/NervanaSystems/neon/commit/682dde664d0e4564dd958c9f5f20daac8bd37a9e

Implementation by NVIDIA: https://devblogs.nvidia.com/parallelforall/efficient-matrix-transpose-cuda-cc/

Paper: [Optimizing memory efficiency for DCNN on GPUs](https://people.engr.ncsu.edu/hzhou/SC-16-DNN.pdf)

![2017-10-29_10-36-13](https://user-images.githubusercontent.com/22738317/32142312-0a4a8b12-bc95-11e7-9b55-c9181aa2cbaf.png)

