![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
- Joined
- Dec 19, 2005
- Messages
- 18,713
Impressed?
"Although a large body of efficient techniques to train DNNs with reduced digital precision has been reported32,33, it is unlikely that such procedures can generally be applied as-is to analog in-memory computing hardware due to the random nature of δGlijδGijl. Since quantization errors coming from rounding to reduced fixed-point precision are not random, DNNs trained in this way are not a priori expected to be suitable for deployment on analog in-memory computing hardware. Techniques that inject random Gaussian noise during training are a much more natural fit to make the network robust to errors from analog in-memory computing hardware. As early as in 1994, it was shown that injecting noise on the synaptic weights during training enhances the tolerance to weight perturbations of multi-layer perceptrons, and the application of this technique to analog neural hardware was discussed34. Recent works have also proposed to apply noise to the layer inputs or pre-activations in order to improve the network tolerance to hardware noise20,23. In this work, we follow the original approach of Murray et al.34 of injecting Gaussian noise to the synaptic weights during training. Next, we discuss different techniques that we introduced together with synaptic weight noise in order to improve the accuracy of inference on ResNet and achieve close to software-equivalent accuracy after transferring the weights to PCM hardware."
https://www.nature.com/articles/s41467-020-16108-9
"Although a large body of efficient techniques to train DNNs with reduced digital precision has been reported32,33, it is unlikely that such procedures can generally be applied as-is to analog in-memory computing hardware due to the random nature of δGlijδGijl. Since quantization errors coming from rounding to reduced fixed-point precision are not random, DNNs trained in this way are not a priori expected to be suitable for deployment on analog in-memory computing hardware. Techniques that inject random Gaussian noise during training are a much more natural fit to make the network robust to errors from analog in-memory computing hardware. As early as in 1994, it was shown that injecting noise on the synaptic weights during training enhances the tolerance to weight perturbations of multi-layer perceptrons, and the application of this technique to analog neural hardware was discussed34. Recent works have also proposed to apply noise to the layer inputs or pre-activations in order to improve the network tolerance to hardware noise20,23. In this work, we follow the original approach of Murray et al.34 of injecting Gaussian noise to the synaptic weights during training. Next, we discuss different techniques that we introduced together with synaptic weight noise in order to improve the accuracy of inference on ResNet and achieve close to software-equivalent accuracy after transferring the weights to PCM hardware."
https://www.nature.com/articles/s41467-020-16108-9
