Review: Neuromorphic Computing at Scale

The authors discuss several key features necessary for neuromorphic systems at scale (distributed hierarchy, sparsity, neuronal scalability, etc.). Which of these features do you believe presents the most significant research challenge, and why? How might overcoming this challenge transform the field?

I think the ultimate challenge from the list of features isn't exactly any of the features presented but a more philosophical one, namely "How closely do we want neuromorphic computing at scale to resemble the human brain?" The features listed were primarily biological, and ultimately the question needs to be asked whether NMC at scale should do its best to resemble actual physical or structural patterns of the brain or if SNNs implemented on memristor crossbars at scale provide enough of an advantage in implementing existing solutions of ANNs. Whatever is most efficient will scale up as long as it's solving real problems. Do we care how much it looks like the brain? As the authors stated, we have limited understanding of the human brain, and this may not be as beneficial to pursue as it would be to implement power savings for comparable performance for the sake of the planet.

The article compares neuromorphic computing's development to the evolution of deep learning, suggesting it awaits its own "AlexNet moment." What specific technological or algorithmic breakthrough might trigger such a moment for neuromorphic computing? What applications would become feasible with such a breakthrough?

AlexNet was a performance breakthrough on classification of ImageNet using just two GPUs. To draw an analogy here, we must think of an industry problem that needs solving and a way to do it with a limited neuromorphic computing system. In thinking about this, I realized the problem likely wasn't a technical one but a more practical one.

Deep learning is incredible and has made huge improvements in virtually all regression/classification/and generative problems. If we continue to scale this up, I think the breakthroughs will continue. The scaling up is happening at such a degree to this point that dedicated nuclear power plants are required to power those breakthroughs. What if there was identical performance for a fraction of the power consumption? I think if SNNs on NMC systems prove to be orders of magnitude less power hungry with comparable performance, we begin to see an explosion in scale like that of the GPUs used for AlexNet just for the sake of pure cost savings. Neuromorphic computing doesn't have to solve any more problems; it just has to do it cheaper.

The authors highlight the gap between hardware implementation and software frameworks in neuromorphic computing compared to traditional deep learning. Develop a proposal for addressing this gap, specifically focusing on how to create interoperability between different neuromorphic platforms.

NVIDIA has seemed to have cracked this already with CUDA, but to my understanding it was developed mostly in-house by a company that virtually had a monopoly on the hardware. When this much is on the line (NVIDIA's software/hardware is now worth more than a trillion dollars), it's hard to imagine an industry standard from multiple countries for developing a cross-NMC platform for multiple chips. I think the best way to address this is a free market approach with each company developing their own software framework and the best one being more widely adopted and scaled up.

In my mind, that's done with a surplus of teaching examples on a specific platform. People tend to stick with what they originally learned on. If tons of examples have been absorbed by LLMs and that specific platform is spit out during prompts about neuromorphic computing, that's what people will use. Examples (and making sure those examples are the dominant ones seen by LLMs) are everything for adoption.

It's also important to hide the math from people. The authors recognized the importance of hiding the backprop from people in Torch and Keras. Users will immediately go back to the mathless version on GPUs and high-level abstraction if they have to do much math beyond figuring out tensor sizes (which are difficult in their own right) and multiplication/division/subtraction/addition. The less math the better, and if users can just hand their inputs and outputs (for supervised problems) and say "learn," the more successful a platform will be.

The review emphasizes the importance of benchmarks for neuromorphic systems. What unique metrics would you propose for evaluating neuromorphic systems that go beyond traditional performance measures like accuracy or throughput? How would you standardize these across diverse neuromorphic architectures?

As stated, it's challenging to provide a list of benchmarks for neuromorphic systems. The areas where neuromorphic computing excels are "lifelong or continual learning, learning in the presence of sparse data, robustness in the presence of noise and variability, and ultralow-power machine learning with real-time sensor data." Are these difficult to benchmark though? We've seen metrics for all of these in class specifically: power, the "forgetfulness" of a network, and prediction accuracy. What's preventing these from being quantified by benchmark algorithms?

The beginning of the article emphasized the adherence to biological functions. I think it would also be interesting to standardize a sort of "braininess" metric specifically: how close does this neuromorphic system resemble the human brain (or a beefed-up version of this)? It would take a lot smarter of a person than me to develop, however.

How might the convergence of emerging memory technologies (like memristors or phase-change memory) with neuromorphic principles lead to new computational capabilities not possible with traditional von Neumann architectures? What specific research directions seem most promising?

Avoiding von Neumann architectures gives you inherent power and speed advantages. What I found interesting (and how great designers end up using their materials) was that memristors can use their inherent "flaws" to an overall advantage in motion detection and online learning. Another example of this would be a great designer who will use two different resistors in series with opposite temperature coefficients to provide a stable resistor across temperature. The most promising direction will be whatever technology's inherent properties (flaws) can be harnessed for interesting performance enhancements.