Sony shows off first combination image sensor and AI chip – ProWellTech
Sony has developed an interesting new hybrid technology: an image sensor with AI processing system integrated into the hardware, which makes it a single integrated system. The benefits and applications for this are potentially enormous as images and code continue to merge.
The idea is quite simple in concept. Take a traditional CMOS image sensor like the one you would find in any phone or camera today and stack it on a logic chip built not only to extract pixels from the sensor but to operate a machine learning model that extracts information from those pixels.
The result is a single electronic assembly that can perform a lot of interesting processing on a photo before that photo is ever sent elsewhere, such as a main logic board, a GPU or the cloud.
To be clear, image sensors already have associated processors that do the usual job of sorting pixels, compressing them into a JPEG and so on. But they are very focused on performing a handful of common tasks very quickly.
The Sony chip, as the company explains, is capable of producing more sophisticated processes and outputs. For example, if the exposure is of a dog in a field, the chip could immediately analyze it for objects and, instead of sending the full image, it simply reports “dog”, “grass” and anything else that is recognized.
He could also perform essentially improvisation changes, such as cutting out everything in the photo but the parts he recognizes and has been told to report – only the flowers, but never the stems, for example.
The advantage of such a system is that it can discard all types of unnecessary or unwanted data before it ever enters the storage or processing pipeline of the main device. This means that less processor power is used first, but it can also be safer and safer.
The cameras in public places may previously obscure faces or plaques. Smart home devices are able to recognize people without ever saving or sending image data. Multiple exposures could be combined to form heat or frequency maps of the camera’s field of view.
You might expect more power drawn or latency from a chip with integrated artificial intelligence processes, but companies like Xnor (recently acquired by Apple) have shown that such activities can be done very quickly and at extremely low cost.
While more complex processing would still be the area of application for larger and more powerful chips, this first pass type is capable of producing a wide variety of valuable data and, properly designed, could prove more robust against attack or abuse.
At the moment, Sony’s “Intelligent Vision Sensor” is still only a prototype, available for order for testing but not for production. But since Sony is one of the leading image sensor suppliers in the world, this is likely to be found in different devices in one form or another.