Fast AI Inference Engine “SoftNeuro®”

World-class AI technology created by masters of image processing

Kawaguchi, Majima, Kuroda, Nishino, Hilaga, Matsuo, Hirasaki, Ogawa, Kurihara

"SoftNeuro®" operates in multiple environments, utilizing learning results that have been obtained through a variety of deep learning frameworks.
During this roundtable, the development team discussed the development history and possible future directions.

What is the background of development and how did it come about?

Hilaga: The in-house Deep Learning inference module (ConvNet) that was developed for the Morpho Scene Classifier™ scene recognition technology wasn't reaully built for customization, so we had to rebuild it, right?
Kuroda: It is true that one of the motivations for development was that it wasn't very well-organized. Another motivation was that we developed ConvNet between 2014 and 2016, and towards the end of that period, my boss pointed out that it could be a little faster, so we did some experiments at hand and found that there was indeed a data structure that made it faster.
Hilaga: ConvNet was implemented with the motivation that it would be nice to be able to do Deep Learning on embedded devices, but there were problems with the software architecture.
Kawaguchi: Was it implemented in C at the time? And the learning side?
Kuroda: It was implemented in C. On the learning side, there were deepnet, Theano, Caffe, etc. deepnet is a minor framework that has been around since early days.
Hilaga: At the time, we didn’t quite know which learning framework to choose.
Nishion: Caffe wasn't around back then in the first place.

What are the advantages of SoftNeuro?

Hilaga: It's easy to use, it's fast and it can be used anywhere. Using this framework doesn't require you to know anything about Deep Learning.
I was aiming for a very easy-to-use inference framework similar to Keras, and I personally wanted something like that.
Another advantage of SoftNeuro is that it can import from various frameworks and run fast on various architectures.
Also, SoftNeuro is both flexible and fast because of the separation of the layer and its execution pattern, which is a concept of routine. I think this is a new feature not found in other inference engines.
Nishino: And, you can perform tuning.

Please tell us how your development was organized in the first half of the deveopment period (~2018)?

Matsuo: It was my first assinged task after my OJT.
At first, I could only load and execute VGG from memory, but by adding importers, various layers, etc, it became a library that worked properly.
There were very few inference engines at the time, but Intel's inference engine was fast. To beat them, we improved speed of SoftNeuro as a whole by applying new ideas at the algorithm and HW level. This was a lot of fun.
Kawaguchi: Where did you come up with the idea of tuning and speeding up on a per device basis?
Kuroda: Thas was Hilaga-san’s idea. Since we implemented various Convolution patterns, we also needed a mechanism to select the most appropriate one.
Nishino: We thought we had a chance to patent it if we could speed up the inference on a per device basis. And then we did, and we got the patent as well.
Kawaguchi: Was it because you had experiences in performance tuning with embedded systems and had the know-how’s?
Hilaga: Yes. It has the effect of adjusting the parallel granularity and cache size.
Kuroda: We created a smartphone demo upon request for GTC Japan 2017 showcase. We also introduced SoftNeuro in INTERNATIONAL TECHNICAL EXHIBITION ON IMAGE TECHNOLOGY AND EQUIPMENT and Embedded Vision Summit the following year. Other companies had segmentation demos at trade shows in 2018, but I believe none had it working on smartphones.
Hilaga: Kurihara-san, what kind of difficulties did you have?
Kurihara: I was the only one working on productization of the engine and tools at the time. There were no documentations... no convert tools... I had to do a lot of organizing and development.
Hilaga-san's code was easy to read and came with documentation, so working with that was not impossible despite the amount of code. But productization became harder and harder as OpenCL implementation, FP16, and quantization came along.

Please tell us about your current development team.

Majima: When I joined the team, the framework of SoftNeuro was already established, but it was still going through a transformtion phase with many variations being added. It badly needed a proper testing and seirous organizing.
Our short-term challenge is speeding up the relase cycle while keeping their quality.
Hirasaki: Unit test is now more enhanced and CI using Jenkins is also introduced.
Productization efficiency has greatly improved since. Time-consuming Dynamic code analysis can now be done in one shot.
There are more and more routines and importers, but I think we can guarantee and improve the quality of our products by enhancing our testing.
Hilaga: Beginning ver. 4, python plug-in and pytest which has direct connection to the library were introduced and we can all agree that this made things much easier.　The code is only about 200,000 lines, so I don't think it's so huge compared to Linux and such.

What are some examples of applications of SoftNeuro?

Matsuo: Our Deep Learning product portfolio has expanded between 2018-2019. For example, Segmentation, Detection, Tracking, etc.
Also, we are recently shifting our focus towards real-world applications, such asTV-related products utilizing tracking technology (e.g. Face Tracker). I have a feeling that SoftNeuro is entering a new phase. It's expanding through products.
Hilaga: It's handy when GPU is not there for use like when using a laptop.
Nishion: Also, it's used in other companies' products.
Kawaguchi: We develop both the inference engine and the importer, which is a great advantange allowing speedy implementatin cycle of both learning and inference.
Hilaga: There seems to be an advantage as well that we keep inference in mind when building the learning system.
Ogawa: From an engineer's perspective, SoftNeuro is easy to embed, and was easy to implement in Unity, etc. With the increase in the latest research in Pytorch, it will be even easier to use if we can make the conversion to ONNX and other systems smoother.
Kawaguchi: The implementation to Vieureka in OJT was also very smooth and easy for new employees. They did it in about two to three months.

What is the future direction of SoftNeuro?

Matsuo: Recently, the performance of competing products are improving, so I'd really like to focus on making SoftNeuro run faster. I'm also interested in developing underlying technologies to speed up the development process.
Kawaguchi: I want SoftNeuro to be running on all smartphones If the inference speed is overwhelming, you would use SoftNeuro. We'll need to support it to run on a variety of processors, though.
Hilaga: I'd like to be able to license the software for developers. Right now we're releasing it as a B2B product, but we'd like to try to expand into B2C. For example, I'm thinking about a framework for downloading software from the web.
Ogawa: In addition to SoftNeuro itself, improving importers is also essential for wider use of SoftNeuro. Like making execusion available without converting the model file.
Hirasaki: Since tuning is a unique feature, I'm interested in a mechanism that can be extended to dtune itself at the time of execution. Making the engine fast can easily cause degration in the result... so a system which ensures accuracy while improving the speed is something we would love to have.

SoftNeuro®

Fast AI Inference Engine

“SoftNeuro®” is one of the world’s fastest deep learning inference engines. It operates in multiple environments, utilizing learning results that have been obtained through a variety of deep learning frameworks.

View Detail

Other Featuring Technologies

Video Stabilization “MovieSolid®”

Creating a globally recognized product with the inspiration of a master craftsman

Image Stabilization “PhotoSolid®”

Image processing technology at the industry's highest level, created by daily research and development

Image Enhancement by AI Based Segmentation and Pixel Filtering “Morpho Semantic Filtering™”

AI-evolved Image processing, allowing everyone to easily take impressive quality photo

Featuring Technology