With the rise of artificial intelligence (AI), medical devices and instruments behave more intelligently by perceiving, analyzing, and judging specific situations. AI aims for software to perform more intelligently, almost human-like, resulting in an increased hardware autonomy. Computer vision is a related and specific form of AI. Computer vision systems attempt to provide active decision making based on data delivered by optical sensors, much like the human visual perception.
Download Tech Guide about multispectral imaging.
Watch Webinar about multispectral imaging
Within medical imaging, artificial intelligence is penetrating rapidly within the field of radiology and full-heart motion tracking but there are other areas of applied medicine where optical sensors with visible and/or near-infrared sensitivity assist clinicians during patient monitoring, diagnosis, digital pathology, optical coherence tomography, surgery and many more.
While many machine learning methods have been developed for different purposes, they all have the same goal of helping machines to improve and learn towards a specific goal. The most popular and prominent machine learning method applied in computer vision artificial intelligence is deep learning, which can find structures and patterns in data that are even beyond the scope of medical specialists.
A huge amount of data is required for deep learning algorithms to work well. In most medical imaging applications, a lot of image data is generated, which is good enough to train very sophisticated algorithms for specific problem-solving tasks. Such algorithms can be trained for real time non-invasive applications.
Some examples of applications that can benefit by applying deep learning to medical image data are the imaging of human tissue and the propagation of light within tissues. The latter is a significant problem in medical imaging when attempting to develop new diagnostic methods.
Biological tissues are heterogenous in composition with several spatial variations in optical properties. Light entering biological tissues undergoes various degrees of scattering. The wavelengths that a tissue may absorb depend on its molecular composition. The absorption of light both within the spectral therapeutic window and outside of it serves as a spectral fingerprint of a specific tissue layer.
Multispectral cameras can be used to capture the overlaid and complex structures of tissues so that deep learning methods can be applied. This enables a more efficient and intelligent pattern matching approach to identify the correct tissue material and to test the robustness of an artificial intelligence decision making process.
Another promising application of AI and computer vision is surgical guidance. This can involve multispectral imaging or single-sensor cameras. Today's surgeon is usually supported by a combination of different optical systems, such as microscopes and cameras. When images of the operational field are fed into an AI engine, specific areas of the image such as blood vessels or tumours can be automatically highlighted.
By using the right optics and tracer substances (such as ICGs), multispectral cameras specially designed for surgical guidance work, as well as AI-driven conventional RGB cameras, can work efficiently in fluorescence-assisted surgeries by pointing out the targeted tissues.
Cameras used in surgery must deliver high frame rates as well as high-quality images with good contrast and color rendering at the right resolution. At a minimum, these cameras must support at least HD resolutions of 1280 x 720 pixels at 30 frames per second (fps) or higher.
More common are full HD resolutions (1920 x 1080) at 60 fps, providing the surgeon with images comparable to a standard flat-panel television at home. Some surgical bays have adopted 4K display resolutions for even greater clarity of small details. Camera resolutions with a horizontal resolution of at least 3840 to 4096 pixels are used for generating an image to be displayed on the 4K screens.
Several different camera resolutions, including 8 MP, 12 MP, 20 MP, and 25 MP can achieve this level of display with many able to achieve a 60 fps rate, providing smooth video sampling that is much higher than the minimum visual perception frequency of the human eye.
There may come a time in the near future where some live surgical displays offer 8K resolution at 60 fps. This will require cameras with 45 MP resolution or higher and with very high-speed interfaces to maintain the frame rates. But even with today's full HD and 4K options, high resolution real-time video systems mean huge amounts of data that must be transferred to the processing and display units.
In the case of image data going through an AI engine, depending on the intelligence process involved, the image data can be reduced, modified, selected and transformed to suit the application. In any case, raw data can be transferred uninterrupted over relatively long distances using high-speed interfaces such as 10GigE Vision (using copper or fiber-optic cables) or CoaXPress over coaxial cables. Similar approaches are relevant in the case of digital pathology, slide scanning, and computer-aided pathology. Digital WSI (whole slide imaging) involves high speed machine vision cameras combined with artificial intelligence for detection, segmentation, diagnosis and analysis of the digitized images.
While early imaging systems in medicine mostly used RGB sensors, today we are seeing many more systems utilizing multispectral cameras. These systems are not limited to the visible spectrum. They can capture information from multiple discretely positioned spectral bands, both inside and outside the visible region. This can reveal properties not seen in the visible spectrum and can be used with various NIR fluorescence techniques to visibly "tag" or highlight tumors, blood vessels, or other features of interest to the surgeon.
Multispectral imaging combines the worlds of spectroscopy and machine vision to create new imaging opportunities. Adding AI to multispectral imaging systems to extract and interpret image data beyond the visible spectrum is a powerful tool. New prism-based multispectral cameras now offer the flexibility to customize the size and location of multispectral bands. This simplifies the creation of different spectral configurations to support specific applications.
Whether it is RGB, monochrome or multispectral imaging systems, when coupled with an artificial intelligence engine, they can observe and evaluate situations beyond human perception and thereby extend and enhance a clinician’s abilities. The AI engine can be a part of the machine vision hardware by combining a machine vision camera with an embedded platform such as Nvidia Jetson, Nvidia Xavier, etc.
More resources:
Design your multispectral area scan camera and reveal the hidden imaging data in your application:
About JAI multispectral customization technology: With Flex-Eye you can now design the ideal multispectral camera for your vision system. JAI’s Flex-Eye concept lets you customize the width and position of your light wavebands (both visible and near infrared) in a single 2-CMOS or 3-CMOS Fusion Series camera, collecting the exact imaging data you need. Choose 3.2-megapixel or 1.6-megapixel sensors with multi-stream output over a high-speed 10 GigE interface. Prism design means precise alignment of wavebands regardless of motion or viewing angle. Perfect for vision systems used in food sorting, agricultural, pharmaceutical, medical and many other applications. Learn more about Fusion Flex-Eye
Read more about JAI multispectral cameras in the Fusion Series
Fusion camera series for multispectral imaging
Download Tech Guide: Multispectral imaging:
Download this FREE tech guide about multispectral imaging for medical and industrial machine vision systems.
Download Tech Guide
Watch webcast: Using prism-based cameras for multispectral applications:
If you are a vision system designer with a project that needs multispectral imaging capability, this free webinar can help you decide whether a prism-based multispectral camera is the right approach. There are several different types of multispectral cameras currently available and the one you select can have a major impact on your project’s success.
Watch the webinar
Contact JAI for help:
Let us help you to find the perfect camera to meet your application requirements in your medical and life sciences application.
Contact a JAI engineer