Machine Vision and Raman Spectroscopy

Machine vision systems are good at bottle and vial feature inspections such as, cap placement, label print quality and readability, fill level, and a variety of container defects such as chips and cracks. A bigger challenge is the inspection of what is inside a bottle. Centice Corporation of Morrisville, NC has developed technology to tackle that difficult problem. Their sensors look through the container to determine the chemistry, size, shape, and color of objects inside transparent and semi-transparent containers rather than characteristics of the containers themselves. Centice's first product, the PASS Rx Pharmaceutical Authentication System combines machine vision with Raman Spectroscopy to look through the bottoms of amber pill vials used in typical retail pharmacies to dispense medication. The purpose of the PASS Rx system is to help the pharmacist assure the contents of the vials dispensed to the patient is precisely the medication prescribed by the doctor. A barcode/RFID/ weight check system is incapable of such an assurance.

Centice's proprietary implementation of Raman Spectroscopy measures the chemical contents of the pills. Sometimes the spectroscopy alone is insufficient to determine whether the correct pills are in the vial. In some cases, families of medications such as Liptor® and Levothroid® have precisely the same spectral signature, but come in a variety of doses that must be differentiated one from another. Other families of drugs have fluorescence characteristics difficult for the spectroscopy to discriminate. These drugs almost always have differences in sizes, shapes, and/or colors. It is up to the machine vision to measure these visible differences. This is a difficult problem because the machine vision camera must look through the amber vial to evaluate the objects inside. The plastic vial bottoms have molded writing and logos, multiple angled surfaces of different thickness, and a variety of mold artifacts and anomalies that must be ignored or subtracted to perform the inspection properly. Figure 1 is an image of an empty vial as seen by the vision system. Figure 2 is an image of a vial full of one of the Levothroids® as seen by the vision system.

Figure 1.

Figure 2.

The machine vision problem is made tractable because the Raman Spectroscopy reduces the number of possible shapes, sizes, and colors that need to be evaluated. In the case of Liptor®, the Raman can reduce the number of individual drugs the vision system must evaluate from a universe of 3000-4000 drugs for a typical pharmacy to the four Liptors® (10mg, 20mg, 40mg, and 80mg). In the case of Liptor®, the machine vision must discriminate between the pills based on size alone because the shape and color of all the pills are the same. Other pills are evaluated on any combination of the three machine vision features. This article describes some approaches to feature extraction that solve this very difficult machine vision challenge. The challenges associated with combining vision features with Raman features for classification are left for another article.

The Challenges

There are several challenges associated with viewing objects through the vial bottom. The writing, mold artifacts, and boundaries between the multiple surfaces of the bottom of the bottle manifest themselves as occlusions and discontinuities in the surfaces and edges of the objects inside the vial. These same artifacts add "false" edges and surfaces in the image not associated with objects that are outside the vial or part of the vial itself. The thicknesses of the different vial bottom surfaces create a different level of magnification for each of the surfaces. The color of the vial material blocks ranges of light spectra (the amber vial material blocks blue spectra). Bin (or, in this case, pill) stacking occurs so that pills occlude each other, shadows are added, perspective (from pills stacked randomly at different angles) must be managed, and pill shapes from more than one side of the pill can appear (imagine a round pill sitting on its edge).

Image Accumulation

Different light modes and spectra illuminate the scene of the bottom of the vial differently. Application of a multi-mode, multi-spectral image accumulation methodology maximizes the information available to discriminate between object types. The Pass Rx is about the size and shape of a midsized coffee maker so it can fit onto the counter of a typical retail pharmacy. That means the camera, optical system, and lighting have to fit into a very compact area with a short object distance so there is space to accommodate the Raman spectrometer, mechanical system, and embedded computer that make up the rest of the system. Centice worked closely with The Imaging Source (Charlotte, NC/Bremen, Germany) to find a camera with the right sensitivity, speed, drivers, API, and cost to meet the requirements of the application. Navitar (Rochester, NY) helped Centice identify an inexpensive lens to minimize fisheye and other undesirable effects. The multi-mode, multi-spectral lighting system was developed by Centice in-house.

Evaluating Shape and Size

There are a number of ways to evaluate the images after they are accumulated. The Raman sensor can almost always determine the pill type to within a range of two to six pill types based on spectral fingerprints alone. The machine vision system then needs to calculate features that describe the size, shape, and/or color well enough to identify the exact strength category to which the pill belongs. The remainder of this article provides a brief description of the approach taken for each of those categories.

The shape and size features are divided into two categories: 1) features of the predominant shape in the bottle irrespective of a priori information, and 2) features that describe conformance to a known shape set. The first method of shape evaluation is preferable because it does not require retrieval of pill information from a data store to start the inspection. Shape feature extraction depends only on the ability to identify those pixels in the image associated with objects inside the vial rather than outside the vial or on the vial, accumulation of those pixels into segments that are contiguous spatially or spectrally, and calculation of features based on the areas or boundaries of those segments. Figure 3 shows results of searches for two pills where the predominant shapes are circles of different sizes.

Figure 3.

Often though, the first category of shape and size features is insufficient to to categorize a pill correctly. In those cases, an expected shape/size set is passed to the vision system that describes the boundaries and areas of the pill family identified by the spectroscopy. This is a very effective method of correctly identifying the shape in the vial, but requires the use of computationally intensive and often, non-deterministic image processing techniques. Figure 4 shows the result of a search for a pill with an expected diamond shape of an expected size. Figure 5 shows that same inspection but with the pills of the incorrect shape and size in the vial.

Figure 4.

Figure 5.

Evaluating Color

The challenge of color is twofold. First, the camera has to be calibrated in a manner that best accommodates the spectra of the vial material. Second, the pixels that hold valid color information must be separated from the pixels that do not. The pixel selection problem is different from the one described in the paragraphs on shape and size features. Those pixels that hold good shape information might not be so good for evaluating color if they fall in an area of shadow or glint. After the pixels are selected, color histograms are calculated for several color spaces. The color channel histograms are then evaluated against reference histograms. Pixels with no useful color information are marked in red in the vial image shown in Figure 6. The original image is shown in figure 7.

Figure 6.

Figure 7.

Conclusion

The combination of Raman Spectroscopy and machine vision provides synergies in the evaluation of objects in packages that cannot be realized by machine vision or spectroscopy alone. The development of new vision techniques to look inside semi-transparent vials was required as the techniques typically available for package inspection are mostly useful for measuring things like labels, barcodes, fill levels, and cap placement. In the case of the PASS Rx, powerful object classification techniques were needed to perform the difficult task of drug authentication.