The students were system integrators, company representatives actively looking for machine vision solutions, and generally people who were interested in seeing how a basic machine vision application can be planned and set up in 10 to 20 minutes.

• Vision China, Shanghai, March 20 - 22, 2012
• Amper, Brno, March 20 - 23, 2012
• WIN, Istanbul, March 29 - April 1, 2012
• Automation World Show, Seoul, April 3 - 6, 2012
• VIT Expo, Moscow, April 17 - 19, 2012

The Imaging Source welcomes all customers to visit its booth to talk with its sales engineers in person. Or you can reach us via email or phone. Click here for contact details.

If you would like more information about The Imaging Source and our products, please contact us.

If you would like more information about The Imaging Source and our products, please contact us.

More than one hundred exhibitors, including The Imaging Source Asia & Beijing United SCI Corp, from nine countries demonstrated their latest products over 3,500 m².

7,629 international visitors from the fields of semiconductor device fabrication, system integration, automotive industries and universities discovered the latest products and trends in machine vision.

If you would like more information about The Imaging Source and our products, please contact us.

If you would like more information about The Imaging Source and our products, please contact us.

A number of companies, which manufacture products for deployment in the fields of automation, control, test, measurement and construction participated in the exhibition.

If you would like more information about The Imaging Source and our products, please contact us.

This is one of the largest and most established exhibitions for surface-mounted device (SMD) and as such it is an absolute must that The Imaging Source be present.

More than 500 electronics manufacturers, including The Imaging Source Asia and its partner, Sunvision Technology, exhibited their latest products and technologies.

If you would like more information about The Imaging Source and our products, please contact us.

Machine vision systems are good at bottle and vial feature inspections such as, cap placement, label print quality and readability, fill level, and a variety of container defects such as chips and cracks. A bigger challenge is the inspection of what is inside a bottle. Centice Corporation of Morrisville, NC has developed technology to tackle that difficult problem. Their sensors look through the container to determine the chemistry, size, shape, and color of objects inside transparent and semi-transparent containers rather than characteristics of the containers themselves. Centice’s first product, the PASS Rx Pharmaceutical Authentication System combines machine vision with Raman Spectroscopy to look through the bottoms of amber pill vials used in typical retail pharmacies to dispense medication. The purpose of the PASS Rx system is to help the pharmacist assure the contents of the vials dispensed to the patient is precisely the medication prescribed by the doctor. A barcode/RFID/ weight check system is incapable of such an assurance.

Centice’s proprietary implementation of Raman Spectroscopy measures the chemical contents of the pills. Sometimes the spectroscopy alone is insufficient to determine whether the correct pills are in the vial. In some cases, families of medications such as Liptor® and Levothroid® have precisely the same spectral signature, but come in a variety of doses that must be differentiated one from another. Other families of drugs have fluorescence characteristics difficult for the spectroscopy to discriminate. These drugs almost always have differences in sizes, shapes, and/or colors. It is up to the machine vision to measure these visible differences. This is a difficult problem because the machine vision camera must look through the amber vial to evaluate the objects inside. The plastic vial bottoms have molded writing and logos, multiple angled surfaces of different thickness, and a variety of mold artifacts and anomalies that must be ignored or subtracted to perform the inspection properly. Figure 1 is an image of an empty vial as seen by the vision system. Figure 2 is an image of a vial full of one of the Levothroids® as seen by the vision system.

Figure 1.

Figure 2.

The machine vision problem is made tractable because the Raman Spectroscopy reduces the number of possible shapes, sizes, and colors that need to be evaluated. In the case of Liptor®, the Raman can reduce the number of individual drugs the vision system must evaluate from a universe of 3000-4000 drugs for a typical pharmacy to the four Liptors® (10mg, 20mg, 40mg, and 80mg). In the case of Liptor®, the machine vision must discriminate between the pills based on size alone because the shape and color of all the pills are the same. Other pills are evaluated on any combination of the three machine vision features. This article describes some approaches to feature extraction that solve this very difficult machine vision challenge. The challenges associated with combining vision features with Raman features for classification are left for another article.

The Challenges

There are several challenges associated with viewing objects through the vial bottom. The writing, mold artifacts, and boundaries between the multiple surfaces of the bottom of the bottle manifest themselves as occlusions and discontinuities in the surfaces and edges of the objects inside the vial. These same artifacts add “false” edges and surfaces in the image not associated with objects that are outside the vial or part of the vial itself. The thicknesses of the different vial bottom surfaces create a different level of magnification for each of the surfaces. The color of the vial material blocks ranges of light spectra (the amber vial material blocks blue spectra). Bin (or, in this case, pill) stacking occurs so that pills occlude each other, shadows are added, perspective (from pills stacked randomly at different angles) must be managed, and pill shapes from more than one side of the pill can appear (imagine a round pill sitting on its edge).

Image Accumulation

Different light modes and spectra illuminate the scene of the bottom of the vial differently. Application of a multi-mode, multi-spectral image accumulation methodology maximizes the information available to discriminate between object types. The Pass Rx is about the size and shape of a midsized coffee maker so it can fit onto the counter of a typical retail pharmacy. That means the camera, optical system, and lighting have to fit into a very compact area with a short object distance so there is space to accommodate the Raman spectrometer, mechanical system, and embedded computer that make up the rest of the system. Centice worked closely with The Imaging Source (Charlotte, NC/Bremen, Germany) to find a camera with the right sensitivity, speed, drivers, API, and cost to meet the requirements of the application. Navitar (Rochester, NY) helped Centice identify an inexpensive lens to minimize fisheye and other undesirable effects. The multi-mode, multi-spectral lighting system was developed by Centice in-house.

Evaluating Shape and Size

There are a number of ways to evaluate the images after they are accumulated. The Raman sensor can almost always determine the pill type to within a range of two to six pill types based on spectral fingerprints alone. The machine vision system then needs to calculate features that describe the size, shape, and/or color well enough to identify the exact strength category to which the pill belongs. The remainder of this article provides a brief description of the approach taken for each of those categories.

The shape and size features are divided into two categories: 1) features of the predominant shape in the bottle irrespective of a priori information, and 2) features that describe conformance to a known shape set. The first method of shape evaluation is preferable because it does not require retrieval of pill information from a data store to start the inspection. Shape feature extraction depends only on the ability to identify those pixels in the image associated with objects inside the vial rather than outside the vial or on the vial, accumulation of those pixels into segments that are contiguous spatially or spectrally, and calculation of features based on the areas or boundaries of those segments. Figure 3 shows results of searches for two pills where the predominant shapes are circles of different sizes.

Figure 3.

Often though, the first category of shape and size features is insufficient to to categorize a pill correctly. In those cases, an expected shape/size set is passed to the vision system that describes the boundaries and areas of the pill family identified by the spectroscopy. This is a very effective method of correctly identifying the shape in the vial, but requires the use of computationally intensive and often, non-deterministic image processing techniques. Figure 4 shows the result of a search for a pill with an expected diamond shape of an expected size. Figure 5 shows that same inspection but with the pills of the incorrect shape and size in the vial.

Figure 4.

Figure 5.

Evaluating Color

The challenge of color is twofold. First, the camera has to be calibrated in a manner that best accommodates the spectra of the vial material. Second, the pixels that hold valid color information must be separated from the pixels that do not. The pixel selection problem is different from the one described in the paragraphs on shape and size features. Those pixels that hold good shape information might not be so good for evaluating color if they fall in an area of shadow or glint. After the pixels are selected, color histograms are calculated for several color spaces. The color channel histograms are then evaluated against reference histograms. Pixels with no useful color information are marked in red in the vial image shown in Figure 6. The original image is shown in figure 7.

Figure 6.

Figure 7.

Conclusion

The combination of Raman Spectroscopy and machine vision provides synergies in the evaluation of objects in packages that cannot be realized by machine vision or spectroscopy alone. The development of new vision techniques to look inside semi-transparent vials was required as the techniques typically available for package inspection are mostly useful for measuring things like labels, barcodes, fill levels, and cap placement. In the case of the PASS Rx, powerful object classification techniques were needed to perform the difficult task of drug authentication.

US Geological Survey (USGS) and US Forest Service hydrologists, and municipal water resource managers utilize automatic water level measurement sensors to track water levels in lakes, rivers, and other bodies of water around the United States. They generally use float, pressure transducer, or ultrasonic transmitter based gauges. Water level measurements (a height in feet or meters) with timestamp and location data are transmitted to a database server via cellphone or satellite.

At times it is difficult or impossible to distinguish between hydrologic events and equipment malfunctions from the data sent to the server. For example, debris or sediment buildup may affect water level measurement. A technician must visit the remote systems, sometimes at great expense, to know what caused anomalous measurements. The uncertainty and expense of such anomalies can be mitigated with a camera based system that transmits an image of the scene along with the water level measurement, timestamp, and location.

Figure 1.

Dr. François Birgand of the Biological and Agricultural Engineering (BAE) Department at North Carolina State University approached GaugeCam.com to help him develop just such a camera-based water level measurement system. GaugeCam.com develops systems that perform machine vision and image processing tasks and presents the results in real-time on the web. The entire scope of the project was to develop an inexpensive, camera-based system to measure and track the level of water in streams, lakes, rivers, ponds, and even the ocean.

First though, Dr. Birgand wanted to perform basic feasibility testing to determine whether it was possible to accurately measure water level in a controlled environment. He and Troy Gilmore, his research assistant, proposed a three phase feasibility study, only moving on to a new phase after the previous one was completed successfully:

1. Phase One: A one week study to determine whether a camera could track the level of water in a bucket or pan in a lab. Do not try to convert the the water level position from the pixel position of the line in the image to the real world coordinates measured in feet/meters required by the USGS.
2. Phase Two: Build a test setup in the lab that provides more variability in the image (lighting and shadow variation, background variation, illumination with IR lighting required for night measurements, etc.). Do the calibration to perform the pixel to world coordinate conversion required to report the water level in feet/meters. Perform studies to measure the repeatability and accuracy of the system under a variety of conditions.
3. Phase Three: Place a camera beside the USGS water level measurement station in Pullen Park on the NCSU campus to determine whether the measurements taken by the camera-based system tracked with the measurements taken by the USGS.

Dr. Birgand was ready to invest additional resources to take the water measurement system into the wild only if the all phases of the feasibility study were successful.

Phase One - Basic Feasibility

The entire purpose of the first phase feasibility study was to determine whether it was possible to find the water edge robustly under relatively good conditions. GaugeCam.com selected a VGA color camera from The Imaging Source to perform the study. This camera was selected because of its durability, broad capabilities (trigger input, strobe output, USB connectivity, etc), low cost, and drivers available for use with both Microsoft Windows® and Linux. The suction cup camera mount that came as part of the The Imaging Source development kit provided a way to fix the camera in a position to watch the water rise in a bread pan. GaugeCam selected a line based edge detection algorithm to look for edge points on a (relatively) horizontal water line and fit the points to a line. This edge detection scheme was both effective and fast at finding the water edge. Figure 2 is a screenshot of one of the measurements. You can see a video of measurements in close to real-time here.

Figure 2.

Phase Two - Water Level Measurement Repeatability and Accuracy in the Lab

The second phase required several months of work. Troy built an apparatus in the BAE lab at NCSU that consists of a clear Plexiglas tube on a table with a water reservoir beneath the table. There is a siphon hose that runs from the top of the tube down to the water reservoir. A pump continuously fills the tube with water at a fairly slow rate. The water enters a siphon hose when the tube is almost full so that the water in the tube is rapidly siphoned back to the reservoir even as the pump continues to fill the tube. The vision system measures the level of the water as it moves up and down in the tube. A replaceable background is centered in the middle of the tube to emulate the appearance of the water line against a flat surface as it would most frequently appear in the wild. Such backgrounds included the concrete and painted metal construction material used in bridges where many of the USGS stations are located. You can see a time-lapse video of the The Imaging Source camera connected to one of the lab computers detecting the water level as it moves up and down in the tube here. A later version of the setup is shown in Figure 1.

As was mentioned previously, a line based edge detection algorithm is used to find the waters edge in the tube. A series of vertical lines in the area inside the tube are evaluated to determine the position of the waters edge. The edge point positions are defined by their pixel row and column positions within the image. The equation for the water line is calculated by a linear regression fit of the edge points. Figure 3. is a diagram from a detailed description of how the line edge detection works. That detailed description can be found here. The actual code for line based edge detection and its use in real world application will be the subject of a later article.

Figure 3.

The next step in the process is to convert the position of the line from the measurement system used in images (pixel units) to world units (feet/meters). Troy add calibration fiducials to the scene. The calibration fiducials are the large black dots to the left and right of the water tube. The scale between the left column of dots and the water tube is a section of scale typically used by the USGS to allow for visual water level checking in the wild. The dots were position at precise locations relative to the scale with the aid of a laser level. Next, an image of the scene is evaluated using blob analysis so their exact position is known in pixel units. The pixel to world coordinate conversion system can then be calculated because the pixel and world coordinate positions of each of the dots is now known. Figure 4. shows the system derived from the dots. The actual code for the pixel to world coordinate conversion model and its use in real world application will be the subject of a later article. A scale calculated from the model is shown as an overlay (in red) on the tube in Figure 1.

Figure 4.

Troy and his associates built infrastructure to capture images with as much variability as possible. They mounted the camera on a track that allowed it to be positioned from one to meters from the tube at different angles and elevations. A track to move a bright white light to a variety of positions and elevations was used to throw shadows and glints onto the scene. Troy systematically captured images and used the water level measurement and pixel to world coordinate conversion algorithms to calculate the accuracy and repeatability of the system.

The USGS wants measurements to a precision of +/- 0.01 feet. Of course, the ability to meet such a criteria, depends on how many pixels per inch are represented by a given field of view. A surprising result of the Phase Two investigation is that the water levels were often measured to within less than a pixel. What does that mean? If the field of view for the 640x480 image is four feet high, that means every vertical pixel in the image represents 0.0083 feet. An increase in the pixel resolution of the camera increases the precision of the system. For instance, The Imaging Source has a 1280x960 pixel resolution camera that would double the precision of the measurements from those made with a 640x480 pixel resolution camera. The camera-based water level measurement system easily met the criteria established for Phase II of the feasibility study.

Phase Three - A controlled field study

Permission was granted by NCSU and the City of Raleigh to install both calibration fiducials and a camera beside the USGS water level measurement station at the stream running through Pullen Park on the NCSU campus. Figure 5 is an image of the scene as it appears during the night with IR illumination.

Several weeks of work were required to acquire permissions, mount the camera, test the lighting and perform other setup chores. As for the results, Troy’s blog notes after he accumulated the first comparison results are duplicated below with permission: Snapshot of GaugeCam and USGS Data – A Qualitative Comparison, Filed under Data, Image processing by Troy

The Pullen Park hydrograph below is a snapshot of provisional data found on the USGS website.

From a qualitative standpoint, we were pleased to find our data set mirrored the USGS data fairly well. We continue to investigate methods to minimize the gaps in data as well as measurement offsets compared to the USGS data. The GaugeCam hydrograph shape is shown below. One outlier was removed manually from the GaugeCam data set.

With the successful completion of the laboratory feasibility study, Dr. Birgand approved the release of additional funds to perform a larger field study to include an additional stream measurement study, coastal tide water level measurement, further data gathering with higher precision measurements at the Pullen, and additional controlled experiments in the BAE laboratory at NCSU.