Noncontact measurement is the fastest growing segment in the metrology world, with new methods and technologies being introduced every year. As with all new solutions, end users want to know how to evaluate the performance of these new products, and for that they turn to standards organizations.

The catch is that standards take time, so there is typically a period with all new technology where suppliers and end users struggle to find common ground. The supplier wants to highlight their benefits and the end user wants to ensure they get the data they need.

The one existing standard for 3-D scanners is the German VDI/VDE 2634, published in August 2002, but it is not recognized as an international standard. There are currently two committees working on standards for scanners. ASTM International’s E57 committee is developing standards for mid-range scanners, but has abandoned their effort for short-range 3-D imagers. The International Organization for Standardization (ISO) Technical Committee 213 Working Group 10 is developing international standards—currently for scanners attached to the end of Cartesian coordinate measurement machines (CMMs)—but possibly in the future on standards for other types of scanners. These standards are likely several years from publication.

Now is the time to consider all possibilities and get involved in the creation of standards that will define how we evaluate existing and future imaging technologies. This article explores the complexities in defining a standard or set of standards for short-range 3-D imaging and scanning systems. It starts with a review of different scanning methods and characteristics and continues with a discussion of proposed test methods and evaluation criteria.

Overview

Short-range 3-D imaging covers point cloud capture devices that typically have a field of view and/or range from a few inches to a few meters. These systems are capable of capturing millions of surface points in seconds and report them as raw point clouds or polygonized mesh.

As the need for comprehensive data grows, the use of short-range imaging systems is following, but it is not always easy to determine whether the point cloud data meets the requirements for the application. End users looking for evaluation criteria are in need of a standard to provide a methodology to test and characterize the available products in the marketplace.

The only short-range 3-D imaging standard in the world today is the VDI/VD 2634, which is a German standard first published in August 2002 and revised for multiple images in December 2008. The ASTM E57 initiative had started a subcommittee to work on short-range 3-D imaging as part of their original scope, but since then has limited the E57 to medium-range systems only.
The International Organization for Standards (ISO) has an active technical committee working on ISO/CD 10360-8—“Geometrical Product Specifications (GPS)—Acceptance and reverification tests for CMM— Part 8—CMMs with optical distance sensor.” The scope of the draft document states that it does not explicitly apply to non-Cartesian CMMs, but may be applied by mutual agreement among the parties involved.

Technology review

There are several types of technologies and dozens of variations being used for short-range 3-D imaging. Each offers a unique set of characteristics that provide different benefits. Understanding how these systems work can help identify where they are best suited. This understanding is also important in formulating test methods that expose the strengths and weaknesses of the systems.

Area imagers

Most area imagers are based on the projection and observation of light onto a surface. Structured light 3-D imagers consist of a projector and receiver covering a slab-like measurement volume. Light is projected onto an area and viewed by a camera (see figure 1). Variations on this include but are not limited to white light projections of various fringes and patterns, laser fringe projection, and laser line sweeping.

Figure 1: This method yields a large number of data points all at once. For example, a system with a 1 megapixel charge-coupled device (CCD) sensor will deliver up to 1 million points per image.

Line scanners
Line scanners use a triangulation method to view a laser line projected onto an object (see figure 2). These systems come in a variety of sizes, with different line lengths, and in some cases with multiple lines and sensors. Some systems use a moving spot instead of a line, but the resulting data is similar in its structure. Laser line scanners collect information on one (or just a few) lines at a time, so they require the use of a 3-D navigation system to position each line as it is measured. As the scanner is moved lines are collected with data rates as high as 10s of thousands of points per second.

These types of scanners are found at the end of CMMs, portable CMMS, and other equipment that can provide the position and orientation for the scanner in real time. They are also available in systems that include real-time photogrammetry to provide the scanner positioning from an array of photogrammetry targets. Because of the requirement for a navigation device, testing of these scanners must be done with the fully integrated system.

Figure 2

Single-point scanners
Single-point scanners provide noncontact measurement to a CMM or six degree of freedom probe by emitting and received light along a single path. This provides access to deeper pockets, because it does not rely on triangulation like the other methods. They are effectively a single point distance meter, and when used in conjunction with a CMM they become noncontact scanners.

CT scanners
CT scanners use computed tomography to X-ray an object one slice at a time. The end result of a CT scan is a 3-D volume including all internal and external features.

Existing standard

The only existing standard covering short-range 3-D imaging is the VDI/VDE 2634 Part 2—“Optical 3-D measuring systems—Optical systems based on area scanning,” published in August 2002. The scope covers optical 3-D measuring systems based on area scanning, so the guideline only applies to a “single view,” where the sensor and its components do not change position during the measurement process. For this reason it is only applicable for the area imagers and not the laser line scanners.

Although the standard itself does not include laser line scanners in its scope, many of the concepts can be considered for evaluation of a laser line scanner as long as it is applied as part of a complete system that delivers 3-D point clouds.

The VDI/VDE 2634 Part 2 consists of acceptance tests on three quality parameters: Probing error, sphere spacing error, and flatness. Measurements are carried out using prescribed artifacts including a sphere, ball bar, and flat object. The sizes of the three artifacts are related to the length of the body diagonal of the 3-D imager’s field of view.

The quality parameter for probing error works on a small part of the measurement volume to record the range of residuals for a best fit sphere. The material and finish of the sphere should be diffuse and can be selected by the manufacturer. The size of the sphere is recommended to be between 10 and 20 percent of the body diagonal. The best fit sphere cannot exclude more than 0.3 percent of the points, and results are to be reported as maximum permissible error (MPE), where no error should exceed the MPE specification. This test is specified to be run in at least ten arbitrary positions within the field of view.

The prescribed procedure for probing error clearly has some shortcomings. The test is only for form and not diameter, so there is no measure of how accurately a system measures the size of a sphere. It could have a very low form error, without any indication that the diameter was in error.

The form evaluation also has some shortcomings. The standard does not prescribe the number of points to be collected. Probing error is defined in the standard as the range of radial deviations after fitting the data to a best-fit sphere. By this definition, the probing error increases monotonically with the number of collected points. It is therefore not fair to compare this measurement as made by a Cartesian CMM, where the number of probing points is specified to be 25, to the measurement as made by a 3-D scanner or imager, where the number of points collected on the surface of a sphere may be more than a million.

The quality parameter for sphere-spacing tests the system capability for measuring 3-D length. The artifact prescribed is a ball bar with the same ball size as in the probing test and set apart by a length greater than 30 percent of the body diagonal (see figure 3).

Figure 3

The sphere-spacing quality parameter is then tested by measuring the ball bar in seven recommend positions within the systems field of view, with the intent of covering the entire measurement volume (See figure 4). Again, no more than 0.3 percent of the data can be excluded for the sphere fits, but these fits can use known radii. The calculated lengths are then reported as MPE.

Figure 4

The flatness quality parameter is tested with a rectangular flat with a diffuse surface. The surface may be selected by the manufacturer. The width must be at least 50mm wide and the length greater than 50 percent of the body diagonal. Measurements must be taken in at least six positions (see figure 5), with the results calculated from a best fit plane. Again, only 0.3 percent of the data may be excluded from the fit. The full range of the remaining residuals is then reported as the flatness MPE.

Figure 5

The probing error and flatness tests are examples of tests that measure form error. These sphericity and flatness tests do not specify the number of points required, so neither test can be compared fairly between different types of scanners or other types of measurement instruments.

It should also be noted that there is a VDI/VDE 2634 Part 3—“Optical 3-D measuring systems—Multiple view systems based on area scanning,” published in December 2008. This standard covers the same three types of error as discussed in Part 2, but was added to define supplements for the measurement of objects using multiple images.

An international standard

The ISO Technical Committee 213 working group 10 is currently working on a draft of ISO/CD 10360-8—“Geometrical Product Specifications (GPS)—Acceptance and reverification tests for coordinate measuring machines (CMM)—Part 8—CMMs with optical distance sensors.” This effort is aimed at addressing the optical distance sensors attached to the end of Cartesian CMMs, which at this point has not been extended to include articulated arm CMMs.

The details of this standard are still in flux, as contributions are coming from all over the world. The latest version proposed was met with resistance from some users and manufacturers. One of the difficulties for this group is finding a methodology to apply CMM procedures to scanners.

This draft includes guidelines for probing error, but covers size and form, which is more extensive than the VDI/VDE standard. Data filtering and exclusion are open topics of discussion. It also has a length measurement error, which is quite a bit more extensive than the VDI/VDE and includes five different lengths at seven different positions—each measured three times. Data elimination is left up to the user, and use condition and filtering is based on manufacturer recommendations. These are all details that are still under discussion by the committee. Flatness measurement is included as an option if the CMM is restricted to flatness.

As mentioned in the VDI/VDE 2634-2 discussion above, there is a problem with a form error measurement in which the number of points is not bounded, because such a method does not provide a fair procedure of comparing different test instruments. One of the suggestions that has been made in the ISO working group is to use a method of local filtering for each of 25 points as in the ISO standard for Cartesian CMMs. This proposed method is still under discussion. Another possibility, which is being adopted by ASTM International for mid-range scanners, is to calculate the standard deviation of the residual errors. This method has the advantage that the measured form error is independent of the number of measured points, but it has the potential disadvantage that it does not provide a way of directly comparing the form error of scanners and Cartesian CMMs.

End-user experience

The goal of a standard is to give the end user a way to evaluate product performance to see if it meets their needs. To accomplish this, a system must be tested as it will be used. This means that for laser line scanners, they need to be tested on their carrying device. For area imagers, they need to be tested over their full volume and in many cases over multiple images that are registered together.

Both standards discussed in this article include tests on spheres and flat artifacts, but the reality is that most users want to measure something else. Users want to know how well a system can measure a feature, an edge, a corner, or down in a hole. These are all complex evaluations that have not yet been addressed.

Other factors that need to be considered are surface finish and lighting, which often play a part in data quality and quantity. Many systems have settings that reduce noise, but at the expense of resolution. The ISO standard draft gives the manufacturer the opportunity to prescribe settings for the tests, but it is possible that different settings yield different results. If the user’s application cannot use the same settings as tested, then their results may be different.

Conclusion

An internationally recognized standard would make it possible for end users to make informed decisions on what technologies can be applied to their applications. A standard would also make it possible to compare different systems to understand the strengths in different technologies.

The ASTM E57 Committee has abandoned their efforts in short-range systems, and the ISO standard, which may only cover CMM-based systems is still years away, but there is a way in which users can get involved. The manufacturers of 3-D scanning and imaging equipment provide very little in the way of test results, especially when it comes to accuracy. Customer demand is the best way to remedy this.

Even though there are no international standards, users can take it upon themselves to ask for data from manufacturers to help them understand how their products will perform in their applications.

References
1. Wikipedia, the free encyclopedia, www.wikipedia.com/cooperation
2. VDI/VDE 2634 Part 2—“Optical 3-D measuring systems —Optical systems based on area scanning,” published August 2002
3. VDI/VDE 2634 Part 3—“Optical 3-D measuring systems—Multiple view systems based on area scanning,” published December 2008
4. ISO/CD 10360-8—“Geometrical Product Specifications (GPS)—Acceptance and reverification tests for coordinate measuring machines (CMM)—Part 8—CMMs with optical distance sensors,” Draft Date: Jan. 25, 2010.