The Infrared GPS system (iGPS) from Metris Inc. can be used to locate and track any point in a rugged indoor environment precisely and accurately. It was used by the Systems, Controls, and Components Research division at Caterpillar Inc., to calibrate the various sensors on a 420 E-series backhoe loader (BHL), to measure the accuracy at key points, and to track the location of the BHL in real time as it performs various operations. A setup of the iGPS is carried out initially, and single-point sensors are mounted at various points on the linkage of the BHL. The sensor calibration is carried out using a combination of these sensors and handheld tools for points that are hard to reach. After the calibration, the linkage kinematics is verified and the accuracy at the bucket tip, a key point of interest, is quantified using a custom interface. To track the BHL in real time, three single-point sensors are mounted on the roof of the machine. The calibration process to locate the BHL in the reference system set up by iGPS, integration of iGPS software into the operator display, and communication of real-time location to the on-machine control module is described. The location information is also used to update the position of the BHL in real time in a 3-D visualization software running on a remote PC. The BHL is shown performing various operations in the 3-D visualization software, which relate quite well to the real world.

Introduction

The backhoe loader (BHL) is a very versatile machine that can be used in light construction to perform a variety of tasks. The Systems, Controls, and Components Research Division at Caterpillar is responsible for developing operator-assist features for the BHL to aid in performing these tasks effectively. These operator-assist features are designed to increase productivity and reduce operator involvement. These are also designed with the aim of raising an unskilled operator to the level of a skilled operator. There are two main focus areas for designing operator-assist features: linkage control and site management. For the first, a key requirement is to be able to locate points on the linkage with the precision and the accuracy the size of a golf ball. To achieve this, an accurate calibration of the various sensors on the BHL and the ability to quantify the accuracy of the kinematics is required. For the second, it is required to locate the BHL in real time and then use this to update the information for 3-D site visualization and site management applications. The Infrared GPS system (iGPS) from Metris can locate and track any point in a rugged indoor environment precisely and accurately. This article describes how the iGPS was used at Caterpillar for sensor calibration and real-time tracking of a backhoe loader.

iGPS setup

The iGPS consists of infrared transmitters mounted at fixed locations, various sensors— handheld and mountable on the linkage—to measure the location of a point in space, wireless data transmitters to transmit position data from the sensors to a laptop server in real time, and either standard or custom software running on the laptop to capture and process the position information in real time.

Typically, four to six transmitters are used to carry out a sensor calibration for a backhoe loader. Figure 1 shows the mounting of four transmitters pointed down into the test area, where the sensor calibration is to be carried out. A setup for the iGPS is carried out to fix the location and orientation of the transmitters in the test area. The six degrees of freedom (6DOF) of each transmitter have to be determined precisely because this determines the accuracy with which a point can be measured in a given area. Metris has proposed a “bundling algorithm” technique for carrying out the setup. During setup, a number of points are collected using a sensor and receiver in the test area, and the bundling algorithm determines the location of the transmitters relative to the test area, essentially using a “reverse triangulation” technique. The points should be chosen carefully, as this influences the accuracy and precision with which points can be measured.

Figure 1: A typical mounting of the iGPS system while carrying out a calibration of the sensors on the machine. Four transmitters (T1, T2, T3, and T4) are shown pointed into the test area.

After determining the position and orientation of all the transmitters, and thereby setting up the coordinate reference frame in which to carry out the measurements, a scale bar is used to set the coordinate frame scale. The results of the setup are then stored in a configuration file. The metrics give a measure of the quality of the setup. These metrics can be monitored to ensure that the points being collected later are within our accuracy specification. If the metrics exceed a certain level, then a setup can be carried out again to bring it back within specification. After the setup, a known length is measured to verify the scale of the coordinate reference frame. The coordinate frame formed by the iGPS has the x-axis of in the direction of T1 to T2, the z-axis is in the direction of the gravity vector, and the y-axis is then obtained from the cross-product of the unit vectors along the x-axis and the z-axis.

Calibration of sensors on an E-series backhoe loader

The iGPS was used to calibrate the sensors on an E-series backhoe loader (BHL). This BHL has a number of sensors that are used to locate the orientation of the various links and thereby locate the position of various points on these links with respect to the machine origin. These include:
1. A pitch-roll sensor to measure the orientation of the frame
2. A swing sensor to measure the swing angle of the backhoe linkage arm by measuring the rotation of the swing link with respect to the frame
3. Boom, stick, and bucket in-cylinder sensors to measure the extension of the hydraulic cylinders affecting the orientations of the boom, stick, and bucket links, respectively
4. Estick in-cylinder sensor to measure the extension of the Estick. This is not shown in Figure 2 because it is located inside the stick link housing and is used to extend the reach of the bucket link.

Figure 2: Backhoe linkage arm showing the various links and the corresponding cylinders affecting their orientation

A geometric relationship between cylinder displacements and link angles is used to translate the boom, stick, and bucket cylinder displacements into their corresponding link angles θ_boom, θ_stick, and θ_bucket. These link angles are then used to compute the location of key points like the center tooth tip location on the bucket or the bucket tip using forward kinematics. The accuracy with which the cylinder displacements are measured and the accuracy of the relationship between cylinder displacements and link angles determines the accuracy with which points like the bucket tip can be located with respect to the BHL origin.

Pitch-roll sensor offsets

The pitch-roll sensor offsets are determined by measuring a number of points on the frame of the backhoe loader. These points are measured using a handheld vector bar and are with respect to the iGPS frame (global frame in the test area). There are two kinds of vector bars that can be used: the metrology-grade vector bar is smaller and more accurate, whereas the ruggedized tech section vector bar shown in Figure 3 can be used to measure points that are hard to reach or are blocked from the view of at least two transmitters when a metrology-grade vector bar is used.

The location of these points on the frame is known with respect to the BHL origin (same as the frame origin). The BHL origin is located at the center of the rear axle. The location of the BHL origin is also measured with respect to the iGPS frame.

Figure 3: A frame point being measured using a handheld ruggedized tech section vector bar

A 4x4 transformation matrix (T matrix) transforms the points in the BHL reference frame to the points in the iGPS frame. The structure of this matrix is known, but we do not know the exact values for the yaw, pitch, and roll of the machine with respect to the iGPS frame. The relationship between the points in the two reference frames is:

P_iGPS=T*P_BHL

P_iGPS=Frame points in the iGPS frame
P_BHL=Frame points in the BHL frame

About six to eight points on the BHL frame are measured using the vector bar. These points should be spread out over the entire frame so as to give a good representation for the orientation of the frame. The locations of these points in the BHL frame are obtained from the engineering drawings of the frame to give the matrix PBHL. The T matrix is given by:

T=P_iGPS*pinv(P_BHL), where pinv refers to the pseudo inverse of the matrix of points. The T matrix can then be used to solve for the actual pitch, roll, and yaw of the frame. By comparing the computed values of pitch and roll, to the values measured from the pitch-roll sensor on the BHL the offsets for the sensor are determined. After correcting for the pitch-roll sensor offset in software, the iGPS frame is rotated by the yaw angle so that the x-axis of the BHL coincides with the x-axis of the iGPS frame. This ensures that when the BHL linkage arm is parallel to the x-z plane of the iGPS reference frame then the swing angle is zero.

Swing sensor offset

A swing sensor offset is the difference in the swing sensor reading from zero when the centerline of the BHL linkage arm is aligned with the x-axis of the BHL reference frame. To measure this, flat sensors are mounted on both sides of the boom pivot pin. The average of these two sensors gives the location of the center of the boom pivot pin. A cylindrical sensor in mounted at the bucket tip. This is shown in Figure 4.

Figure 4: iGPS sensors and wireless data transmitters shown mounted at the Boom link pivot pin and the bucket tip

The swing arm is rotated so that the central plane of the BHL linkage arm containing the center of the boom pivot pin and the bucket tip is parallel to the x-z plane of the iGPS reference system. This represents the swing zero for the BHL linkage arm. The reading of the swing sensor in this position gives the magnitude of the offset in the swing sensor. The arm can be swung to different orientations and an average value of the offset can be calculated by comparing the swing sensor reading to the absolute value of the angle of the centerline of the link arm.

Boom, stick, and bucket cylinder offsets

The procedure for computation of the boom, stick, and bucket cylinder offsets is almost the same, with the only difference being that the sensors are mounted on different pins. Therefore, only the computation of the boom cylinder offset is described here. Flat sensors are mounted on either side of the boom pivot as shown in figure 4. Flat sensors are mounted on either side of the stick pivot (i.e., the point where the boom link connects to the stick link). The center of the pivots can be computed by measuring the average value of the flat sensors mounted on either side of the pivot pins. The BHL linkage arm is positioned at zero swing. The actual value of the boom link angle can be calculated by computing the angle that the centerline connecting the pivots makes with the x-axis. The boom cylinder displacement can be computed by using the geometric relationship between the boom link angle and the boom cylinder displacement. This cylinder displacement is compared to the cylinder displacement obtained from the boom in-cylinder sensor. The difference gives the cylinder offset for that location. The boom link is moved through the entire range and an average value of the offset is computed.

This procedure assumes that the geometric relationship between link angles and cylinder displacements is accurate. This assumption is generally not true because of the presence of slop between the link pins, the bending of the piston rod in the hydraulic cylinder, and other nonlinear effects that affect this relationship. A more accurate calibration is to directly measure the cylinder displacement (by mounting sensors at the end points of the cylinders). The cylinder displacement and the link angle are measured at the same time. By collecting a number of points through the entire range of extension of the boom cylinder, we can come up with a data-based relationship between the link angles and the cylinder displacements, which improves the accuracy with which we can locate key points on the link.

This procedure is repeated to compute the offset for the stick and bucket cylinder and, if necessary, to come up with the data-based relationship between link angles and cylinder displacements.

Estick cylinder offset

The Estick is slightly different in the sense that this link slides inside the stick link housing, extending the reach of the bucket. It forms a prismatic joint with the stick. The sensors are mounted on the stick pivot and the bucket pivot (i.e., the point where the Estick connects to the bucket link). The distance between pivots can be converted into a displacement of the Estick cylinder. This is compared to the value measured by the Estick in-cylinder sensor to obtain the offset. This value of the offset is calculated at different points through the length of the Estick cylinder.

Estimating the static accuracy at key points on the BHL linkage arm

After the calibration, we obtain “true” readings from the in-cylinder sensors. These cylinder displacements obtained from the sensors are used to calculate the link angles. These link angles can be used to calculate the locations of key points on the BHL linkage arm in the local link reference frames as well as the BHL reference frame using forward kinematics. Computing the location of key points accurately in the BHL frame, located at the center of rear axle with the vertical axis parallel to gravity, is required for developing various operator-assist control features. It is also important to estimate the accuracy with which we can locate these on the BHL linkage arm. Estimating the static accuracy at one such key point, the bucket tip, is described here.

A single-point cylindrical sensor is mounted at the bucket tip, as shown in figure 5. The value measured by the sensor is translated to the BHL frame origin so as to be able to compare it to the bucket tip value calculated within the on-machine control module using forward kinematics. The bucket tip is moved to various locations in space and the static accuracy of the bucket tip is calculated at each location by measuring the difference between the calculated value and the measured value. If the bucket tip errors are outside the desired accuracy specification, then the calibration is repeated.

Figure 5: A cylindrical iGPS sensor is mounted between the third and fourth teeth of the bucket link. The center tooth tip or the bucket tip is at this location. The yellow receiver suspended by a tape is not part of the setup.

Sometimes it is also possible to determine which sensors are responsible for the errors primarily by looking at the distribution of errors in space. For example, a consistent error in the XY plane on both sides of the Swing zero link plane is an indicator that there is an error in the swing sensor calibration. The swing sensor calibration is repeated and the accuracy at the bucket tip is measured again.

Estimating the dynamic accuracy at key points on the BHL linkage arm

It is also required to estimate the dynamic accuracy at key points to determine the accuracy with which we can locate the points while performing various operations (when the BHL linkage arm is in motion). Here again we focus on measuring the dynamic accuracy at the bucket tip, a key point of interest. One particular application of interest is in trying to measure the accuracy with which we can locate the bucket tip during a typical trench-digging operation.

Figure 6: Interface developed to collect data in real time

A cylindrical sensor is mounted at the bucket tip, as shown in figure 5. An interface shown in figure 6 was designed to capture the position of the sensor position in real time, along with the position of the bucket tip calculated by the onboard control module on the BHL. To compare the computed bucket tip value to that obtained from the iGPS sensor, it is critical that the data be collected at the same rate. A data rate of 25 Hertz was found to work well for a normal digging motion.

The data was collected as the bucket tip was being moved slightly above the ground in a plane parallel to it. The data collection was started when the bucket tip was the farthest from the machine. The boom and stick links are then moved smoothly in a normal digging motion, while the bucket tip is moving closer to the BHL frame. The bucket tip moves in a plane above the ground through the entire motion. The data collection is stopped when the bucket tip stops moving. This is repeated for multiple cycles and in different planes and orientations of the swing link. The average value gives an estimate of the accuracy with which we can locate the bucket tip of the BHL during a typical operation.

This accuracy may be different during actual digging operation in soil because of other nonlinear effects that were mentioned earlier like the presence of slop between the link pins, bending of the piston rod in the hydraulic cylinder, deflection of the link arm, etc., which will diminish the accuracy with which we can locate key points on the link arm. A few methods have been considered to quantify this, but will not be described in this article.

Real-time tracking of the backhoe loader

One of the key applications of the iGPS for Caterpillar Research is to be able to mimic the capability of GPS in an indoor arena. This allows various operator-assist features that rely on a site-reference system to be developed in an indoor arena, and the demonstrations of these features to managers and customers can be carried out in a controlled environment that is not affected by the weather. The iGPS system from Metris provides a high-accuracy reference system to locate and track the BHL in an indoor arena.

To locate the BHL with respect to an earth-fixed reference frame, the six degrees of freedom (6DOF) of the BHL frame have to be computed in real-time. The 6DOF for the BHL frame are: xorigin, yorigin, zorigin (i.e., the location of the origin of the BHL frame), yaw, pitch, and roll of the BHL frame. Three cylindrical sensors are mounted on the roof of the BHL, as shown in figure 7. The real-time position of these three sensors can be used to calculate the 6DOF of the BHL and hence track the BHL in motion with respect to the earth-fixed iGPS frame.

Figure 7: Three cylindrical sensors are shown screwed into yellow cylindrical tubes on the roof of the BHL.

The fixed locations of the three cylindrical sensors with respect to the BHL origin have to be computed initially. The BHL origin is located at the center of the rear axles. The rear  axle ends are measured using the metrology grade vector bar and the location of the BHL origin can be calculated with respect to the iGPS frame. The locations of the three cylindrical sensors are measured with respect to the iGPS frame. The locations of the three sensors with respect to the BHL can be calculated by subtracting the BHL origin with respect to the iGPS frame with from the sensor locations with respect to the iGPS frame.

A sensor frame is created (with the origin at one of the cylindrical sensors) using the sensor locations with respect to the BHL origin and this has a fixed transformation to the BHL frame. As the BHL moves around in the indoor arena, the positions of the sensors are constantly updated and this is used to compute the location of the sensor frame continuously. This can then be used to compute the location of the BHL frame (the location of the BHL origin and the orientation in with respect to the iGPS frame) in rea time, and this can be used to track the BHL in real time.

As a first step, standard software provided by Metris was used to come up with the fixed transformation between the sensor frame and the BHL frame, and to test the real-time updating of the sensor information. The next step was to integrate this with the operator display so as to be able to turn the tracking feature on and off, and to be able to display the location of the BHL as it moves around on the operator display.

A stand-alone application was created that could be called from the operator display. This application would interface with the sensors and calculate the location of the sensor frame in real time. This information is transmitted to the BHL control module software, which computes the location of the BHL frame in real time. This application is called when the tracking feature is turned on by the operator and is closed when turned off by the operator. There was a slight delay before the data from the sensors was available to the display software, which could be attributed to the initialization of the various modules being carried out each time the tracking was turned on. Another issue was that the application would continue to run even if the operator display software would shut down suddenly. This would cause the position updating to sometimes fail when the tracking feature was turned on the next time. It was desirable to have the tracking application fully integrated within the operator display software to avoid such issues.

A library was created using the stand-alone application as a reference, which could be integrated into the operator display software. This library would interface directly with the sensors, but in this case the information would be available directly within the operator display software. The initialization for some of the modules within the library was done on startup of the operator display software. This minimized the delay from the time when the tracking feature was turned on to when the position data from the sensors was received. This would also eliminate issues caused by the operator display closing suddenly because the modules associated with the sensor interface library would also be closed down when this happened.

The BHL could now be tracked in real time as it moved around an indoor arena. The operator turns on the tracking feature from the display within the cab. If the software does not receive position information from the sensors, an error message is displayed to the operator. The position data from the sensors is displayed on the operator display screen and verified by comparing this to the data obtained from the standard software with the BHL in a stationary position. The BHL is driven around the arena to verify that the position is updated in real time and that the data is reliable. This was a qualitative test and not a quantitative measure of the accuracy of the streaming position data.

The sensor information is then sent across to the BHL control module and the location of the BHL frame is calculated in real time. It has to be noted that when we say real time, we mean “soft” real time, because there could be delays in the communication of the position information that cannot be quantified. These delays have to be measured and quantified before being considered for closed-loop tracking control applications. Our main interest in tracking the BHL in real time was to display it in 3-D visualization software for centralized site management and an augmented reality display for the operator in the cab. This application can tolerate slight delays in the communication of the position of the sensors as long as these delays are not visually apparent.

Integration with 3-D display software and augmented reality

The real-time position of the BHL as well as the location of the link arm is sent from the BHL control software to the 3-D display software in real time. This information is used to update the BHL image in the 3-D display software continuously. In figure 8, a screen shot of the 3-D display software shows the BHL digging a trench. The position of the BHL in the indoor arena is known and corresponds really well to the actual position in the arena as shown by the actual video.

The 3-D display software can also be simultaneously running on the onboard display computer. The image captured in figure 8 was captured when the 3-D display software was running on a remote computer. The BHL location and the link position information were transmitted wirelessly to the remote computer.

Figure 8: The BHL is shown digging a trench in the 3-D display software. The corresponding shot from a video camcorder is embedded in the right corner to show the correlation to the real world.

A trench-digging operation is initially carried out in a stationary position of the BHL to verify that the data from the iGPS sensors and the motion of the image in the display software was in accordance with the motion of the BHL in the real world. A slight delay was observed in updating the image in the display software, which could be because of the refresh rate in the display software. The position from the iGPS sensors was being updated at close to 25 Hz, but there were some delays associated with the information being sent from the display software to the control software, where the location of the BHL was computed and then sent wirelessly across to the remote computer on which the 3-D display software was running. The delay could have also been because the BHL image had a high polygon count (a factor affecting the refresh rate of the image) and this could have slowed down the update rate.

Figure 9: Screen shot of the 3-D display software screen showing the BHL driving around in the indoor arena. The corresponding shot from a video camcorder is embedded in the right corner to show the correlation to the real world.

This was clearly evident when the BHL was driven around in the indoor arena. Even though the BHL frame location was being updated at a fairly fast rate, there was a delay in the image updating in the 3-D display software. A screen shot of the BHL driving around in the indoor arena, along with the corresponding image captured with a video camcorder, is shown in figure 9. The picture may look slightly different because the door to the indoor arena appears closed in the 3-D display software. Also, the location of the camera in the display software is different from the location of the physical video camcorder. A closer inspection will reveal that the two images match up well.

Applications of this could include remote control of the BHL and 3-D visualization for centralized site management. In addition, another application that was considered is an augmented reality view for the operator in the cab. This involves providing the operator with an enhanced view of the machine environment to assist in navigating blind spots and obstacles that may not be clearly visible from the cab. If the location of buried objects like pipes, wires, and other utilities can be mapped out, these can be displayed in the 3-D display software in relation to the linkage of the BHL. This avoids the need to have another person directing the operator while working around such utilities

Conclusions

The increased push towards automation is not just to reduce manpower needs, but also to create a safer working environment, which is one of the primary goals for Caterpillar. A first step towards automation is the use of operator-assist features. To implement this, it is important to quantify the accuracy with which a key point like a bucket tip can be located. iGPS from Metris was used to calibrate accurately and precisely the various sensors on an E-series BHL. The “true” values from these sensors were then used to calculate the location of key points on the linkage arm in the BHL frame using forward kinematics. The static and dynamic accuracies of the BHL were quantified for one such key point: the bucket tip. This is useful for features such as operator-assisted digging.

Another aspect of operator assistance is to provide an augmented view of the digging area to the BHL operator and provide the capability to remotely monitor the operations for overall site management. iGPS was used to track the BHL in an indoor arena as it was carrying out various manual operations. This data was sent to a 3-D visualization software running on the onboard display computer as well as a remote computer to display the BHL on the site performing various operations. The onboard view can be used to develop augmented reality features to warn the operator when the linkage is too close to obstacles or while digging around utilities like power lines. The remote display can be used to develop remote control and site management applications to manage multiple machines working on a site. iGPS is not just an effective system for feature development, but is also very useful to demonstrate the developed features to managers and customers in a controlled indoor arena.

References

iGPS User’s Manual: Indoor GPS Guide
www.cat.com ->Products -> machines -> Backhoe-Loaders
www.cat.com/cda/layout?m=172964&x=7)

Indoor GPS Technology For Metrology: Constellation 3DI

Indoor GPS Error Budget and Specifications, 2004