We’ve been asked by customers how many decimal places are enough. It’s either because they want to maximize the resolution of their purchase or they expect far more resolution from the equipment than is reasonable.

We’ve seen a lot, even a meter being set to read 100,000.001. That’s 100 million counts, though this instrument (i.e., readout) wasn’t even stable to 1 count. The extra 0.001 meant nothing.

Then there’s the other extreme, where some say that rounding is affected. They might be right, depending on what they’re trying to accomplish. Many meters will round based on an invisible digit to the right of the decimal place. That means if a meter can read 10,000 counts by a resolution of ±1 count, the meter is likely rounding based on 10,000.X, where X equals 0.1 – 0.9. Thus, 10,000.6 becomes 10,001, and 10,000.3 becomes 10,000.

This article isn’t about formulas or rounding. But it is about determining how many decimal places are enough.

Repeatability and resolution

Let’s start with a situation where we can set the appropriate resolution to display some degree of repeatability between measurements.

 …