Rounding errors in software are common. When are they acceptable? When are they a matter of life and death? How should a tester investigate rounding errors? Veteran tester Mark Rutz provides an overview of rounding errors and tests.

A rounding error occurs when an input or calculated number is stored less accurately than the exact value. Consider the fraction of 2/3. The decimal representation of 2/3 is 0.666… where the six is repeated forever. Because the exact value is an infinitely long decimal, the approximate value used in the software will be a rounded value with 7 at the ending decimal place (e.g. 0.6667). Rounding errors like these are common and, frankly, unavoidable in software. The key questions are, do they matter in your application and how do you test for unintended consequences? I’ll split this subject into two areas of investigation. 

First, what is the level of accuracy your system requires for proper function? Consider software that uses position to perform its function. The distance between degrees of latitude at the equator is approximately 69 miles (111 kilometers). This means a position with an accuracy of 0.1 degree would be within about 7 miles of the real value. This accuracy may be sufficient for determining the city you are in but would be unreliable in determining the building you are standing by. In this example, testing should focus on monitoring the system to verify the location information maintains the appropriate level of accuracy throughout normal and extended periods of operation. Having an external source of accurate location data for comparison will be key.

Second, how is the value determined throughout system operation? There are different things to consider if the value is completely recalculated each cycle versus a value accumulated over time. For example, a position that uses fresh GPS data in each cycle is less prone to error than a position that is accumulated over time as in inertial navigation systems. Small errors in the delta value can accumulate over time to become significant errors in system data.

Boundary value testing of input data is a good place to start with system values that are recalculated each cycle. The tester should verify the system properly handles input values that cross from values that round down to values that round up without causing system errors.

For system values based on accumulated data, the tester should focus on checking for compounding errors. Consider a value where each calculation has an error of 0.0001. If the required accuracy is to the first decimal, you must run the system for a minimum of 1000 cycles to check for failure. If the system runs a cycle every minute, 1000 cycles will be performed in 16 hours and 40 minutes. If the system runs a cycle every second, it’s just 16 minutes and 40 seconds. Be sure to execute the system long enough for the accumulation of errors to become significant.

This is a practical example of the application of the Boundary Value Analysis testing technique. Rounding issues are common and it is incumbent upon software testers to understand these types of errors and to test properly for them. 

For historical examples of rounding error impacts, you can read the BBC article, “Why turning a plane’s computer off and on again could save your life”. Learn more about topics such as Boundary value analysis in the ISTQB Foundation Level certification syllabus.