⎕CT Comparison Tolerance

The setting of the system variable ⎕CT (comparison tolerance) determines the accuracy of the comparative and logical functions. Comparison Tolerance will only matter, in practice, if either of the arguments of an affected function is represented internally as a floating-point number.

The following primitive functions are affected by ⎕CT:


             ⌈ ⌊ < ≤ = ≥ > ≠  ∊ ⍳  |  ⍷  ≡  ≢  ~ 

For equality tests, two numbers are judged to be equal if the magnitude of their difference does not exceed the value of ⎕CT multiplied by the larger of their magnitudes. X≥Y is true if X-Y is greater than or equal to ⎕CT multiplied by larger of the magnitudes of X or Y.    X>Y is true if X≥Y is true and X=Y is not.

The effect on ⌈ and ⌊ is similar. Both of these functions will have no effect on an integer. A value which is close to an integer by proportionately less than ⎕CT returns that integer irrespective of the direction in which it differs from that integer. All other values behave as expected. The residue function | is fuzzy, and thus A|B will return 0 if B÷A is within ⎕CT of an integer value.

The default in 32-bit implementations of APLX is 1E¯13, and in 64-bit implementation is 3E¯15. It can be reset by assignment to a value between 0 and just less than 1.

             ⎕CT                     (⎕CT at normal setting)
       1E¯13
             4=3.9
       0

Compare it with the result produced by the same expression after ⎕CT has been changed:

             ⎕CT← .026               (4×⎕CT is greater than the
             4=3.9                    difference between the two numbers)
       1
             ⎕PP←15                  (X is less than 1 by
             ⎕←X←1-.9×⎕CT             proportionately less than ⎕CT)
       0.9766
             ⌊X
       1
             ⌈X
       1

Special considerations for 64-bit versions of APLX

In APLX64, integers are represented as 64-bit numbers, and floating-point numbers are represented in 64-bit IEEE floating-point format, with 53 bits of precision. The default value of ⎕CT is 3E¯15. This means that, if you are dealing with numbers larger than around 2*48 (approx 2.8E14), you may get different answers for operations which depend on ⎕CT, according on whether the number is represented internally as an integer or a floating-point number. This is because operations which act on integers are carried out using exact arithmetic, without reference to ⎕CT, whereas operations which involve floating-point numbers do take account of ⎕CT.

Consider this sequence:

      X←2*50
      X
1125899906842624
      ⎕DR X
2
      X=X+1
0

      Y←1.0×X
      Y
1125899906842624
      ⎕DR Y
3
      Y=Y+1
1

In this example, X is represented internally as a 64-bit integer. It is distinct from the 64-bit integer X+1, because ⎕CT is ignored when APLX compares numbers represented as integers.

In contrast, Y (which has the same value as X) is held internally as a floating-point number. When the comparison with Y+1 is made, APLX reports that they are equal because the relative difference is less than ⎕CT.

Note that, in 32-bit versions of APLX, the behavior is the same, but you do not normally notice it because, with the default value of ⎕CT, no number which could be represented as a 32-bit integer is within comparison tolerance of the adjacent integer values.

Practical implications for 64-bit APLX applications

If the magnitude of the numbers you are dealing with is less than 2*48, then with the default value of ⎕CT there should be no problems. The results of arithmetic and comparison operations will not depend on whether the numbers are represented internally as floating-point or integer.

If the magnitude of the numbers you are dealing with is between 2*48 and 2*53, you might need to reduce ⎕CT to ensure consistent results, or alternatively force the numbers to be represented as integers or as floats before making the comparisons.

If the magnitude of the numbers you are dealing with is greater than 2*53, then numbers represented internally as floating-point cannot be converted to integer because there is not enough precision in the floating-point representation to know which integer is the correct one. If you want exact comparisons, you need to ensure that the numbers remain in integer form (for example, do not place them in the same array as non-integral values).