💾 Archived View for gemini.bortzmeyer.org › rfc-mirror › rfc956.txt captured on 2022-07-16 at 23:10:13.
⬅️ Previous capture (2021-11-30)
-=-=-=-=-=-=-
Network Working Group D.L. Mills
Request for Comments: 956 M/A-COM Linkabit
September 1985
Algorithms for Synchronizing Network Clocks
Status of this Memo
This RFC discussed clock synchronization algorithms for the
ARPA-Internet community, and requests discussion and suggestions for
improvements. Distribution of this memo is unlimited.
Table of Contents
1. Introduction
2. Majority-Subset Algorithms
3. Clustering Algorithms
4. Application to Time-Synchronization Data
5. Summary and Conclusions
6. References
Appendix
A. Experimental Determination of Internet Host Clock Accuracies
A1. UDP Time Protocol Experiment
A2. ICMP Timestamp Message Experiment
A3. Comparison of UDP and ICMP Time
List of Tables
Table 1. C(n,k) for n from 2 to 20
Table 2. Majority Subsets for n = 3,4,5
Table 3. Clustering Algorithm using UDP Time Protocol Data
Table 4. Clustering Algorithm using ICMP Timestamp Data
Table 5. ISI-MCON-GW Majority-Subset Algorithm
Table 6. ISI-MCON-GW Clustering Algorithm
Table 7. LL-GW (a) Majority-Subset Algorithm
Table 8. LL-GW (a) Clustering Algorithm
Table 9. LL-GW (b) Majority-Subset Algorithm
Table 10. LL-GW (b) Clustering Algorithm
Table A1. UDP Host Clock Offsets for Various Internet Hosts
Table A2. UDP Offset Distribution < 9 sec
Table A3. UDP Offset Distribution < 270 sec
Table A4. ICMP Offset Distribution < 9 hours
Table A5. ICMP Offset Distribution < 270 sec
Table A6. ICMP Offset Distribution < 27 sec
Table A7. ICMP Offset Distribution < .9 sec
Table A8. Comparison of UDP and ICMP Host Clock Offsets
Mills [Page 1]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
1. Introduction
The recent interest within the Internet community in determining
accurate time from a set of mutually suspicious network clocks has
been prompted by several occasions in which gross errors were found
in usually reliable, highly accurate clock servers after seasonal
thunderstorms which disrupted their primary power supply. To these
sources of error should be added those due to malfunctioning
hardware, defective software and operator mistakes, as well as random
errors in the mechanism used to set and/or synchronize the clocks via
Internet paths. The results of these errors can range from simple
disorientation to major disruption, depending upon the operating
system, when files or messages are incorrectly timestamped or the
order of critical network transactions is altered.
This report suggests a stochastic model based on the principles of
maximum-likelihood estimation, together with algorithms for computing
a good estimator from a number of time-offset samples measured
between one or more clocks connected via network links. The model
provides a rational method for detecting and resolving errors due to
faulty clocks or excessively noisy links. Included in this report
are descriptions of certain experiments conducted with Internet hosts
and ARPANET paths which give an indication of the effectiveness of
the algorithms.
Several mechanisms have been specified in the Internet protocol suite
to record and transmit the time at which an event takes place,
including the ICMP Timestamp message [6], Time Protocol [7], Daytime
protocol [8] and IP Timestamp option [9]. A new Network Time
Protocol [12] has been proposed as well. Additional information on
network time synchronization can be found in the References at the
end of this document. Synchronization protocols are described in [3]
and [12] and synchronization algorithms in [2], [5] and [10].
Experimental results on measured roundtrip delays and clock offsets
in the Internet are discussed in [4] and [11]. A comprehensive
mathematical treatment of clock synchronization can be found in [1].
In [10] the problem of synchronizing a set of mutually suspicious
clocks is discussed and algorithms offered which maximize in some
sense the expectation that a correct set of "good" clocks can be
extracted from the population including also "bad" ones. The
technique is based upon overlapping, discrete confidence intervals
which are assigned a-priori. The model assumes the reasonable
presumption that "bad" clocks display errors far outside these
confidence intervals, so can be easily identified and discarded from
the voting process.
Mills [Page 2]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
As apparent from the data summarized in Appendix A, host clocks in a
real network commonly indicate various degrees of dispersion with
respect to each other and to a standard-time reference such as a
radio clock. The sources of dispersion include random errors due to
observational phenomena and the synchronization mechanism itself, if
used, as well as systematic errors due to hardware or software
failure, poor radio reception conditions or operator mistakes. In
general, it is not possible to accurately quantify whether the
dispersion of any particular clock qualifies the clock as "good" or
"bad," except on a statistical basis. Thus, from a practical
standpoint, a statistical-estimation approach to the problem is
preferred over a discrete-decision one.
A basic assumption in this report is that the majority of "good"
clocks display errors clustered around a zero offset relative to
standard time, as determined for example from a radio clock, while
the remaining "bad" clocks display errors distributed randomly over
the observing interval. The problem is to select the good clocks
from the bad and to estimate the correction to apply to the local
clock in order to display the most accurate time. The algorithms
described in this report attempt to do this using maximum-likelihood
techniques, which are theory.
It should be noted that the algorithms discussed in [10] and in this
report are are basically filtering and smoothing algorithms and can
result in errors, sometimes gross ones, if the sample distribution
departs far from a-priori assumptions. Thus, a significant issue in
the design of these algorithms is robustness in the face of skewed
sample data sets. The approach in [10] uses theorem-proving to
justify the robustness of the discrete algorithms presented;
however, the statistical models in this report are not suited for
that. The approach taken in this report is based on detailed
observation and experiments, a summary of which is included as an
appendix. While this gives an excellent qualitative foundation upon
which to judge robustness, additional quantitative confidence should
be developed through the use of statistical mechanics.
2. Majority-Subset Algorithms
A stochastic model appropriate to a system of mutually suspicious
clocks can be constructed as follows. An experiment consists of one
or more measurements of time differences or offsets between several
clocks in the network. Usually, but not necessarily, one of the
clocks is the local clock at the observer and observations are
conducted with each of several other clocks in the network. The fact
that some clocks are presumed more accurate or trusted more highly
than others can be expressed by weighting the measurements
Mills [Page 3]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
accordingly. The result is a set of statistics, including means and
variances, from which the observer is able to estimate the best time
at which to set the local clock.
A maximum-likelihood estimator is a statistic that maximizes the
probability that a particular outcome of an experiment is due to a
presumed set of assumptions on the constraints of the experiment.
For example, if it is assumed that at least k of n observations
include only samples from a single distribution, then a
maximum-likelihood estimator for the mean of that distribution might
be computed as follows: Determine the variance for every k-sample
subset of the n observations. Then select the subset with smallest
variance and use its mean as the estimator for the distribution mean.
For instance, let n be the number of clocks and k be the next largest
integer in n/2, that is, the minimum majority. A majority subset is
a subset consisting of k of the n offset measurements. Each of these
subsets can be represented by a selection of k out of n
possibilities, with the total number of subsets equal to C(n,k). The
number of majority subsets is tallied for n from 2 to 20 in Table 1.
(n,k) C(n,k) (n,k) C(n,k)
---------------------- ----------------------
(2,2) 1 (11,6) 462
(3,2) 3 (12,7) 792
(4,3) 4 (13,7) 1716
(5,3) 10 (14,8) 3003
(6,4) 15 (15,8) 6435
(7,4) 35 (16,9) 11440
(8,5) 56 (17,9) 24310
(9,5) 126 (18,10) 43758
(10,6) 210 (19,10) 92378
(20,11) 167960
Table 1. C(n,k) for n from 2 to 20
Obviously, the number of computations required becomes awkward as n
increases beyond about 10. Representative majority subsets for n =
3,4,5 are shown in Table 2.
Mills [Page 4]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
C(3,2) C(4,3) C(5,3)
------ ------ ------
1,2 1,2,3 1,2,3
1,3 1,2,4 1,2,4
2,3 1,3,4 1,2,5
2,3,4 1,3,4
1,3,5
1,4,5
2,3,4
2,3,5
2,4,5
3,4,5
Table 2. Majority Subsets for n = 3,4,5
Choosing n = 5, for example, requires calculation of the mean and
variance for ten subsets indexed as shown in Table 2.
A maximum-likelihood algorithm with provision for multiple samples
and weights might operate as follows: Let n be the number of clocks
and w(1),w(2),...,w(n) a set of integer weights, with w(i) the weight
associated with the ith clock. For the ith clock three accumulators
W(i), X(i) and Y(i) are provided, each initialized to zero. The ith
clock is polled some number of times, with each reply x causing n(i)
to be added to W(i), as well as the weighted sample offset n(i)*x
added to X(i) and its square (n(i)*x)2 added to Y(i). Polling is
continued for each of the n clocks in turn.
Next, using a majority-subset table such as shown in Table 2,
calculate the total weight W = sum(W(i)) and weighted sums X =
sum(X(i)) and Y = sum(Y(i)*Y(i)) for each i in the jth majority
subset (row). From W, X and Y calculate the mean m(j) and variance
s(j):
m(j) = X/W and s(j) = Y/W - m(j)*m(j) .
When this is complete for all rows, select the row j with the
smallest s(j) and return the associated mean m(j) as the
maximum-likelihood estimate of the local-clock offset.
Mills [Page 5]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
3. Clustering Algorithms
Another method for developing a maximum-likelihood estimator is
through the use of clustering algorithms. These algorithms operate
to associate points in a sample set with clusters on the basis of
stochastic properties and are most useful when large numbers of
samples are available. One such algorithm operates on a sample set
to selectively discard points presumed outside the cluster as
follows:
1. Start with a sample set of n observations {x(1),x(2),...,x(n)
2. Compute the mean of the n observations in the sample set.
Discard the single sample x(i) with value furthest from the
mean, leaving n-1 observations in the set.
3. Continue with step 2 until only a single observation is left,
at which point declare its value the maximum-likelihood
estimator.
This algorithm will usually (but not necessarily) converge to the
desired result if the majority of observations are the result of
"good" clocks, which by hypothesis are clustered about zero offset
relative to the reference clock, with the remainder scattered
randomly over the observation interval.
The following Table 3 summarizes the results of this algorithm
applied to the UDP data shown in Appendix A, which represents the
measured clock offsets of 163 host clocks in the Internet system.
These data were assembled using the UDP Time protocol [7], in which
time is represented to a precision of one second. Each line of the
table represents the result of step 2 above along with the size of
the sample set and its (unweighted) mean and variance. The "Discard"
column shows the value of the observation discarded at that step.
Mills [Page 6]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
Size Mean Var Discard
-------------------------------
163 -210 9.1E+6 -38486
162 26 172289 3728
161 3 87727 3658
160 -20 4280 -566
150 -17 1272 88
100 -18 247 -44
50 -4 35 8
20 -1 0 -2
19 -1 0 -2
18 -1 0 -2
17 -1 0 1
16 -1 0 -1
15 -1 0 -1
14 -1 0 -1
13 0 0 0
1 0 0 0
Table 3. Clustering Algorithm using UDP Time Protocol Data
In Table 3 only a few of the 163 steps are shown, including those
near the beginning which illustrate a rapid convergence as the
relatively few outliers are discarded. The large outlier discarded
in the first step is almost certainly due to equipment or operator
failure. The two outliers close to one hour discarded in the next two
steps are probably simple operator mistakes like entering summer time
instead of standard time. By the time only 50 samples are left, the
error has shrunk to about 4 sec and the largest outlier is within 12
sec of the estimate. By the time only 20 samples are left, the error
has shrunk to about a second and the variance has vanished for
practical purposes.
The following Table 4 summarizes the results of the clustering
algorithm applied to the ICMP data shown in Appendix A, which
represents the measured clock offsets of 504 host clocks in the
Internet system. These data were assembled using ICMP Timestamp
messages [6], in which time is represented to a precision of one
millisecond. The columns in Table 4 should be interpreted in the
same way as in Table 3, except that the data in Table 4 are in
milliseconds, while the data in Table 3 are in seconds.
Mills [Page 7]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
Size Mean Var Discard
-------------------------------
504 -3.0E+6 3.2E+14 8.6E+7
500 -3.3E+6 2.9E+14 8.6E+7
450 -1.6E+6 3.0E+13 -2.5E+7
400 29450 2.2E+11 3.6E+6
350 -3291 4.1E+9 -185934
300 3611 1.6E+9 -95445
250 2967 6.8E+8 66743
200 4047 2.3E+8 39288
150 1717 8.6E+7 21346
100 803 1.9E+7 10518
80 1123 8.4E+6 -4863
60 1119 3.1E+6 4677
50 502 1.5E+6 -2222
40 432 728856 2152
30 84 204651 -987
20 30 12810 338
15 28 2446 122
10 7 454 49
8 -2 196 24
6 -9 23 0
4 -10 5 -13
2 -8 0 -8
Table 4. Clustering Algorithm using ICMP Timestamp Data
As in Table 3 above, only some of the 504 steps are shown in Table 4.
The distinguishing feature of the data in Table 4 is that the raw
data are much more noisy - only some 30 host clocks are closer than
one second from the reference clock, while half were further than one
minute and over 100 further than one hour from it. The fact that the
algorithm converged to within 8 msec of the reference time under
these conditions should be considered fairly remarkable in view of
the probability that many of the outliers discarded are almost
certainly due to defective protocol implementations. Additional
information on these experiments is presented in Appendix A.
Mills [Page 8]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
4. Application to Time-Synchronization Data
A variation of the above algorithms can be used to improve the offset
estimates from a single clock by discarding noise samples produced by
occasional retransmissions in the network, for example. A set of n
independent samples is obtained by polling the clock. Then, a
majority-subset table is used to compute the m(j) and s(j) statistics
and the maximum-likelihood estimate determined as above. For this
purpose the majority-subset table could include larger subsets as
well. In this manner the maximum-likelihood estimates from each of
several clocks can be determined and used in the algorithm above.
In order to test the effectiveness of this algorithm, a set of
experiments was performed using two WIDEBAND/EISN gateways equipped
with WWVB radio clocks and connected to the ARPANET. These
experiments were designed to determine the limits of accuracy when
comparing these clocks via ARPANET paths. One of the gateways
(ISI-MCON-GW) is located at the Information Sciences Institute near
Los Angeles, while the other (LL-GW) is located at Lincoln
Laboratories near Boston. Both gateways consist of PDP11/44
computers running the EPOS operating system and clock-interface
boards with oscillators phase-locked to the WWVB clock.
The clock indications of the WIDEBAND/EISN gateways were compared
with the DCNet WWVB reference clock using ICMP Timestamp messages,
which record the individual timestamps with a precision of a
millisecond. However, the path over the ARPANET between these
gateways and the measurement host can introduce occasional
measurement errors as large as several seconds. In principle the
effect of these errors can be minimized by using a large sample
population; however, use of the above algorithms allows higher
accuracies to be obtained with far fewer samples.
Measurements were made separately with each of the two gateways by
sending an ICMP Timestamp Request message from the ARPANET address of
DCN1 to the ARPANET address of the gateway and computing the
round-trip delay and clock offset from the ICMP Timestamp Reply
message. This process was continued for 1000 message exchanges,
which took from seven minutes to several hours, depending on the
sample interval selected.
The tables below summarize the results of both the majority-subset
and clustering algorithms applied to the data from three experiments,
one with ISI-MCON-GW and two with LL-GW. The ISI-MCON-GW and LL-GW
(a) experiments were designed to determine the limits of accuracy
when using a continuous sequence of request/reply volleys, which
Mills [Page 9]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
resulted in over two samples per second. The remaining LL-GW (b)
experiment was designed to determine the limits of accuracy using a
much lower rate of about one sample every ten seconds.
For each of the three experiments two tables are shown, one using the
majority-subset algorithm and the other the clustering algorithm. The
two rows of the majority-subset tables show the statistics derived
both from the raw data and from the filtered data processed by a
C(5,3) majority-subset algorithm. In all cases the extrema and
variance are dramatically less for the filtered data than the raw
data, lending credence to the conjecture that the mean statistic for
the filtered data is probably a good maximum-likelihood estimator of
the true offset.
Mean Var Max Min
--------------------------------------------
Raw data 637 2.1E+7 32751 -1096
C(5,3) -15 389 53 -103
Table 5. ISI-MCON-GW Majority-Subset Algorithm
Size Mean Var Discard
-------------------------------
1000 637 2.1E+7 32751
990 313 1.0E+7 32732
981 15 1.0E+6 32649
980 -18 2713 -1096
970 -15 521 -122
960 -15 433 -97
940 -15 332 -75
900 -15 239 26
800 -15 141 12
700 -16 87 5
600 -17 54 -31
500 -16 33 -5
400 -18 18 -9
300 -19 7 -12
200 -19 2 -21
100 -18 0 -19
1 -17 0 -17
Table 6. ISI-MCON-GW Clustering Algorithm
Mills [Page 10]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
Mean Dev Max Min
--------------------------------------------
Raw data 566 1.8E+7 32750 -143
C(5,3) -23 81 14 -69
Table 7. LL-GW (a) Majority-Subset Algorithm
Size Mean Var Discard
-------------------------------
1000 566 1.8E+7 32750
990 242 8.5E+6 32726
983 10 1.0E+6 32722
982 -23 231 -143
980 -23 205 -109
970 -22 162 29
960 -23 128 13
940 -23 105 -51
900 -24 89 1
800 -25 49 -9
700 -26 31 -36
600 -26 21 -34
500 -27 14 -20
400 -29 7 -23
300 -30 3 -33
200 -29 1 -27
100 -29 0 -28
1 -29 0 -29
Table 8. LL-GW (a) Clustering Algorithm
Mean Dev Max Min
--------------------------------------------
Raw data 378 2.1E+7 32760 -32758
C(5,3) -21 1681 329 -212
Table 9. LL-GW (b) Majority-Subset Algorithm
Mills [Page 11]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
Size Mean Var Discard
-------------------------------
1000 377 2.1E+7 -32758
990 315 1.0E+7 32741
981 18 1.1E+6 32704
980 -16 16119 -1392
970 -17 5382 554
960 -21 3338 311
940 -24 2012 168
900 -22 1027 -137
800 -23 430 -72
700 -23 255 -55
600 -22 167 -45
500 -22 109 -40
400 -21 66 -6
300 -18 35 -29
200 -17 15 -23
100 -19 3 -15
50 -21 0 -19
20 -21 0 -21
10 -20 0 -20
1 -20 0 -20
Table 10. LL-GW (b) Clustering Algorithm
The rows of the clustering tables show the result of selected steps
in the algorithm as it discards samples furthest from the mean. The
first twenty steps or so discard samples with gross errors over 30
seconds. These samples turned out to be due to a defect in the
timestamping procedure implemented in the WIDEBAND/EISN gateway code
which caused gross errors in about two percent of the ICMP Timestamp
Reply messages. These samples were left in the raw data as received
in order to determine how the algorithms would behave in such extreme
cases. As apparent from the tables, both the majority-subset and
clustering algorithms effectively coped with the situation.
In the statement of the clustering algorithm the terminating
condition was specified as when only a single sample is left in the
sample set. However, it is not necessary to proceed that far. For
instance, it is known from the design of the experiment and the
reference clocks that accuracies better than about ten milliseconds
are probably unrealistic. A rough idea of the accuracy of the mean
is evident from the deviation, computed as the square root of the
variance. Thus, attempts to continue the algorithm beyond the point
where the variance drops below 100 or so are probably misguided.
This occurs when between 500 and 900 samples remain in the sample
Mills [Page 12]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
set, depending upon the particular experiment. Note that in any case
between 300 and 700 samples fall within ten milliseconds of the final
estimate, depending on experiment.
Comparing the majority-subset and clustering algorithms on the basis
of variance reveals the interesting observation that the result of
the C(5,3) majority-subset algorithm is equivalent to the clustering
algorithm when between roughly 900 and 950 samples remain in the
sample set. This together with the moderately high variance in the
ISI-MCON-GW and LL-GW (b) cases suggests a C(6,4) or even C(7,4)
algorithm might yield greater accuracies.
5. Summary and Conclusions
The principles of maximum-likelihood estimation are well known and
widely applied in communication electronics. In this note two
algorithms using these principles are proposed, one based on
majority-subset techniques appropriate for cases involving small
numbers of samples and the other based on clustering techniques
appropriate for cases involving large numbers of samples.
The algorithms were tested on raw data collected with Internet hosts
and gateways over ARPANET paths for the purpose of setting a local
host clock with respect to a remote reference while maintaining
accuracies in the order of ten milliseconds. The results demonstrate
the effectiveness of these algorithms in detecting and discarding
glitches due to hardware or software failure or operator mistakes.
They also demonstrate that time synchronization can be maintained
across the ARPANET in the order of ten milliseconds in spite of
glitches many times the mean roundtrip delay.
The results point to the need for an improved time-synchronization
protocol combining the best features of the ICMP Timestamp message
[6] and UDP Time protocol [7]. Among the features suggested for this
protocol are the following:
1. The protocol should be based on UDP, which provides the
flexibility to handle simultaneous, multiplexed queries and
responses.
2. The message format should be based on the ICMP Timestamp
message format, which provides the arrival and departure times
at the server and allows the client to calculate the roundtrip
delay and offset accurately.
Mills [Page 13]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
3. The data format should be based on the UDP Time format, which
specifies 32-bit time in seconds since 1 January 1900, but
extended additional bits for the fractional part of a second.
4. Provisions to specify the expected accuracy should be included
along with information about the reference clock or
synchronizing mechanism, as well as the expected drift rate
and the last time the clock was set or synchronized.
The next step should be formulating an appropriate protocol with the
above features, together with implementation and test in the Internet
environment. Future development should result in a distributed,
symmetric protocol, similar perhaps to those described in [1], for
distributing highly reliable timekeeping information using a
hierarchical set of trusted clocks.
6. References
1. Lindsay, W.C., and A.V. Kantak. Network synchronization of
random signals. IEEE Trans. Comm. COM-28, 8 (August 1980),
1260-1266.
2. Mills, D.L. Time Synchronization in DCNET Hosts. DARPA Internet
Project Report IEN-173, COMSAT Laboratories, February 1981.
3. Mills, D.L. DCNET Internet Clock Service. DARPA Network Working
Group Report RFC-778, COMSAT Laboratories, April 1981.
4. Mills, D.L. Internet Delay Experiments. DARPA Network Working
Group Report RFC-889, M/A-COM Linkabit, December 1983.
5. Mills, D.L. DCN Local-Network Protocols. DARPA Network Working
Group Report RFC-891, M/A-COM Linkabit, December 1983.
6. Postel, J. Internet Control Message Protocol. DARPA Network
Working Group Report RFC-792, USC Information Sciences Institute,
September 1981.
7. Postel, J. Time Protocol. DARPA Network Working Group Report
RFC-868, USC Information Sciences Institute, May 1983.
8. Postel, J. Daytime Protocol. DARPA Network Working Group Report
RFC-867, USC Information Sciences Institute, May 1983.
9. Su, Z. A Specification of the Internet Protocol (IP) Timestamp
Option. DARPA Network Working Group Report RFC-781. SRI
International, May 1981.
Mills [Page 14]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
10. Marzullo, K., and S. Owicki. Maintaining the Time in a
Distributed System. ACM Operating Systems Review 19, 3 (July
1985), 44-54.
11. Mills, D.L. Experiments in Network Clock Synchronization. DARPA
Network Working Group Report RFC-957, M/A-COM Linkabit, September
1985.
12. Mills, D.L. Network Time Protocol (NTP). DARPA Network Working
Group Report RFC-958, M/A-COM Linkabit, September 1985.
Appendix A.
Experimental Determination of Internet Host Clock Accuracies
Following is a summary of the results of three experiments designed
to reveal the accuracies of various Internet host clocks. The first
experiment uses the UDP Time protocol, which is limited in precision
to one second, while the second uses the ICMP Timestamp message,
which extends the precision to one millisecond. In the third
experiment the results indicated by UDP and ICMP are compared. In
the UDP Time protocol time is indicated as a 32-bit field in seconds
past 0000 UT on 1 January 1900, while in the ICMP Timestamp message
time is indicated as a 32-bit field in milliseconds past 0000 UT of
each day.
All experiments described herein were conducted from Internet host
DCN6.ARPA, which is normally synchronized to a WWV radio clock. In
order to improve accuracy during the experiments, the DCN6.ARPA host
was resynchronized to a WWVB radio clock. As the result of several
experiments with other hosts equipped with WWVB and WWV radio clocks
and GOES satellite clocks, it is estimated that the maximum
measurement error in the following experiments is less than about 30
msec relative to standard NBS time determined at the Boulder/Fort
Collins transmitting sites.
A1. UDP Time Protocol Experiment
In the first experiment four UDP Time protocol requests were sent
at about three-second intervals to each of the 1775 hosts listed
in the NIC Internet host table. A total of 555 samples were
received from 163 hosts and compared with a local reference based
on a WWVB radio clock, which is known to be accurate to within a
few milliseconds. Not all of these hosts were listed as
supporting the UDP Time protocol in the NIC Internet host table,
while some that were listed as supporting this protocol either
failed to respond or responded with various error messages.
Mills [Page 15]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
In the following table "Host" is the canonical name of the host
and "Count" the number of replies received. The remaining data
represent the time offset, in seconds, necessary to correct the
local (reference) clock to agree with the host cited. The "Max"
and "Min" represent the maximum and minimum of these offsets,
while "Mean" represents the mean value and "Var" the variance, all
rounded to the nearest second.
Host Count Max Min Mean Var
-----------------------------------------------------------
BBN-CLXX.ARPA 4 -11 -12 -11 0
BBN-KIWI.ARPA 4 -11 -12 -11 0
BBN-META.ARPA 4 -11 -12 -11 0
BBNA.ARPA 1 22 22 22 0
BBNG.ARPA 4 87 87 87 0
BELLCORE-CS-GW.ARPA 3 72 71 71 0
BLAYS.PURDUE.EDU 2 -1 -1 -1 0
CMU-CC-TE.ARPA 4 -94 -95 -94 0
CMU-CS-C.ARPA 3 6 5 5 0
CMU-CS-CAD.ARPA 4 -37 -37 -37 0
CMU-CS-CFS.ARPA 3 -42 -43 -42 0
CMU-CS-G.ARPA 3 -30 -31 -30 0
CMU-CS-GANDALF.ARPA 3 -42 -43 -42 0
CMU-CS-H.ARPA 4 -36 -37 -36 0
CMU-CS-IUS.ARPA 3 -44 -45 -44 0
CMU-CS-IUS2.ARPA 3 -44 -44 -44 0
CMU-CS-K.ARPA 3 -31 -31 -31 0
CMU-CS-SAM.ARPA 4 -74 -75 -74 0
CMU-CS-SPEECH.ARPA 4 -39 -40 -39 0
CMU-CS-SPEECH2.ARPA 4 -49 -50 -49 0
CMU-CS-SPICE.ARPA 4 -131 -132 -131 0
CMU-CS-THEORY.ARPA 4 -36 -37 -36 0
CMU-CS-UNH.ARPA 4 -44 -45 -44 0
CMU-CS-VLSI.ARPA 4 -66 -66 -66 0
CMU-RI-ARM.ARPA 3 -41 -41 -41 0
CMU-RI-CIVE.ARPA 3 -44 -45 -44 0
CMU-RI-FAS.ARPA 4 -27 -28 -27 0
CMU-RI-ISL1.ARPA 4 -18 -19 -18 0
CMU-RI-ISL3.ARPA 3 -49 -50 -49 0
CMU-RI-LEG.ARPA 3 -33 -33 -33 0
CMU-RI-ML.ARPA 4 42 42 42 0
CMU-RI-ROVER.ARPA 4 -48 -49 -48 0
CMU-RI-SENSOR.ARPA 2 -40 -41 -40 0
CMU-RI-VI.ARPA 3 -65 -65 -65 0
COLUMBIA.ARPA 1 8 8 8 0
CU-ARPA.CS.CORNELL.EDU 4 5 3 4 0
CYPRESS.ARPA 4 2 1 1 0
Mills [Page 16]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
DCN1.ARPA 4 0 0 0 0
DCN5.ARPA 4 0 0 0 0
DCN6.ARPA 4 0 0 0 0
DCN7.ARPA 4 -1 -1 0 0
DCN9.ARPA 4 -3 -3 -3 0
DEVVAX.TN.CORNELL.EDU 2 3659 3658 3658 0
ENEEVAX.ARPA 4 73 72 72 0
FORD-WDL1.ARPA 4 -59 -60 -59 0
FORD1.ARPA 4 0 0 0 0
GUENEVERE.PURDUE.EDU 3 1 0 0 0
GVAX.CS.CORNELL.EDU 4 -18 -18 -18 0
IM4U.ARPA 4 -6 -6 -6 0
IPTO-FAX.ARPA 4 0 0 0 0
KANKIN.ARPA 4 -3 -4 -3 0
MERLIN.PURDUE.EDU 2 3 3 3 0
MIT-ACHILLES.ARPA 4 16 16 16 0
MIT-AGAMEMNON.ARPA 4 -63 -64 -63 0
MIT-ANDROMACHE.ARPA 4 -28 -28 -28 0
MIT-APHRODITE.ARPA 4 -7 -8 -7 0
MIT-APOLLO.ARPA 4 -8 -9 -8 0
MIT-ARES.ARPA 4 -25 -26 -25 0
MIT-ARTEMIS.ARPA 4 -34 -35 -34 0
MIT-ATHENA.ARPA 4 -37 -37 -37 0
MIT-ATLAS.ARPA 4 -26 -26 -26 0
MIT-CASTOR.ARPA 4 -35 -35 -35 0
MIT-DAFFY-DUCK.ARPA 2 -72 -73 -72 0
MIT-DEMETER.ARPA 4 -28 -29 -28 0
MIT-GOLDILOCKS.ARPA 1 -20 -20 -20 0
MIT-HECTOR.ARPA 4 -23 -24 -23 0
MIT-HELEN.ARPA 4 6 5 5 0
MIT-HERA.ARPA 4 -34 -35 -34 0
MIT-HERACLES.ARPA 4 -36 -36 -36 0
MIT-JASON.ARPA 4 -36 -37 -36 0
MIT-MENELAUS.ARPA 4 -32 -33 -32 0
MIT-MULTICS.ARPA 3 25 23 24 0
MIT-ODYSSEUS.ARPA 4 20 19 19 0
MIT-ORPHEUS.ARPA 4 -34 -35 -34 0
MIT-PARIS.ARPA 4 -35 -35 -35 0
MIT-POSEIDON.ARPA 4 -39 -41 -40 0
MIT-PRIAM.ARPA 4 -24 -25 -24 0
MIT-REAGAN.ARPA 4 115 115 115 0
MIT-THESEUS.ARPA 4 -43 -44 -43 0
MIT-TRILLIAN.ARPA 4 -38 -39 -38 0
MIT-TWEETY-PIE.ARPA 3 106 105 105 0
MIT-ZERMATT.ARPA 4 -75 -76 -75 0
MIT-ZEUS.ARPA 4 -37 -39 -38 0
MOL.ARPA 2 -3 -3 -3 0
Mills [Page 17]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
MUNGO.THINK.COM 4 -1 -1 -1 0
NETWOLF.ARPA 4 158 157 157 0
ORBIT.ARPA 3 -4 -5 -4 0
OSLO-VAX.ARPA 3 3729 3727 3728 1
PATCH.ARPA 1 18 18 18 0
RADC-MULTICS.ARPA 4 -14 -15 -14 0
RICE-ZETA.ARPA 1 -31 -31 -31 0
RICE.ARPA 1 7 7 7 0
ROCHESTER.ARPA 4 -18 -18 -18 0
ROCK.THINK.COM 4 2 2 2 0
SCRC-QUABBIN.ARPA 4 -100 -100 -100 0
SCRC-RIVERSIDE.ARPA 4 -128 -128 -128 0
SCRC-STONY-BROOK.ARPA 4 -100 -100 -100 0
SCRC-VALLECITO.ARPA 4 -57 -57 -57 0
SCRC-YUKON.ARPA 4 -65 -65 -65 0
SEBASTIAN.THINK.COM 4 -14 -15 -14 0
SEISMO.CSS.GOV 3 -1 -1 0 0
SRI-BISHOP.ARPA 4 -40 -41 -40 0
SRI-DARWIN.ARPA 2 -29 -30 -29 0
SRI-HUXLEY.ARPA 2 -28 -29 -28 0
SRI-KIOWA.ARPA 4 -29 -30 -29 0
SRI-LASSEN.ARPA 3 -11 -12 -11 0
SRI-MENDEL.ARPA 4 74 73 73 0
SRI-PINCUSHION.ARPA 4 -50 -51 -50 0
SRI-RITTER.ARPA 4 -23 -24 -23 0
SRI-TIOGA.ARPA 4 127 127 127 0
SRI-UNICORN.ARPA 4 -38486 -38486 -38486 0
SRI-WHITNEY.ARPA 4 -24 -24 -24 0
SRI-YOSEMITE.ARPA 4 -26 -27 -26 0
SU-AIMVAX.ARPA 2 -54 -55 -54 0
SU-COYOTE.ARPA 1 14 14 14 0
SU-CSLI.ARPA 4 -1 -1 -1 0
SU-PSYCH.ARPA 1 -52 -52 -52 0
SU-SAFE.ARPA 1 -60 -60 -60 0
SU-SIERRA.ARPA 4 -53 -53 -53 0
SU-SUSHI.ARPA 4 -105 -106 -105 0
SU-WHITNEY.ARPA 2 -14 -14 -14 0
TESLA.EE.CORNELL.EDU 3 -2 -3 -2 0
THORLAC.THINK.COM 4 -20 -20 -20 0
TRANTOR.ARPA 4 4 3 3 0
TZEC.ARPA 4 -6 -7 -6 0
UBALDO.THINK.COM 4 -13 -13 -13 0
UCI-CIP.ARPA 2 -566 -567 -566 0
UCI-CIP2.ARPA 2 -175 -175 -175 0
UCI-CIP3.ARPA 2 -89 -90 -89 0
UCI-CIP4.ARPA 2 -51 -51 -51 0
UCI-CIP5.ARPA 2 -26 -28 -27 1
Mills [Page 18]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
UCI-ICSA.ARPA 2 -24 -24 -24 0
UCI-ICSC.ARPA 1 0 0 0 0
UCI-ICSD.ARPA 1 -24 -24 -24 0
UCI-ICSE.ARPA 1 -10 -10 -10 0
UDEL-DEWEY.ARPA 1 88 88 88 0
UDEL-MICRO.ARPA 2 64 64 64 0
UIUC.ARPA 4 105 103 104 0
UIUCDCSB.ARPA 4 65 65 65 0
UMD1.ARPA 4 0 0 0 0
UMICH1.ARPA 4 -1 -1 0 0
UO.ARPA 4 -2 -3 -2 0
USC-ISI.ARPA 4 -45 -45 -45 0
USC-ISIC.ARPA 4 28 26 27 0
USC-ISID.ARPA 4 26 25 25 0
USC-ISIE.ARPA 4 -53 -54 -53 0
USC-ISIF.ARPA 4 -29 -29 -29 0
USGS2-MULTICS.ARPA 3 75 74 74 0
UT-ALAMO.ARPA 4 22 22 22 0
UT-BARKLEY.ARPA 4 57 56 56 0
UT-EMIL.ARPA 4 29 28 28 0
UT-GOTTLOB.ARPA 4 42 41 41 0
UT-HASKELL.ARPA 4 6 6 6 0
UT-JACQUES.ARPA 4 21 20 20 0
UT-SALLY.ARPA 3 1 0 0 0
VALENTINE.THINK.COM 4 -10 -11 -10 0
WENCESLAS.THINK.COM 4 -2 -3 -2 0
XAVIER.THINK.COM 4 -14 -14 -14 0
XEROX.ARPA 4 0 0 0 0
YAXKIN.ARPA 3 -4 -5 -4 0
YON.THINK.COM 4 -11 -12 -11 0
ZAPHOD.PURDUE.EDU 4 -230 -231 -230 0
ZOTZ.ARPA 4 17 16 16 0
Table A1. UDP Host Clock Offsets for Various Internet Hosts
The above list includes several host clocks known to be
synchronized to various radio clocks, including DCN1.ARPA (WWVB),
DCN6.ARPA (WWV) and FORD1.ARPA (GOES). Under normal radio
receiving conditions these hosts should be accurate to well within
a second relative to NBS standard time. Certain other host clocks
are synchronized to one of these hosts using protocols described
in RFC-891, including DCN5.ARPA, DCN7.ARPA and UMD1.ARPA
(synchronized to DCN1.ARPA) and UMICH1.ARPA (synchronized to
FORD1.ARPA). It is highly likely, but not confirmed, that several
other hosts with low offsets derive local time from one of these
hosts or from other radio clocks.
Mills [Page 19]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
The raw statistics computed from the weighted data indicate a mean
of -261 sec, together with a maximum of 3729 sec and a minimum of
-38486 sec. Obviously, setting a local clock on the basis of
these statistics alone would result in a gross error.
A closer look at the distribution of the data reveals some
interesting features. Table A2 is a histogram showing the
distribution within a few seconds of reference time. In this and
following tables, "Offset" is in seconds and indicates the
lower-valued corner of the histogram bin, which extends to the
next higher value, while "Count" indicates the number of samples
falling in that bin.
Offset Count Offset Count
------------- -------------
0 sec 13 (continued)
1 1 -1 3
2 1 -2 3
3 2 -3 3
4 1 -4 2
5 2 -5 0
6 1 -6 2
7 1 -7 1
8 1 -8 1
9 0 -9 0
> 9 30 < -9 95
Table A2. Offset Distribution < 9 sec
A total of 16 of the 163 host clocks are within a second in
accuracy, while a total of 125 are off more than ten seconds. It
is considered highly likely that most of the 16 host clocks within
a second in offset are synchronized directly or indirectly to a
radio clock. Table A3 is a histogram showing the distribution over
a larger scale.
Mills [Page 20]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
Offset Count Offset Count
------------- -------------
0 sec 35 (continued)
30 3 -30 50
60 8 -60 42
90 3 -90 8
120 1 -120 4
150 1 -150 2
180 0 -180 1
210 0 -210 0
240 0 -240 1
270 0 -270 0
> 270 2 < -270 2
Table A3. Offset Distribution < 270 sec
A total of 138 of the 163 host clocks are within a minute in
accuracy, while a total of four host clocks are off more than 4.5
minutes. It is considered likely that most host clocks, with the
exception of the 16 identified above as probably synchronized to a
radio clock, are set manually by an operator. Inspection of the
raw data shows some hosts to be very far off; for instance,
SRI-UNICORN.ARPA is off more than ten hours. Note the interesting
skew in the data, which show that most host clocks are set slow
relative to standard time.
A2. ICMP Timestamp Messages Experiment
The the second experiment four ICMP Timestamp messages were sent
at about three-second intervals to each of the 1775 hosts and 110
gateways listed in the NIC Internet host table. A total of 1910
samples were received from 504 hosts and gateways and compared
with a local reference based on a WWVB radio clock, which is known
to be accurate to within a few milliseconds. Support for the ICMP
Timestamp messages is optional in the DoD Internet protocol suite,
so it is not surprising that most hosts and gateways do not
support it. Moreover, bugs are known to exist in several widely
distributed implementations of this feature. The situation proved
an interesting and useful robustness test for the clustering
algorithm described in the main body of this note.
While the complete table of ICMP offsets by host is too large to
reproduce here, the following Tables A4 through A7 show the
interesting characteristics of the distribution. The raw
statistics computed from the weighted data indicate a mean of
-2.8E+6 msec, together with a maximum of 8.6E+7 msec and a minimum
of -8.6E+7 msec. Setting a local clock on the basis of these
Mills [Page 21]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
statistics alone would be ridiculous; however, as described in the
main body of this note, use of the clustering algorithm improves
the estimate to within 8 msec of the correct value. The apparent
improvement of about six orders in magnitude is so remarkable as
to require a closer look at the distributions.
The reasons for the remarkable success of the clustering algorithm
are apparent from closer examination of the sequence of histograms
shown in Tables A4 through A7. Table A4 shows the distribution in
the scale of hours, from which it is evident that 80 percent of
the samples lie in a one-hour band either side of zero offset;
but, strangely enough, there is a significant dispersion in
samples outside of this band, especially in the negative region.
It is almost certain that most or all of the latter samples
represent defective ICMP Timestamp implementations. Note that
invalid timestamps and those with the high-order bit set
(indicating unknown or nonstandard time) have already been
excluded from these data.
Offset Count Offset Count
------------- -------------
0 hr 204 (continued)
1 10 -1 194
2 0 -2 0
3 0 -3 2
4 0 -4 17
5 0 -5 10
6 0 -6 1
7 0 -7 22
8 0 -8 20
9 0 -9 0
> 9 0 < -9 13
Table A4. ICMP Offset Distribution < 9 hours
Table A5 shows the distribution compressed to the range of 4.5
minutes. About half of the 370 samples remaining after the
outliers beyond 4.5 minutes are excluded lie in the band 30
seconds either side of zero offset, with a gradual tapering off to
the limits of the table. This type of distribution would be
expected in the case of host clocks set manually by an operator.
Mills [Page 22]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
Offset Count Offset Count
------------- -------------
0 sec 111 (continued)
30 25 -30 80
60 26 -60 28
90 13 -90 18
120 7 -120 19
150 5 -150 9
180 3 -180 10
210 3 -210 6
240 1 -240 2
270 2 -270 2
> 270 29 < -270 105
Table A5. ICMP Offset Distribution < 270 sec
Table A6 shows the distribution compressed to the range of 27
seconds. About 29 percent of the 188 samples remaining after the
outliers beyond 27 seconds are excluded lie in the band 3 seconds
either side of zero offset, with a gradual but less pronounced
tapering off to the limits of the table. This type of
distribution is consistent with a transition region in which some
clocks are set manually and some by some kind of protocol
interaction with a reference clock. A fair number of the clocks
showing offsets in the 3-27 second range have probably been set
using the UDP Time protocol at some time in the past, but have
wandered away as the result of local-oscillator drifts.
Offset Count Offset Count
------------- -------------
0 sec 32 (continued)
3 15 -3 22
6 9 -6 12
9 6 -9 8
12 13 -12 8
15 5 -15 5
18 8 -18 9
21 8 -21 7
24 9 -24 3
27 6 -27 3
> 27 114 < -27 202
Table A6. ICMP Offset Distribution < 27 sec
Finally, Table A7 shows the distribution compressed to the range
of 0.9 second. Only 30 of the original 504 samples have survived
and only 12 of these are within a band 0.1 seconds either side of
Mills [Page 23]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
zero offset. The latter include those clocks continuously
synchronized to a radio clock, such as the DCNet clocks, some
FORDnet and UMDnet clocks and certain others.
Offset Count Offset Count
------------- -------------
0 sec 6 (continued)
.1 3 -.1 6
.2 1 -.2 3
.3 1 -.3 0
.4 0 -.4 0
.5 1 -.5 2
.6 0 -.6 0
.7 1 -.7 0
.8 4 -.8 2
.9 0 -.9 0
> .9 208 < -.9 266
Table A7. ICMP Offset Distribution < .9 sec
The most important observation that can be made about the above
histograms is the pronounced central tendency in all of them, in
spite of the scale varying over six orders of magnitude. Thus, a
clustering algorithm which operates to discard outliers from the
mean will reliably converge on a maximum-likelihood estimate close
to the actual value.
A3. Comparison of UDP and ICMP Time
The third experiment was designed to assess the accuracies
produced by the various host implementations of the UDP Time
protocol and ICMP Timestamp messages. For each of the hosts
responding to the UDP Time protocol in the first experiment a
separate test was conducted using both UDP and ICMP in the same
test, so as to minimize the effect of clock drift. Of the 162
hosts responding to UDP requests, 45 also responded to ICMP
requests with apparently correct time, but the remainder either
responded with unknown or nonstandard ICMP time (29) or failed to
respond to ICMP requests at all (88).
Table A8 shows both the UDP time (seconds) and ICMP time
(milliseconds) returned by each of the 45 hosts responding to both
UDP and ICMP requests. The data are ordered first by indicated
UDP offset and then by indicated ICMP offset. The seven hosts at
the top of the table are continuously synchronized, directly or
indirectly to a radio clock, as described earlier under the first
Mills [Page 24]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
experiment. It is probable, but not confirmed, that those hosts
below showing discrepancies of a second or less are synchronized
on occasion to one of these hosts.
Host UDP time ICMP time
-------------------------------------------------
DCN6.ARPA 0 sec 0 msec
DCN7.ARPA 0 0
DCN1.ARPA 0 -6
DCN5.ARPA 0 -7
UMD1.ARPA 0 8
UMICH1.ARPA 0 -21
FORD1.ARPA 0 31
TESLA.EE.CORNELL.EDU 0 132
SEISMO.CSS.GOV 0 174
UT-SALLY.ARPA -1 -240
CU-ARPA.CS.CORNELL.EDU -1 -514
UCI-ICSE.ARPA -1 -1896
UCI-ICSC.ARPA 1 2000
DCN9.ARPA -7 -6610
TRANTOR.ARPA 10 10232
COLUMBIA.ARPA 11 12402
GVAX.CS.CORNELL.EDU -12 -11988
UCI-CIP5.ARPA -15 -17450
RADC-MULTICS.ARPA -16 -16600
SU-WHITNEY.ARPA 17 17480
UCI-ICSD.ARPA -20 -20045
SU-COYOTE.ARPA 21 21642
MIT-MULTICS.ARPA 27 28265
BBNA.ARPA -34 -34199
UCI-ICSA.ARPA -37 -36804
ROCHESTER.ARPA -42 -41542
SU-AIMVAX.ARPA -50 -49575
UCI-CIP4.ARPA -57 -57060
SU-SAFE.ARPA -59 -59212
SU-PSYCH.ARPA -59 -58421
UDEL-MICRO.ARPA 62 63214
UIUCDCSB.ARPA 63 63865
BELLCORE-CS-GW.ARPA 71 71402
USGS2-MULTICS.ARPA 76 77018
BBNG.ARPA 81 81439
UDEL-DEWEY.ARPA 89 89283
UCI-CIP3.ARPA -102 -102148
UIUC.ARPA 105 105843
UCI-CIP2.ARPA -185 -185250
UCI-CIP.ARPA -576 -576386
OSLO-VAX.ARPA 3738 3739395
Mills [Page 25]
RFC 956 September 1985
Algorithms for Synchronizing Network Clocks
DEVVAX.TN.CORNELL.EDU 3657 3657026
PATCH.ARPA -86380 20411
IPTO-FAX.ARPA -86402 -1693
NETWOLF.ARPA 10651435 -62164450
Table A8. Comparison of UDP and ICMP Host Clock Offsets
Allowing for various degrees of truncation and roundoff abuse in
the various implementations, discrepancies of up to a second could
be expected between UDP and ICMP time. While the results for most
hosts confirm this, some discrepancies are far greater, which may
indicate defective implementations, excessive swapping delays and
so forth. For instance, the ICMP time indicated by UCI-CIP5.ARPA
is almost 2.5 seconds less than the UDP time.
Even though the UDP and ICMP times indicated by OSLO-VAX.ARPA and
DEVVAX.TN.CORNELL.EDU agree within nominals, the fact that they
are within a couple of minutes or so of exactly one hour early
(3600 seconds) lends weight to the conclusion they were
incorrectly set, probably by an operator who confused local summer
and standard time.
Something is clearly broken in the case of PATCH.ARPA,
IPTO-FAX.ARPA and NETWOLF.ARPA. Investigation of the PATCH.ARPA
and IPTO-FAX.ARPA reveals that these hosts were set by hand
accidently one day late (-86400 seconds), but otherwise close to
the correct time-of-day. Since the ICMP time rolls over at 2400
UT, the ICMP offset was within nominals. No explanation is
available for the obviously defective UDP and ICMP times indicated
by NETWOLF.ARPA, although it was operating within nominals at
least in the first experiment.
Mills [Page 26]