Network Working Group V. Paxson Request for Comments: 2330 Lawrence Berkeley National Lab Category: Informational G. Almes Advanced Network & Services J. Mahdavi M. Mathis Pittsburgh Supercomputer Center May 1998 Framework for IP Performance Metrics 1. Status of this Memo This memo provides information for the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited. 2. Copyright Notice Copyright (C) The Internet Society (1998). All Rights Reserved. Table of Contents 1. STATUS OF THIS MEMO.............................................1 2. COPYRIGHT NOTICE................................................1 3. INTRODUCTION....................................................2 4. CRITERIA FOR IP PERFORMANCE METRICS.............................3 5. TERMINOLOGY FOR PATHS AND CLOUDS................................4 6. FUNDAMENTAL CONCEPTS............................................5 6.1 Metrics......................................................5 6.2 Measurement Methodology......................................6 6.3 Measurements, Uncertainties, and Errors......................7 7. METRICS AND THE ANALYTICAL FRAMEWORK............................8 8. EMPIRICALLY SPECIFIED METRICS..................................11 9. TWO FORMS OF COMPOSITION.......................................12 9.1 Spatial Composition of Metrics..............................12 9.2 Temporal Composition of Formal Models and Empirical Metrics.13 10. ISSUES RELATED TO TIME........................................14 10.1 Clock Issues...............................................14 10.2 The Notion of "Wire Time"..................................17 11. SINGLETONS, SAMPLES, AND STATISTICS............................19 11.1 Methods of Collecting Samples..............................20 11.1.1 Poisson Sampling........................................21 11.1.2 Geometric Sampling......................................22 11.1.3 Generating Poisson Sampling Intervals...................22 Paxson, et. al. Informational [Page 1] RFC 2330 Framework for IP Performance Metrics May 1998 11.2 Self-Consistency...........................................24 11.3 Defining Statistical Distributions.........................25 11.4 Testing For Goodness-of-Fit................................27 12. AVOIDING STOCHASTIC METRICS....................................28 13. PACKETS OF TYPE P..............................................29 14. INTERNET ADDRESSES VS. HOSTS...................................30 15. STANDARD-FORMED PACKETS........................................30 16. ACKNOWLEDGEMENTS...............................................31 17. SECURITY CONSIDERATIONS........................................31 18. APPENDIX.......................................................32 19. REFERENCES.....................................................38 20. AUTHORS' ADDRESSES.............................................39 21. FULL COPYRIGHT STATEMENT.......................................40 3. Introduction The purpose of this memo is to define a general framework for particular metrics to be developed by the IETF's IP Performance Metrics effort, begun by the Benchmarking Methodology Working Group (BMWG) of the Operational Requirements Area, and being continued by the IP Performance Metrics Working Group (IPPM) of the Transport Area. We begin by laying out several criteria for the metrics that we adopt. These criteria are designed to promote an IPPM effort that will maximize an accurate common understanding by Internet users and Internet providers of the performance and reliability both of end- to-end paths through the Internet and of specific 'IP clouds' that comprise portions of those paths. We next define some Internet vocabulary that will allow us to speak clearly about Internet components such as routers, paths, and clouds. We then define the fundamental concepts of 'metric' and 'measurement methodology', which allow us to speak clearly about measurement issues. Given these concepts, we proceed to discuss the important issue of measurement uncertainties and errors, and develop a key, somewhat subtle notion of how they relate to the analytical framework shared by many aspects of the Internet engineering discipline. We then introduce the notion of empirically defined metrics, and finish this part of the document with a general discussion of how metrics can be 'composed'. The remainder of the document deals with a variety of issues related to defining sound metrics and methodologies: how to deal with imperfect clocks; the notion of 'wire time' as distinct from 'host time'; how to aggregate sets of singleton metrics into samples and Paxson, et. al. Informational [Page 2] RFC 2330 Framework for IP Performance Metrics May 1998 derive sound statistics from those samples; why it is recommended to avoid thinking about Internet properties in probabilistic terms (such as the probability that a packet is dropped), since these terms often include implicit assumptions about how the network behaves; the utility of defining metrics in terms of packets of a generic type; the benefits of preferring IP addresses to DNS host names; and the notion of 'standard-formed' packets. An appendix discusses the Anderson-Darling test for gauging whether a set of values matches a given statistical distribution, and gives C code for an implementation of the test. In some sections of the memo, we will surround some commentary text with the brackets {Comment: ... }. We stress that this commentary is only commentary, and is not itself part of the framework document or a proposal of particular metrics. In some cases this commentary will discuss some of the properties of metrics that might be envisioned, but the reader should assume that any such discussion is intended only to shed light on points made in the framework document, and not to suggest any specific metrics. 4. Criteria for IP Performance Metrics The overarching goal of the IP Performance Metrics effort is to achieve a situation in which users and providers of Internet transport service have an accurate common understanding of the performance and reliability of the Internet component 'clouds' that they use/provide. To achieve this, performance and reliability metrics for paths through the Internet must be developed. In several IETF meetings criteria for these metrics have been specified: + The metrics must be concrete and well-defined, + A methodology for a metric should have the property that it is repeatable: if the methodology is used multiple times under identical conditions, the same measurements should result in the same measurements. + The metrics must exhibit no bias for IP clouds implemented with identical technology, + The metrics must exhibit understood and fair bias for IP clouds implemented with non-identical technology, + The metrics must be useful to users and providers in understanding the performance they experience or provide, Paxson, et. al. Informational [Page 3] RFC 2330 Framework for IP Performance Metrics May 1998 + The metrics must avoid inducing artificial performance goals. 5. Terminology for Paths and Clouds The following list defines terms that need to be precise in the development of path metrics. We begin with low-level notions of 'host', 'router', and 'link', then proceed to define the notions of 'path', 'IP cloud', and 'exchange' that allow us to segment a path into relevant pieces. host A computer capable of communicating using the Internet protocols; includes "routers". link A single link-level connection between two (or more) hosts; includes leased lines, ethernets, frame relay clouds, etc. routerA host which facilitates network-level communication between hosts by forwarding IP packets. path A sequence of the form < h0, l1, h1, ..., ln, hn >, where n >= 0, each hi is a host, each li is a link between hi-1 and hi, each h1...hn-1 is a router. A pair is termed a 'hop'. In an appropriate operational configuration, the links and routers in the path facilitate network-layer communication of packets from h0 to hn. Note that path is a unidirectional concept. subpath Given a path, a subpath is any subsequence of the given path which is itself a path. (Thus, the first and last element of a subpath is a host.) cloudAn undirected (possibly cyclic) graph whose vertices are routers and whose edges are links that connect pairs of routers. Formally, ethernets, frame relay clouds, and other links that connect more than two routers are modelled as fully-connected meshes of graph edges. Note that to connect to a cloud means to connect to a router of the cloud over a link; this link is not itself part of the cloud. exchange A special case of a link, an exchange directly connects either a host to a cloud and/or one cloud to another cloud. cloud subpath A subpath of a given path, all of whose hosts are routers of a given cloud. Paxson, et. al. Informational [Page 4] RFC 2330 Framework for IP Performance Metrics May 1998 path digest A sequence of the form < h0, e1, C1, ..., en, hn >, where n >= 0, h0 and hn are hosts, each e1 ... en is an exchange, and each C1 ... Cn-1 is a cloud subpath. 6. Fundamental Concepts 6.1. Metrics In the operational Internet, there are several quantities related to the performance and reliability of the Internet that we'd like to know the value of. When such a quantity is carefully specified, we term the quantity a metric. We anticipate that there will be separate RFCs for each metric (or for each closely related group of metrics). In some cases, there might be no obvious means to effectively measure the metric; this is allowed, and even understood to be very useful in some cases. It is required, however, that the specification of the metric be as clear as possible about what quantity is being specified. Thus, difficulty in practical measurement is sometimes allowed, but ambiguity in meaning is not. Each metric will be defined in terms of standard units of measurement. The international metric system will be used, with the following points specifically noted: + When a unit is expressed in simple meters (for distance/length) or seconds (for duration), appropriate related units based on thousands or thousandths of acceptable units are acceptable. Thus, distances expressed in kilometers (km), durations expressed in milliseconds (ms), or microseconds (us) are allowed, but not centimeters (because the prefix is not in terms of thousands or thousandths). + When a unit is expressed in a combination of units, appropriate related units based on thousands or thousandths of acceptable units are acceptable, but all such thousands/thousandths must be grouped at the beginning. Thus, kilo-meters per second (km/s) is allowed, but meters per millisecond is not. + The unit of information is the bit. + When metric prefixes are used with bits or with combinations including bits, those prefixes will have their metric meaning (related to decimal 1000), and not the meaning conventional with computer storage (related to decimal 1024). In any RFC that defines a metric whose units include bits, this convention will be followed and will be repeated to ensure clarity for the reader. Paxson, et. al. Informational [Page 5] RFC 2330 Framework for IP Performance Metrics May 1998 + When a time is given, it will be expressed in UTC. Note that these points apply to the specifications for metrics and not, for example, to packet formats where octets will likely be used in preference/addition to bits. Finally, we note that some metrics may be defined purely in terms of other metrics; such metrics are call 'derived metrics'. 6.2. Measurement Methodology For a given set of well-defined metrics, a number of distinct measurement methodologies may exist. A partial list includes: + Direct measurement of a performance metric using injected test traffic. Example: measurement of the round-trip delay of an IP packet of a given size over a given route at a given time. + Projection of a metric from lower-level measurements. Example: given accurate measurements of propagation delay and bandwidth for each step along a path, projection of the complete delay for the path for an IP packet of a given size. + Estimation of a constituent metric from a set of more aggregated measurements. Example: given accurate measurements of delay for a given one-hop path for IP packets of different sizes, estimation of propagation delay for the link of that one-hop path. + Estimation of a given metric at one time from a set of related metrics at other times. Example: given an accurate measurement of flow capacity at a past time, together with a set of accurate delay measurements for that past time and the current time, and given a model of flow dynamics, estimate the flow capacity that would be observed at the current time. This list is by no means exhaustive. The purpose is to point out the variety of measurement techniques. When a given metric is specified, a given measurement approach might be noted and discussed. That approach, however, is not formally part of the specification. A methodology for a metric should have the property that it is repeatable: if the methodology is used multiple times under identical conditions, it should result in consistent measurements. Backing off a little from the word 'identical' in the previous paragraph, we could more accurately use the word 'continuity' to describe a property of a given methodology: a methodology for a given metric exhibits continuity if, for small variations in conditions, it Paxson, et. al. Informational [Page 6] RFC 2330 Framework for IP Performance Metrics May 1998 results in small variations in the resulting measurements. Slightly more precisely, for every positive epsilon, there exists a positive delta, such that if two sets of conditions are within delta of each other, then the resulting measurements will be within epsilon of each other. At this point, this should be taken as a heuristic driving our intuition about one kind of robustness property rather than as a precise notion. A metric that has at least one methodology that exhibits continuity is said itself to exhibit continuity. Note that some metrics, such as hop-count along a path, are integer- valued and therefore cannot exhibit continuity in quite the sense given above. Note further that, in practice, it may not be practical to know (or be able to quantify) the conditions relevant to a measurement at a given time. For example, since the instantaneous load (in packets to be served) at a given router in a high-speed wide-area network can vary widely over relatively brief periods and will be very hard for an external observer to quantify, various statistics of a given metric may be more repeatable, or may better exhibit continuity. In that case those particular statistics should be specified when the metric is specified. Finally, some measurement methodologies may be 'conservative' in the sense that the act of measurement does not modify, or only slightly modifies, the value of the performance metric the methodology attempts to measure. {Comment: for example, in a wide-are high-speed network under modest load, a test using several small 'ping' packets to measure delay would likely not interfere (much) with the delay properties of that network as observed by others. The corresponding statement about tests using a large flow to measure flow capacity would likely fail.} 6.3. Measurements, Uncertainties, and Errors Even the very best measurement methodologies for the very most well behaved metrics will exhibit errors. Those who develop such measurement methodologies, however, should strive to: Paxson, et. al. Informational [Page 7] RFC 2330 Framework for IP Performance Metrics May 1998 + minimize their uncertainties/errors, + understand and document the sources of uncertainty/error, and + quantify the amounts of uncertainty/error. For example, when developing a method for measuring delay, understand how any errors in your clocks introduce errors into your delay measurement, and quantify this effect as well as you can. In some cases, this will result in a requirement that a clock be at least up to a certain quality if it is to be used to make a certain measurement. As a second example, consider the timing error due to measurement overheads within the computer making the measurement, as opposed to delays due to the Internet component being measured. The former is a measurement error, while the latter reflects the metric of interest. Note that one technique that can help avoid this overhead is the use of a packet filter/sniffer, running on a separate computer that records network packets and timestamps them accurately (see the discussion of 'wire time' below). The resulting trace can then be analyzed to assess the test traffic, minimizing the effect of measurement host delays, or at least allowing those delays to be accounted for. We note that this technique may prove beneficial even if the packet filter/sniffer runs on the same machine, because such measurements generally provide 'kernel-level' timestamping as opposed to less-accurate 'application-level' timestamping. Finally, we note that derived metrics (defined above) or metrics that exhibit spatial or temporal composition (defined below) offer particular occasion for the analysis of measurement uncertainties, namely how the uncertainties propagate (conceptually) due to the derivation or composition. 7. Metrics and the Analytical Framework As the Internet has evolved from the early packet-switching studies of the 1960s, the Internet engineering community has evolved a common analytical framework of concepts. This analytical framework, or A- frame, used by designers and implementers of protocols, by those involved in measurement, and by those who study computer network performance using the tools of simulation and analysis, has great advantage to our work. A major objective here is to generate network characterizations that are consistent in both analytical and practical settings, since this will maximize the chances that non- empirical network study can be better correlated with, and used to further our understanding of, real network behavior. Paxson, et. al. Informational [Page 8] RFC 2330 Framework for IP Performance Metrics May 1998 Whenever possible, therefore, we would like to develop and leverage off of the A-frame. Thus, whenever a metric to be specified is understood to be closely related to concepts within the A-frame, we will attempt to specify the metric in the A-frame's terms. In such a specification we will develop the A-frame by precisely defining the concepts needed for the metric, then leverage off of the A-frame by defining the metric in terms of those concepts. Such a metric will be called an 'analytically specified metric' or, more simply, an analytical metric. {Comment: Examples of such analytical metrics might include: propagation time of a link The time, in seconds, required by a single bit to travel from the output port on one Internet host across a single link to another Internet host. bandwidth of a link for packets of size k The capacity, in bits/second, where only those bits of the IP packet are counted, for packets of size k bytes. routeThe path, as defined in Section 5, from A to B at a given time. hop count of a route The value 'n' of the route path. } Note that we make no a priori list of just what A-frame concepts will emerge in these specifications, but we do encourage their use and urge that they be carefully specified so that, as our set of metrics develops, so will a specified set of A-frame concepts technically consistent with each other and consonant with the common understanding of those concepts within the general Internet community. These A-frame concepts will be intended to abstract from actual Internet components in such a way that: + the essential function of the component is retained, + properties of the component relevant to the metrics we aim to create are retained, + a subset of these component properties are potentially defined as analytical metrics, and Paxson, et. al. Informational [Page 9] RFC 2330 Framework for IP Performance Metrics May 1998 + those properties of actual Internet components not relevant to defining the metrics we aim to create are dropped. For example, when considering a router in the context of packet forwarding, we might model the router as a component that receives packets on an input link, queues them on a FIFO packet queue of finite size, employs tail-drop when the packet queue is full, and forwards them on an output link. The transmission speed (in bits/second) of the input and output links, the latency in the router (in seconds), and the maximum size of the packet queue (in bits) are relevant analytical metrics. In some cases, such analytical metrics used in relation to a router will be very closely related to specific metrics of the performance of Internet paths. For example, an obvious formula (L + P/B) involving the latency in the router (L), the packet size (in bits) (P), and the transmission speed of the output link (B) might closely approximate the increase in packet delay due to the insertion of a given router along a path. We stress, however, that well-chosen and well-specified A-frame concepts and their analytical metrics will support more general metric creation efforts in less obvious ways. {Comment: for example, when considering the flow capacity of a path, it may be of real value to be able to model each of the routers along the path as packet forwarders as above. Techniques for estimating the flow capacity of a path might use the maximum packet queue size as a parameter in decidedly non-obvious ways. For example, as the maximum queue size increases, so will the ability of the router to continuously move traffic along an output link despite fluctuations in traffic from an input link. Estimating this increase, however, remains a research topic.} Note that, when we specify A-frame concepts and analytical metrics, we will inevitably make simplifying assumptions. The key role of these concepts is to abstract the properties of the Internet components relevant to given metrics. Judgement is required to avoid making assumptions that bias the modeling and metric effort toward one kind of design. {Comment: for example, routers might not use tail-drop, even though tail-drop might be easier to model analytically.} Finally, note that different elements of the A-frame might well make different simplifying assumptions. For example, the abstraction of a router used to further the definition of path delay might treat the router's packet queue as a single FIFO queue, but the abstraction of Paxson, et. al. Informational [Page 10] RFC 2330 Framework for IP Performance Metrics May 1998 a router used to further the definition of the handling of an RSVP- enabled packet might treat the router's packet queue as supporting bounded delay -- a contradictory assumption. This is not to say that we make contradictory assumptions at the same time, but that two different parts of our work might refine the simpler base concept in two divergent ways for different purposes. {Comment: in more mathematical terms, we would say that the A-frame taken as a whole need not be consistent; but the set of particular A-frame elements used to define a particular metric must be.} 8. Empirically Specified Metrics There are useful performance and reliability metrics that do not fit so neatly into the A-frame, usually because the A-frame lacks the detail or power for dealing with them. For example, "the best flow capacity achievable along a path using an RFC-2001-compliant TCP" would be good to be able to measure, but we have no analytical framework of sufficient richness to allow us to cast that flow capacity as an analytical metric. These notions can still be well specified by instead describing a reference methodology for measuring them. Such a metric will be called an 'empirically specified metric', or more simply, an empirical metric. Such empirical metrics should have three properties: + we should have a clear definition for each in terms of Internet components, + we should have at least one effective means to measure them, and + to the extent possible, we should have an (necessarily incomplete) understanding of the metric in terms of the A-frame so that we can use our measurements to reason about the performance and reliability of A-frame components and of aggregations of A-frame components. Paxson, et. al. Informational [Page 11] RFC 2330 Framework for IP Performance Metrics May 1998 9. Two Forms of Composition 9.1. Spatial Composition of Metrics In some cases, it may be realistic and useful to define metrics in such a fashion that they exhibit spatial composition. By spatial composition, we mean a characteristic of some path metrics, in which the metric as applied to a (complete) path can also be defined for various subpaths, and in which the appropriate A-frame concepts for the metric suggest useful relationships between the metric applied to these various subpaths (including the complete path, the various cloud subpaths of a given path digest, and even single routers along the path). The effectiveness of spatial composition depends: + on the usefulness in analysis of these relationships as applied to the relevant A-frame components, and + on the practical use of the corresponding relationships as applied to metrics and to measurement methodologies. {Comment: for example, consider some metric for delay of a 100-byte packet across a path P, and consider further a path digest of P. The definition of such a metric might include a conjecture that the delay across P is very nearly the sum of the corresponding metric across the exchanges (ei) and clouds (Ci) of the given path digest. The definition would further include a note on how a corresponding relation applies to relevant A-frame components, both for the path P and for the exchanges and clouds of the path digest.} When the definition of a metric includes a conjecture that the metric across the path is related to the metric across the subpaths of the path, that conjecture constitutes a claim that the metric exhibits spatial composition. The definition should then include: Paxson, et. al. Informational [Page 12] RFC 2330 Framework for IP Performance Metrics May 1998 + the specific conjecture applied to the metric, + a justification of the practical utility of the composition in terms of making accurate measurements of the metric on the path, + a justification of the usefulness of the composition in terms of making analysis of the path using A-frame concepts more effective, and + an analysis of how the conjecture could be incorrect. 9.2. Temporal Composition of Formal Models and Empirical Metrics In some cases, it may be realistic and useful to define metrics in such a fashion that they exhibit temporal composition. By temporal composition, we mean a characteristic of some path metric, in which the metric as applied to a path at a given time T is also defined for various times t0 < t1 < ... < tn < T, and in which the appropriate A-frame concepts for the metric suggests useful relationships between the metric applied at times t0, ..., tn and the metric applied at time T. The effectiveness of temporal composition depends: + on the usefulness in analysis of these relationships as applied to the relevant A-frame components, and + on the practical use of the corresponding relationships as applied to metrics and to measurement methodologies. {Comment: for example, consider a metric for the expected flow capacity across a path P during the five-minute period surrounding the time T, and suppose further that we have the corresponding values for each of the four previous five-minute periods t0, t1, t2, and t3. The definition of such a metric might include a conjecture that the flow capacity at time T can be estimated from a certain kind of extrapolation from the values of t0, ..., t3. The definition would further include a note on how a corresponding relation applies to relevant A-frame components. Note: any (spatial or temporal) compositions involving flow capacity are likely to be subtle, and temporal compositions are generally more subtle than spatial compositions, so the reader should understand that the foregoing example is intentionally naive.} When the definition of a metric includes a conjecture that the metric across the path at a given time T is related to the metric across the path for a set of other times, that conjecture constitutes a claim that the metric exhibits temporal composition. The definition should then include: Paxson, et. al. Informational [Page 13] RFC 2330 Framework for IP Performance Metrics May 1998 + the specific conjecture applied to the metric, + a justification of the practical utility of the composition in terms of making accurate measurements of the metric on the path, and + a justification of the usefulness of the composition in terms of making analysis of the path using A-frame concepts more effective. 10. Issues related to Time 10.1. Clock Issues Measurements of time lie at the heart of many Internet metrics. Because of this, it will often be crucial when designing a methodology for measuring a metric to understand the different types of errors and uncertainties introduced by imperfect clocks. In this section we define terminology for discussing the characteristics of clocks and touch upon related measurement issues which need to be addressed by any sound methodology. The Network Time Protocol (NTP; RFC 1305) defines a nomenclature for discussing clock characteristics, which we will also use when appropriate [Mi92]. The main goal of NTP is to provide accurate timekeeping over fairly long time scales, such as minutes to days, while for measurement purposes often what is more important is short-term accuracy, between the beginning of the measurement and the end, or over the course of gathering a body of measurements (a sample). This difference in goals sometimes leads to different definitions of terminology as well, as discussed below. To begin, we define a clock's "offset" at a particular moment as the difference between the time reported by the clock and the "true" time as defined by UTC. If the clock reports a time Tc and the true time is Tt, then the clock's offset is Tc - Tt. We will refer to a clock as "accurate" at a particular moment if the clock's offset is zero, and more generally a clock's "accuracy" is how close the absolute value of the offset is to zero. For NTP, accuracy also includes a notion of the frequency of the clock; for our purposes, we instead incorporate this notion into that of "skew", because we define accuracy in terms of a single moment in time rather than over an interval of time. A clock's "skew" at a particular moment is the frequency difference (first derivative of its offset with respect to true time) between the clock and true time. Paxson, et. al. Informational [Page 14] RFC 2330 Framework for IP Performance Metrics May 1998 As noted in RFC 1305, real clocks exhibit some variation in skew. That is, the second derivative of the clock's offset with respect to true time is generally non-zero. In keeping with RFC 1305, we define this quantity as the clock's "drift". A clock's "resolution" is the smallest unit by which the clock's time is updated. It gives a lower bound on the clock's uncertainty. (Note that clocks can have very fine resolutions and yet be wildly inaccurate.) Resolution is defined in terms of seconds. However, resolution is relative to the clock's reported time and not to true time, so for example a resolution of 10 ms only means that the clock updates its notion of time in 0.01 second increments, not that this is the true amount of time between updates. {Comment: Systems differ on how an application interface to the clock reports the time on subsequent calls during which the clock has not advanced. Some systems simply return the same unchanged time as given for previous calls. Others may add a small increment to the reported time to maintain monotone-increasing timestamps. For systems that do the latter, we do *not* consider these small increments when defining the clock's resolution. They are instead an impediment to assessing the clock's resolution, since a natural method for doing so is to repeatedly query the clock to determine the smallest non-zero difference in reported times.} It is expected that a clock's resolution changes only rarely (for example, due to a hardware upgrade). There are a number of interesting metrics for which some natural measurement methodologies involve comparing times reported by two different clocks. An example is one-way packet delay [AK97]. Here, the time required for a packet to travel through the network is measured by comparing the time reported by a clock at one end of the packet's path, corresponding to when the packet first entered the network, with the time reported by a clock at the other end of the path, corresponding to when the packet finished traversing the network. We are thus also interested in terminology for describing how two clocks C1 and C2 compare. To do so, we introduce terms related to those above in which the notion of "true time" is replaced by the time as reported by clock C1. For example, clock C2's offset relative to C1 at a particular moment is Tc2 - Tc1, the instantaneous difference in time reported by C2 and C1. To disambiguate between the use of the terms to compare two clocks versus the use of the terms to compare to true time, we will in the former case use the phrase "relative". So the offset defined earlier in this paragraph is the "relative offset" between C2 and C1. Paxson, et. al. Informational [Page 15] RFC 2330 Framework for IP Performance Metrics May 1998 When comparing clocks, the analog of "resolution" is not "relative resolution", but instead "joint resolution", which is the sum of the resolutions of C1 and C2. The joint resolution then indicates a conservative lower bound on the accuracy of any time intervals computed by subtracting timestamps generated by one clock from those generated by the other. If two clocks are "accurate" with respect to one another (their relative offset is zero), we will refer to the pair of clocks as "synchronized". Note that clocks can be highly synchronized yet arbitrarily inaccurate in terms of how well they tell true time. This point is important because for many Internet measurements, synchronization between two clocks is more important than the accuracy of the clocks. The is somewhat true of skew, too: as long as the absolute skew is not too great, then minimal relative skew is more important, as it can induce systematic trends in packet transit times measured by comparing timestamps produced by the two clocks. These distinctions arise because for Internet measurement what is often most important are differences in time as computed by comparing the output of two clocks. The process of computing the difference removes any error due to clock inaccuracies with respect to true time; but it is crucial that the differences themselves accurately reflect differences in true time. Measurement methodologies will often begin with the step of assuring that two clocks are synchronized and have minimal skew and drift. {Comment: An effective way to assure these conditions (and also clock accuracy) is by using clocks that derive their notion of time from an external source, rather than only the host computer's clock. (These latter are often subject to large errors.) It is further preferable that the clocks directly derive their time, for example by having immediate access to a GPS (Global Positioning System) unit.} Two important concerns arise if the clocks indirectly derive their time using a network time synchronization protocol such as NTP: + First, NTP's accuracy depends in part on the properties (particularly delay) of the Internet paths used by the NTP peers, and these might be exactly the properties that we wish to measure, so it would be unsound to use NTP to calibrate such measurements. + Second, NTP focuses on clock accuracy, which can come at the expense of short-term clock skew and drift. For example, when a host's clock is indirectly synchronized to a time source, if the synchronization intervals occur infrequently, then the host will sometimes be faced with the problem of how to adjust its current, incorrect time, Ti, with a considerably different, more accurate time it has just learned, Ta. Two general ways in which this is Paxson, et. al. Informational [Page 16] RFC 2330 Framework for IP Performance Metrics May 1998 done are to either immediately set the current time to Ta, or to adjust the local clock's update frequency (hence, its skew) so that at some point in the future the local time Ti' will agree with the more accurate time Ta'. The first mechanism introduces discontinuities and can also violate common assumptions that timestamps are monotone increasing. If the host's clock is set backward in time, sometimes this can be easily detected. If the clock is set forward in time, this can be harder to detect. The skew induced by the second mechanism can lead to considerable inaccuracies when computing differences in time, as discussed above. To illustrate why skew is a crucial concern, consider samples of one-way delays between two Internet hosts made at one minute intervals. The true transmission delay between the hosts might plausibly be on the order of 50 ms for a transcontinental path. If the skew between the two clocks is 0.01%, that is, 1 part in 10,000, then after 10 minutes of observation the error introduced into the measurement is 60 ms. Unless corrected, this error is enough to completely wipe out any accuracy in the transmission delay measurement. Finally, we note that assessing skew errors between unsynchronized network clocks is an open research area. (See [Pa97] for a discussion of detecting and compensating for these sorts of errors.) This shortcoming makes use of a solid, independent clock source such as GPS especially desirable. 10.2. The Notion of "Wire Time" Internet measurement is often complicated by the use of Internet hosts themselves to perform the measurement. These hosts can introduce delays, bottlenecks, and the like that are due to hardware or operating system effects and have nothing to do with the network behavior we would like to measure. This problem is particularly acute when timestamping of network events occurs at the application level. In order to provide a general way of talking about these effects, we introduce two notions of "wire time". These notions are only defined in terms of an Internet host H observing an Internet link L at a particular location: + For a given packet P, the 'wire arrival time' of P at H on L is the first time T at which any bit of P has appeared at H's observational position on L. Paxson, et. al. Informational [Page 17] RFC 2330 Framework for IP Performance Metrics May 1998 + For a given packet P, the 'wire exit time' of P at H on L is the first time T at which all the bits of P have appeared at H's observational position on L. Note that intrinsic to the definition is the notion of where on the link we are observing. This distinction is important because for large-latency links, we may obtain very different times depending on exactly where we are observing the link. We could allow the observational position to be an arbitrary location along the link; however, we define it to be in terms of an Internet host because we anticipate in practice that, for IPPM metrics, all such timing will be constrained to be performed by Internet hosts, rather than specialized hardware devices that might be able to monitor a link at locations where a host cannot. This definition also takes care of the problem of links that are comprised of multiple physical channels. Because these multiple channels are not visible at the IP layer, they cannot be individually observed in terms of the above definitions. It is possible, though one hopes uncommon, that a packet P might make multiple trips over a particular link L, due to a forwarding loop. These trips might even overlap, depending on the link technology. Whenever this occurs, we define a separate wire time associated with each instance of P seen at H's position on the link. This definition is worth making because it serves as a reminder that notions like *the* unique time a packet passes a point in the Internet are inherently slippery. The term wire time has historically been used to loosely denote the time at which a packet appeared on a link, without exactly specifying whether this refers to the first bit, the last bit, or some other consideration. This informal definition is generally already very useful, as it is usually used to make a distinction between when the packet's propagation delays begin and cease to be due to the network rather than the endpoint hosts. When appropriate, metrics should be defined in terms of wire times rather than host endpoint times, so that the metric's definition highlights the issue of separating delays due to the host from those due to the network. We note that one potential difficulty when dealing with wire times concerns IP fragments. It may be the case that, due to fragmentation, only a portion of a particular packet passes by H's location. Such fragments are themselves legitimate packets and have well-defined wire times associated with them; but the larger IP packet corresponding to their aggregate may not. Paxson, et. al. Informational [Page 18] RFC 2330 Framework for IP Performance Metrics May 1998 We also note that these notions have not, to our knowledge, been previously defined in exact terms for Internet traffic. Consequently, we may find with experience that these definitions require some adjustment in the future. {Comment: It can sometimes be difficult to measure wire times. One technique is to use a packet filter to monitor traffic on a link. The architecture of these filters often attempts to associate with each packet a timestamp as close to the wire time as possible. We note however that one common source of error is to run the packet filter on one of the endpoint hosts. In this case, it has been observed that some packet filters receive for some packets timestamps corresponding to when the packet was *scheduled* to be injected into the network, rather than when it actually was *sent* out onto the network (wire time). There can be a substantial difference between these two times. A technique for dealing with this problem is to run the packet filter on a separate host that passively monitors the given link. This can be problematic however for some link technologies. See [Pa97] for a discussion of the sorts of errors packet filters can exhibit. Finally, we note that packet filters will often only capture the first fragment of a fragmented IP packet, due to the use of filtering on fields in the IP and transport protocol headers. As we generally desire our measurement methodologies to avoid the complexity of creating fragmented traffic, one strategy for dealing with their presence as detected by a packet filter is to flag that the measured traffic has an unusual form and abandon further analysis of the packet timing.} 11. Singletons, Samples, and Statistics With experience we have found it useful to introduce a separation between three distinct -- yet related -- notions: + By a 'singleton' metric, we refer to metrics that are, in a sense, atomic. For example, a single instance of "bulk throughput capacity" from one host to another might be defined as a singleton metric, even though the instance involves measuring the timing of a number of Internet packets. + By a 'sample' metric, we refer to metrics derived from a given singleton metric by taking a number of distinct instances together. For example, we might define a sample metric of one-way delays from one host to another as an hour's worth of measurements, each made at Poisson intervals with a mean spacing of one second. Paxson, et. al. Informational [Page 19] RFC 2330 Framework for IP Performance Metrics May 1998 + By a 'statistical' metric, we refer to metrics derived from a given sample metric by computing some statistic of the values defined by the singleton metric on the sample. For example, the mean of all the one-way delay values on the sample given above might be defined as a statistical metric. By applying these notions of singleton, sample, and statistic in a consistent way, we will be able to reuse les