Network Working Group N. Brownlee Request for Comments: 2123 The University of Auckland Category: Informational March 1997 Traffic Flow Measurement: Experiences with NeTraMet Status of this Memo This memo provides information for the Internet community. This memo does not specify an Internet standard of any kind. Distribution of this memo is unlimited. Abstract This memo records experiences in implementing and using the Traffic Flow Measurement Architecture and Meter MIB. It discusses the implementation of NeTraMet (a traffic meter) and NeMaC (a combined manager and meter reader), considers the writing of meter rule sets and gives some guidance on setting up a traffic flow measurement system using NeTraMet. Table of Contents 1 Introduction 2 1.1 NeTraMet structure and development . . . . . . . . . . . . . . 3 1.2 Scope of this document . . . . . . . . . . . . . . . . . . . . 4 2 Implementation 4 2.1 Choice of meter platform . . . . . . . . . . . . . . . . . . . 4 2.2 Programming support requirements . . . . . . . . . . . . . . . 5 2.2.1 DOS environment . . . . . . . . . . . . . . . . . . . . . 6 2.2.2 Unix environment . . . . . . . . . . . . . . . . . . . . . 7 2.3 Implementing the meter . . . . . . . . . . . . . . . . . . . . 7 2.3.1 Data structures . . . . . . . . . . . . . . . . . . . . . 7 2.3.2 Packet matching . . . . . . . . . . . . . . . . . . . . . 8 2.3.3 Testing groups of rule addresses . . . . . . . . . . . . . 8 2.3.4 Compression of address masks . . . . . . . . . . . . . . . 9 2.3.5 Ignoring unwanted flow data . . . . . . . . . . . . . . . 10 2.3.6 Observing meter reader activity . . . . . . . . . . . . . 11 2.3.7 Meter memory management . . . . . . . . . . . . . . . . . 12 2.4 Data collection . . . . . . . . . . . . . . . . . . . . . . . 14 2.5 Restarting a meter . . . . . . . . . . . . . . . . . . . . . . 15 2.6 Performance . . . . . . . . . . . . . . . . . . . . . . . . . 16 3 Writing rule sets 16 3.1 Rule set to observe all flows . . . . . . . . . . . . . . . . 17 3.2 Specifying flow direction, using computed attributes . . . . . 18 3.3 Subroutines . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.4 More complicated rule sets . . . . . . . . . . . . . . . . . . 23 Brownlee Informational [Page 1] RFC 2123 Traffic Flow Measurement March 1997 4 Flow data files 26 4.1 Sample flow data file . . . . . . . . . . . . . . . . . . . . 27 4.2 Flow data file features . . . . . . . . . . . . . . . . . . . 28 4.3 Terminating and restarting meter reading . . . . . . . . . . . 29 5 Analysis applications 30 6 Using NeTraMet in a measurement system 31 6.1 Examples of NeTraMet in production use . . . . . . . . . . . . 31 7 Acknowledgments 33 8 References 33 9 Security Considerations 34 10 Author's Address 34 1 Introduction Early in 1992 my University needed to develop a system for recovering the costs of its Internet traffic. In March of that year I attended the Internet Accounting Working Group's session at the San Diego IETF, where I was delighted to find that the Group had produced a detailed architecture for measuring network traffic and were waiting for someone to try implementing it. During 1992 I produced a prototype measurement system, using balanced binary trees to store information about traffic flows. This work was reported at the Washington IETF in November 1992. The prototype performed well, but it made no attempt to recover memory from old flows, and the overheads in managing the balanced trees proved to be unacceptably high. I moved on to develop a production-quality system, this time using hash tables to index the flow information. This version was called NeTraMet (the Network Traffic Meter), and was released as free software in October 1993. Since then I have continued working on NeTraMet, producing new releases two or three times each year. NeTraMet is now in production use at many sites around the world. It is difficult to estimate the number of sites, but there is an active NeTraMet mailing list, which had about 130 subscribers in March 1996. Early in 1996 the Realtime Traffic Flow Measurement Working Group (RTFM) was chartered to move the Traffic Flow Measurement Architecture on to the IETF standards track. This document records traffic flow measurement experience gained through three years experience with NeTraMet. Brownlee Informational [Page 2] RFC 2123 Traffic Flow Measurement March 1997 1.1 NeTraMet structure and development The Traffic Flow Architecture document [1] describes four components: - METERS, which are attached to the network at the points where it is desired to measure the traffic, - METER READERS, which read data from meters and store it for later use, - MANAGERS, which configure meters and control meter readers, and - ANALYSIS APPLICATIONS, which process the data from meter readers so as to produce whatever reports are required. NeTraMet is a computer program which implements the Traffic Meter, stores the measured flow data in memory, and provides an SNMP agent so as to make it available to Meter Readers. The NeTraMet distribution files include NeMaC, which is a combined Manager and Meter Reader capable of managing an arbitrary number of meters, each of which may be using its own rule set, and having its flow data collected at its own specified intervals. The NeTraMet distribution also includes several rudimentary Analysis Applications, allowing users to produce simple plots from NeMaC's flow data files (fd_filter and fd_extract) and to monitor - in real time - the flows at a remote meter (nm_rc and nifty). Since the first release the Traffic Meter MIB [2] has been both improved and simplified. Significant changes have included better ways to specify traffic flows (i.e. more actions and better control structures for the Packet Matching Engine), and computed attributes (class and kind). These changes have been prompted by operational requirements at sites using NeTraMet, and have been tested extensively in successive versions of NeTraMet. NeTraMet is widely used to collect usage data for Internet Service Providers. This is especially so in Australia and New Zealand, but there are also active users at sites around the world, for example in Canada, France, Germany and Poland. NeTraMet is very useful as a tool for understanding exactly where traffic is flowing in large networks. Since the Traffic Meters perform considerable data reduction (as specified by their rule sets) they significantly reduce the volume of data to be read by Meter Readers. This characteristic makes NeTraMet particularly effective for networks with many remote sites. An example of this (the Kawaihiko network) is briefly described below. Brownlee Informational [Page 3] RFC 2123 Traffic Flow Measurement March 1997 As well as providing data for post-observation analysis, NeTraMet can be used for real-time network monitoring and trouble-shooting. The NeTraMet distribution includes 'nifty,' an X/Motif application which monitors traffic flows and attempts to highlight those which are 'interesting.' 1.2 Scope of this document This document presents the experience gained from three years work with the Traffic Flow Measurement Architecture. Its contents are grouped as follows - Implementation issues for NeTraMet and NeMaC, - How rule files work, and how to write them for particular purposes, and - How to use NeTraMet and NeMaC for short-term and long-term flow measurement. 2 Implementation 2.1 Choice of meter platform As pointed out in the Architecture document [1], the goal of the Realtime Traffic Flow Measurement Working Group is to develop a standard for the Traffic Meter, with the goal of seeing it implemented in network devices such as hubs, switches and routers. Until the Architecture is well enough developed to allow this, it has sufficed to implement the meter as a program running on a general- purpose computer system. The choice of computer system for NeTraMet was driven by the need to choose one which would be widely available within the Internet community. One strong possibility was a Unix system, since these are commonly used for a variety of network support and management tasks. For the initial implementation, however, Unix would have had some disadvantages: - The wide variety of different Unix systems can increase the difficulties of software support. - The cost of a Unix system as a meter is too high to allow users to run meters simultaneously at many points within their networks. Brownlee Informational [Page 4] RFC 2123 Traffic Flow Measurement March 1997 Another factor in choosing the platform was system performance. When I first started implementing NeTraMet it was impossible to predict how much processing workload was needed for a viable meter. Similarly, I had no idea how much memory would be required for code or data. I therefore chose to implement NeTraMet on a DOS PC. This was because: - It is a minimum system in all respects. If the meter works well on such a system, it can be implemented on almost any hardware (including routers, switches, etc.) - It is an inexpensive system. Sites can easily afford to have many meters around their networks. - It is a simple system, and one which I had complete control over. This allowed me to implement effective instrumentation to monitor the meter's performance, and to include a wide variety of performance optimisations in the code. Once the meter was running I needed a manager to download rule files to it. Since a single manager and meter reader can effectively support a large number of meters, a Unix environment for NeMaC was a natural choice. There are fewer software support problems for NeMaC than for NeTraMet since NeMaC has minimal support needs - it only needs to open a UDP socket to the SNMP port on each controlled meter. Early NeTraMet distributions contained only the PC meter and Unix manager. In later releases I ported NeTraMet (the meter) to Unix, and extended the control features of NeMaC (the combined manager and meter reader). I have also experimented with porting NeMaC to the DOS system. This is not difficult, but doesn't seem to be worth pursuing. The current version of NeTraMet is a production-quality traffic measurement system which has been in continuous use at the University of Auckland for nearly two years. 2.2 Programming support requirements To implement the Traffic Flow Meter I needed a programming environment providing good support for the following: - observation of packet headers on the network; - system timer with better than 10 ms resolution; - IP (Internet Protocol), for communications with manager and meter reader; Brownlee Informational [Page 5] RFC 2123 Traffic Flow Measurement March 1997 - SNMP, for the agent implementing the Meter MIB. 2.2.1 DOS environment For the PC I chose to use Ethernet as the physical network medium. This is simply an initial choice, being the medium used within the University of Auckland's data network. Interfaces for other media could easily be added as they are needed. In the PC environment a variety of 'generalised' network interfaces are available. I considered those available from companies such as Novell, DEC and Microsoft and decided against them, partly because they are proprietary, and partly because they did not appear to be particularly easy to use. Instead I chose the CRYNWR Packet Drivers [3] . These are available for a wide variety of interface cards and are simple and clearly documented. They support Ethernet's promiscuous mode, allowing one to observe headers for every passing packet in a straightforward manner. One disadvantage of the Packet Drivers is that it is harder to use them with newer user shells (such as Microsoft Windows), but this was irrelevant since I intended to run the meter as the only program on a dedicated machine. Timing on the PC presented a challenge since the BIOS timer routines only provide a clock tick about 18 times each second, which limits the available time resolution. Initially I made do with a timing resolution of one second for packets, since I believed that most flows existed for many seconds. In recent years it has become apparent that many flows have lifetimes well under a second. To measure them properly with the Traffic Flow Meter one needs times resolved to 10 millisecond intervals, this being the size of TimeTicks, the most common time unit within SNMP [4]. Since all the details of the original PC are readily available [5], it was not difficult to understand the underlying hardware. I have written PC timer routines for NeTraMet which read the hardware timer with 256 times the resolution of the DOS clock ticks, i.e. about 5 ticks per millisecond. There are many TCP/IP implementations available for DOS, but most of them are commercial software. Instead I chose Waterloo TCP [6], since this was available (including full source code) as public domain software. This was necessary since I needed to modify it to allow me to save incoming packet headers at the same time as forwarding packets destined for the meter to the IP handler routines. For SNMP I chose CMU SNMP [7], since again this was available (with full source code) as public domain software. This made it fairly simple to port it from Unix to the PC. Brownlee Informational [Page 6] RFC 2123 Traffic Flow Measurement March 1997 Finally, for the NeTraMet development I used Borland's Turbo C and Turbo Assembler. Although many newer C programming environments are now available, I have been perfectly happy with Turbo C version 2 for the NeTraMet project! 2.2.2 Unix environment In implementing the Unix meter, the one obvious problem was 'how do I get access to packet headers?' Early versions of the Unix meter were implemented using various system-specific interfaces on a SunOS 4.2 system. Later versions use libpcap [8], which provides a portable method of obtaining access to packet headers on a wide range of Unix systems. I have verified that this works very well for ethernet interfaces on Solaris, SunOS, Irix, DEC Unix and Linux, and for FDDI interfaces on Solaris. libpcap provides timestamps for each packet header with resolution determined by the system clock, which is certainly better than 10 ms! All Unix systems provide TCP/IP capabilities, so that was not an issue. For SNMP I used CMU SNMP, exactly as on the PC. 2.3 Implementing the meter This section briefly discusses the data structures used by the meter, and the