Network Working Group S. Armstrong Request for Comments: 1301 Xerox A. Freier Apple K. Marzullo Cornell February 1992 Multicast Transport Protocol Status of this Memo This memo provides information for the Internet community. It does not specify an Internet standard. Distribution of this memo is unlimited. Summary This memo describes a protocol for reliable transport that utilizes the multicast capability of applicable lower layer networking architectures. The transport definition permits an arbitrary number of transport providers to perform realtime collaborations without requiring networking clients (aka, applications) to possess detailed knowledge of the population or geographical dispersion of the participating members. It is not network architectural specific, but does implicitly require some form of multicasting (or broadcasting) at the data link level, as well as some means of communicating that capability up through the layers to the transport. Keywords: reliable transport, multicast, broadcast, collaboration, networking. Table of Contents 1. Introduction 2 2. Protocol description 3 2.1 Definition of terms 3 2.2 Packet format 6 2.2.1. Protocol version 7 2.2.2. Packet type and modifier 7 2.2.3. Subchannel 9 2.2.4. Source connection identifier 9 2.2.5. Destination connection identifier 10 2.2.6. Message acceptance 10 2.2.7. Heartbeat 12 2.2.8. Window 12 2.2.9. Retention 12 Armstrong, Freier & Marzullo [Page 1] RFC 1301 Multicast Transport Protocol February 1992 2.3 Transport addresses 12 2.3.1. Unknown transport address 12 2.3.2. Web's multicast address 13 2.3.3. Member addresses 13 3. Protocol behavior 13 3.1. Establishing a transport 13 3.1.1. Join request 14 3.1.2. Join confirm/deny 16 3.2 Maintaining data consistency 17 3.2.1. Transmit tokens 17 3.2.2. Data transmission 20 3.2.3. Empty packets 23 3.2.4. Missed data 26 3.2.5. Retrying operations 26 3.2.6. Retransmission 27 3.2.7. Duplicate suppression 29 3.2.8. Banishment 29 3.3 Terminating the transport 29 3.3.1. Voluntary quits 30 3.3.2. Master quit 30 3.3.3. Banishment 30 3.4 Transport parameters 30 3.4.1. Quality of service 30 3.4.2. Selecting parameter values 31 3.4.3. Caching member information 33 A. Appendix: MTP as an Internet Protocol transport 34 A.1 Internet Protocol multicast addressing 34 A.2 Encapsulation 35 A.3 Fields of the bridge protocol 35 A.4 Relationship to other Internet Transports 36 References 36 Footnotes 37 Security Considerations 37 Authors' Addresses 38 1. Introduction This document describes a flow controlled, atomic multicasting transport protocol (MTP). The purpose of this document is to present sufficient information to implement the protocol. The MTP design has been influenced by the large body of the networking and distributed systems literature and technology that has been introduced during the last decade and a half. Representative sources include [Xer81], [BSTM79] and [Pos81] for transport design, and [Bog83] and [DIX82] for general concepts of broadcast and multicast. [CLZ87] influenced MTP's retransmission mechanisms, and [Fre84] influenced the transport timings. MTP over IP uses mechanisms Armstrong, Freier & Marzullo [Page 2] RFC 1301 Multicast Transport Protocol February 1992 described in [Dee89]. MTP's ordering and agreement protocols were influenced by work done in [CM87], [JB89] and [Cri88]. Finally, a description of MTP's philosophy and its motivation can be found in [AFM91]. 2. Protocol description MTP is a transport in that it is a client of the network layer (as defined by the OSI networking model) [1]. MTP provides reliable delivery of client data between one or more communicating processes, as well as a predefined principal process. The collection of processes is called a web. In addition to transporting data reliably and efficiently, MTP provides the synchronization necessary for web members to agree on the order of receipt of all messages and can agree on the delivery of the message even in the face of partitions. This ordering and agreement protocol uses serialized tokens granted by the master to producers. The processes may have any one of three levels of capability. One member must be the master. The master instantiates and controls the behavior of the web, including its membership and performance. Non master members may be either producer/consumers or pure consumers. The former class of member is permitted to transmit user data to the entire membership (and expected to logically hear itself), while the latter is prohibited from transmitting user data. MTP is a negative acknowledgement protocol, exploiting the highly reliable delivery of the local area and wide area network technologies of today. Successful delivery of data is accepted by consuming stations silently rather than having the successful delivery noted to the producing process, thus reducing the amount of reverse traffic required to maintain synchronization. 2.1 Definition of terms The following terms are used throughout this document. They are defined here to eliminate ambiguity. consumer A consumer is a transport that is capable only of receiving user data. It may transmit control packets, such as negative acknowledgements, but may never transmit any requests for the transmit token or any form of data or empty messages. heartbeat A heartbeat is an interval of time, nominally measured in milliseconds. It is a key parameter in the transport's Armstrong, Freier & Marzullo [Page 3] RFC 1301 Multicast Transport Protocol February 1992 state and can be adapted to the requirements of the transport's client to provide the desired quality of service. master The master is the principal member of the web. The master capability is a superset of a producer member. The master is mainly responsible for giving out transmit tokens to members who wish to send data, and overseeing the web's membership and operational parameters. member A web member is any process that has been permitted to join the web (by the master) as well as the master itself. membership Every member is classified as to its intentions for class joining the web. Membership classes are defined to be consumer, producer and master. Each successive class is a formal superset of the previous. message An MTP message is a concatenation of the user data portions of a series of data packets with the last packet in the series carrying an end of message indication. A message may contain any number of bytes of user data, including zero. NSAP The network service access point. This is the network address, or the node address of the machine, where a service is available. producer Producer is a class of membership that is a formal superset of a consumer. A producer is permitted (and expected) to transmit client data as well as consume data transmitted by other producers. retention Retention is one of the three fundamental parameters that make up the transport's state (along with heartbeat and window). Retention is a number of heartbeats, and though applied in several different circumstances, is primarily used as the number of heartbeats a producing client must maintain buffered data should it need to be retransmitted. token In order to transmit, a producer must first be in possesion of a token. Tokens are granted only by the master and include the message sequence number. Consequently, they are fundamental in the operation of the ordering and agreement protocol used by MTP. Armstrong, Freier & Marzullo [Page 4] RFC 1301 Multicast Transport Protocol February 1992 TSAP The transport service access point. This is the address that uniquely defines particular instantiation of a service. TSAPs are formed by logically concatenating the node's NSAP with a transport identifier (and perhaps a packet/protocol type). user data User data is the client information carried in MTP data packets and treated as uninterpreted octets by the transport. The end of message and subchannel indicators are also be treated as user data. web A collection of processes collaborating on the solution of a single problem. window The window is one of the fundamental elements of the transport's state that can be controlled to affect the quality of service being provided to the client. It represents the number of user data carrying packets that may be multicast into the web during a heartbeat by a single member. Armstrong, Freier & Marzullo [Page 5] RFC 1301 Multicast Transport Protocol February 1992 2.2 Packet format An MTP packet consists of a transport protocol header followed by a variable amount of data. The protocol header, shown in Figure 1, is part of every packet. The remainder of the packet is either user data (packet type = data) or additional transport specific information. The fields in the header are statically defined as n-bit wide quantities. There are no undefined fields or fields that may at any time have undefined values. Reserved fields, if they exist, must always have a value of zero. 0 7 8 15 16 23 24 31 ---------------------------------------------------------- ----- | protocol | packet | type | client | | | version | type | modifier | channel | | ---------------------------------------------------------- | | | | | source connection identifier | | ---------------------------------------------------------- | | | | | destination connection identifier | ---------------------------------------------------------- transport | | header | message acceptance criteria | ---------------------------------------------------------- | | | | | heartbeat | | ---------------------------------------------------------- | | | | | | window | retention | | ---------------------------------------------------------- ----- | | | | | | | | | | (data content and format | | dependent on packet type | data | and modifier) | fields | | | | | | | | | | | ---------------------------------------------------------- ----- Figure 1. MTP packet format Armstrong, Freier & Marzullo [Page 6] RFC 1301 Multicast Transport Protocol February 1992 2.2.1. Protocol version The first 8 bits of the packet are the protocol version number. This document describes version 1 of the Multicast Transport Protocol and thus the version field has a value of 0x01. 2.2.2. Packet type and modifier The second byte of the header is the packet type and the following byte contains the packet type modifier. Typical control message exchanges are in a request/response pair. The modifier field simplifies the construction of responses by permitting reuse of the incoming message with minimal modification. The following table gives the packet type field values along with their modifiers. The modifiers are valid only in the context of the type. In the prose of the definitions and later in the document, the syntax for referring to one of the entries described in the following table will be type[modifier]. For example, a reference to data[eow] would be a packet of type data with an end of window modifier. type modifier description data(0) data(0) The packet is one that contains user information. Only the process possessing a transmit token is permitted to send data unless specifically requested to retransmit previously transmitted data. All packets of type data are multicast to the entire web. eow(1) A data packet with the eow (end of window) modifier set indicates that the transmitter intends to send no more packets in this heartbeat either because it has sent as many as permitted given the window parameter or simply has no more data to send during the current heartbeat. This is not client information but rather a hint to be used by transport providers to synchronize the computation and transmission of naks. eom(2) Data[eom] marks the end of the message to the consumers, and the surrendering of the transmit token to the master. And like a data[eow] a data[eom] packet implies the end of window. nak(1) request(0) A nak[request] packet is a consumer requesting a retransmission of one or more Armstrong, Freier & Marzullo [Page 7] RFC 1301 Multicast Transport Protocol February 1992 data packets. The data field contains an ordered list of packet sequence numbers that are being requested. Naks of any form are always unicast. deny(1) A nak[deny] message indicates that the producer source of the nak[deny]) cannot retransmit one or more of the packets requested. The process receiving the nak[deny] must report the failure to its client. empty(2) dally(0) An empty[dally] packet is multicast to maintain synchronization when no client data is available. cancel(1) If a producer finds itself in possession of a transmit token and has no data to send, it may cancel the token[request] by multicasting an empty[cancel] message. hibernate(2) If the master possesses all of the web's transmit tokens and all outstanding messages have been accepted or rejected, the master may transmit empty[hibernate] packets at a rate significantly slower than indicated by the web's value of heartbeat. join(3) request(0) A join[request] packet is sent by a process wishing to join a web to the web's unknown TSAP (see section 2.2.5). confirm(1) The join[confirm] packet is the master's confirmation of the destination's request to join the web. It will be unicast by the master (and only the master) to the station that sent the join[request]. deny(2) A join[deny] packet indicates permission to join the web was denied. It may only be transmitted by the master and will be unicast to the member that sent the join[request]. quit(4) request(0) A quit[request] may be unicast to the master by any member of the web at any time to indicate the sending process wishes to withdraw from the web. Any member may unicast a quit to another member requesting that the Armstrong, Freier & Marzullo [Page 8] RFC 1301 Multicast Transport Protocol February 1992 destination member quit the web due to intolerable behavior. The master may multicast a quit[request] requiring that the entire web disband. The request will be multicast at regular heartbeat intervals until there are no responses to retention requests. confirm(1) The quit[confirm] packet is the indication that a quit[request] has been observed and appropriate local action has been taken. Quit[confirm] are always unicast. token(5) request(0) A token[request] is a producing member requesting a transmit token from the master. Such packets are unicast to the master. confirm(1) The token[confirm] packet is sent by the master to assign the transmit token to a member that has requested it. token[confirm] will be unicast to the member being granted the token. isMember(6) request(0) An isMember[request] is soliciting verification that the target member is a recognized member of the web. All forms of the isMember packet are unicast to a specific member. confirm(1) IsMember[confirm] packets are positive responses to isMember[requests]. deny(2) If the member receiving the isMember[request] cannot confirm the target's membership in the web, it responds with a isMember[deny]. 2.2.3. Subchannel The fourth byte of the transport header contains the client's subchannel value. The default value of the subchannel field is zero. Semantics of the subchannel value are defined by the transport client and therefore are only applicable to packets of type data. All other packet types must have a subchannel value of zero. 2.2.4. Source connection identifier The source connection identifier field is a 32 bit field containing a transmitting system unique value assigned at the time the transport Armstrong, Freier & Marzullo [Page 9] RFC 1301 Multicast Transport Protocol February 1992 is created. The field is used in identifying the particular transport instantiation and is a component of the TSAP. Every packet transmitted by the transport must have this field set. 2.2.5. Destination connection identifier The destination connection identifier is the 32 bit identifier of the target transport. From the point of view of a process sending a packet, there are three types of destination connection identifiers. First, there is the unknown connection identifier (0x00000000). The unknown value is used only as the destination connection identifier in the join[request] packet. Second, there is the multicast connection identifier gleaned from the join[confirm] message sent by the master. The multicast connection identifier is used in conjunction with the multicast NSAP to form the destination TSAP of all packets multicast to the entire web [2]. The last class of connection identifier is a unicast identifier and is used to form the destination TSAP when unicasting packets to individual members. Every member of the web has associated with it a unicast connection identifier that is used to form its own unicast TSAP. 2.2.6. Message acceptance MTP ensures that all processes agree on which messages are accepted and in what order they are accepted. The master controls this aspect of the protocol by controlling allocation of transmit tokens and setting the status of messages. Once a token for a message has been assigned (see section 3.2.1) the master sets the status of that message according to the following rules [AFM91]: If the master has seen the entire message (i.e., has seen the data[eom] and all intervening data packets), the status is accepted. If the master has not seen the entire message but believes the message sender is still operational and connected to the master (as determined by the master), the status is pending. If the master has not seen the entire message and believes the sender to have failed or partitioned away, the status is rejected. Message status is carried in the message acceptance record (see Figure 2) of every packet, and processes learn the status of earlier messages by processing this information. The acceptance criteria is a multiple part record that carries the Armstrong, Freier & Marzullo [Page 10] RFC 1301 Multicast Transport Protocol February 1992 rules of agreement to determine the message acceptance. The most significant 8 bits is a flag that, if not zero, indicates synchronization is required. The field may vary on a per message basis as directed by producing transport's client. The default is that no synchronization is required. The second part of the record is a 12 element vector that represents the status of the last 12 messages transmitted into the web. 0 7 8 15 16 23 24 31 --------------------------------------------------------- | | | | synchro | tri-state bitmask[12] | --------------------------------------------------------- | message | packet sequence | | sequence number | number | --------------------------------------------------------- Figure 2. Message acceptance record Each element of the array is two bits in length and may have one of three values: accepted(0), pending(1) or rejected(2). Initially, the bit mask is set to all zeros. When the token for message m is transmitted, the first (left-most) element of the vector represents the the state of message m - 1, the second element of the vector is the status of message m - 2, and so forth. Therefore the status of the last 12 messages are visible, the status of older messages are lost, logically by shifting the elements out of the vector. Only the master is permitted to set the status of messages. The master is not permitted to shift a status of pending beyond the end of the vector. If that situation arises, the master must instead not confirm any token[request] until the oldest message can be marked as either rejected or accepted. Message sequence numbers are 16 bit unsigned values. The field is initialized to zero by the master when the transport is initialized, and incremented by one after each token is granted. Only the master is permitted to change the value of the message sequence number. Once granted, that message sequence number is consumed and the state of the message must eventually become either accepted or rejected. No transmit tokens may be granted if the assignment of a message sequence number that would cause a value of pending to be shifted beyond the end of the status vector. Packet sequence numbers are unsigned 16 bit numbers assigned by the producing process on a per message basis. Packet sequence numbers start at a value of zero for each new message and are incremented by one (consumed) for each data packet making up the message. Consumers Armstrong, Freier & Marzullo [Page 11] RFC 1301 Multicast Transport Protocol February 1992 detecting missing packet sequence numbers must send a nak[request] to the appropriate producer to recover the missed data. Control packets always contain the message acceptance criteria with a synchronization flag set to zero (0x00), the highest message sequence number observed and a packet sequence number one greater than previously observed. Control packets do not consume any sequence numbers. Since control messages are not reliably delivered, the acceptance criteria should only be checked to see if they fall within the proper range of message numbers, relative to the current message number of the receiving station. The range of acceptable sequence numbers should be m-11 to m-13, inclusive, where m is the current message number. 2.2.7. Heartbeat Heartbeat is an unsigned 32 bit field that has the units of milliseconds. The value of heartbeat is shared by all members of the web. By definition at least one packet (either data, empty or quit from the master) will be multicast into the web within every heartbeat period. 2.2.8. Window The allocation window (or simply window) is a 16 bit unsigned field that indicates the maximum number of data packets that can be multicasted by a member in a single heartbeat. It is the sum of the retransmitted and new data packets. 2.2.9. Retention The retention field is a 16 bit unsigned value that is the number of heartbeats for which a producer must retain transmitted client data and state for the purpose of retransmission. 2.3 Transport addresses Associated with each transport are logically three transport service access points (TSAP), logically formed by the concatenation of a network service access point (NSAP) and a transport connection identifier. These TSAPs are the unknown TSAP, the web's multicast TSAP and each individual member's TSAP. 2.3.1. Unknown transport address Stations that are just joining must use the multicast NSAP associated with the transport, but are not yet aware of either the web's multicast TSAP the master process' TSAP. Therefore, joining stations Armstrong, Freier & Marzullo [Page 12] RFC 1301 Multicast Transport Protocol February 1992 fabricate a temporary TSAP (referred to as a unknown TSAP) by using a connection identifier reserved to mean unknown (0x00000000). The join[confirm] message will be sourced from the master's TSAP and will include the multicast transport connection identifier in the data field. Those values must be extracted from the join[confirm] and remembered by the joining process. 2.3.2. Web's multicast address The multicast TSAP is formed by logically concatenating the multicast NSAP associated with the transport creation and the transport connection identifier returned in the data field of the join[confirm] packet. If more than one network is involved in the web, then the multicast transport address becomes a list, one for each network represented. This list is supplied in the data field of token[confirm] packets. The multicast TSAP is used as the target for all messages that are destined to the entire web, such as data and empty. The master's decision to abandon the transport (quit) is also sent to the multicast transport address. 2.3.3. Member addresses The member TSAP is formed by using the process' unicast NSAP concatenated with a locally generated unique connection identifier. That TSAP must be the source of every packet transmitted by the process, regardless of its destination, for the lifetime of the transport. Packets unicast to specific members must contain the appropriate TSAP. For producers and consumers this is not difficult. The only TSAPs of interest are the master and the station(s) currently transmitting data. 3. Protocol behavior This section defines the expectations of the protocol implementation. These expectations should not be considered guidelines or hints, but rather part the protocol. 3.1 Establishing a transport Before any rendezvous can be affected, a process must first acquire an NSAP that will be the service access point for the instantiation [3]. The process that first establishes at that NSAP is referred to as the master of the web. The decision as to what process acts as the master must be made a priori in order to guarantee unambiguous Armstrong, Freier & Marzullo [Page 13] RFC 1301 Multicast Transport Protocol February 1992 creation in the face of network partitions. The process should make a robust effort to verify that the NSAP being used is not already in service. It may do so by repeatedly sending join[requests] to the web's unknown TSAP. If there is no response to repeated transmissions the process may be relatively confident that the NSAP is not in use and proceed with the creation of the web. If not, the creation must be aborted and the situation reported to its client. 3.1.1. Join request Additional members may join the web at any time after the establishment of the master by the joining process sending a join[request] to the unknown TSAP. The joining process should have already assigned a unique connection identifier to its transport instantiation that will be used in the source TSAP of the join[request]. The join[request] must contain zeros in all of the acceptance fields. The heartbeat, window and retention parameters are filled in as requested by the transport provider's client. The data of the message must contain the type, class and quality of service parameters that the client has requested. field class definition membership class master(0) There can be only a single web master, and that member has all privileges of a producer class member plus those acquitted only to the master. producer(1) A process that has producer class membership wishes to transmit data into the web as well as consume. consumer(2) A consumer process is a read only process. It will send naks in order to reliably receive data but will never ask for or be permitted to take possession of a transmit token. transport class reliable(0) Specifies a reliable transport, i.e., one that will generate and process naks. The implication is that the data will be reliably delivered or the failure will be detected and reported to the client. unreliable(1) The transport supports best Armstrong, Freier & Marzullo [Page 14] RFC 1301 Multicast Transport Protocol February 1992 effort delivery. Such a transport may still fail if the error rates are too high, but tolerable loss or corruption of data will be permitted [4]. transport type NxN(0) The transport will accept multiple processes with producing capability. 1xN(1) A 1xN transport permits only a single producer whose identity was established a priori. The client's desire for minimum throughput (expressed in kilobytes per second) is the lowest value that will be accepted. That throughput is calculated using the heartbeat and window parameters of the transport, and the maximum data unit size, not by measuring actual traffic. Any member that suggests a combination of those parameters that result in an unacceptable throughput will be ignored or asked to withdraw from the web. A joining client may also suggest a maximum data unit size. This field is expressed as a number of bytes that can be included in a data packet as client data. If no response is received in a single heartbeat, the join[request] should be retransmitted using the same source TSAP so the master can detect the difference between a new process and a retransmission of a join[request]. Armstrong, Freier & Marzullo [Page 15] RFC 1301 Multicast Transport Protocol February 1992 3.1.2. Join confirm/deny Only the master of the web will respond to join[request]. The response may either permit the entry of the new process or deny it. The request to join may be denied because the new member is specifying service parameters that are in conflict with those established by the master. If the join is confirmed the join[confirm] will be unicast by the master with a data field that contains the web's current operating parameters. If those parameters are unacceptable to the joining process it may decide to withdraw from the web. Otherwise the parameters must be accepted as the current operating values. 0 7 8 15 16 23 24 31 ---------------------------------------------------------- ----- | protocol | packet | type | client | | | version | type | modifier | channel | | ---------------------------------------------------------- | | | | | source connection identifier | | ---------------------------------------------------------- | | | | | destination connection identifier | ---------------------------------------------------------- transport | | header | message acceptance criteria | ---------------------------------------------------------- | | | | | heartbeat | | ---------------------------------------------------------- | | | | | | window | retention | | ---------------------------------------------------------- ----- | member | transport | transport | | | | class | class | type | reserved | | ---------------------------------------------------------- | minimum | maximum data | data | throughput | unit size | ---------------------------------------------------------- | | multicast connection | | | identifier | | ---------------------------------------------------------- ----- Figure 3. join packet The join[confirm] will also contain the multicast connection identifier. This must be used to form the TSAP that will be the destination for all multicast messages for the transport. The source Armstrong, Freier & Marzullo [Page 16] RFC 1301 Multicast Transport Protocol February 1992 of the join[confirm] message will be the master's TSAP and must be recorded by the member for later use. The master must be in possession of all the transmit tokens when it sends a join[confirm]. Requiring the master to have the transmit tokens insures that the joining member will enter the web and observe only complete messages. It also permits a notification of the master's client of the join so that application state may be automatically sent to the newly joining member. The newly joined member may be on a network not previously represented in the web's membership, thus requiring a new multicast TSAP be added to the existing list. The entire list will be conveyed in the data field of all subsequent token[confirm] messages (described later). 3.2 Maintaining data consistency The transport is responsible for maintaining the consistency of the data submitted for delivery by producing clients. The actual client data, while representing the bulk of the information that flows through the web, is accompanied by significant amounts of protocol state information. In addition to the state information piggybacked with the client data, there is a minimum amount of protocol packets that are purely for use by the transport, invisible to the transport client. 3.2.1. Transmit tokens Before any process may transmit client data or state it must first possess a transmit token. It may acquire the token by transmitting a token[request] to the master. Requests should be unicast to the master's TSAP and should be retransmitted at intervals approximately equal to the heartbeat. Since it is the central source for a transmit token, the master may apply some fairness algorithms to the passing of permission to transmit. At a minimum the requests should be queued in a first in, first out order. Duplicate requests from a single member should be ignored, keeping instead the first unhonored request. When appropriate, the master will send a member with a request pending a token[confirm]. The data field of the response contains all the multicast TSAPs that are represented in the current web at that point in time. If the master detects no data or heartbeat messages being transmitted into the web it will assume the token is lost, presumably because the member holding the token has failed or has become partitioned away from the master. In such cases, the master may attempt to confirm the state of the process (perhaps by sending isMember[request]). If the member does not respond it is removed from the active members of the web, the message is marked as rejected, the token is assumed by the Armstrong, Freier & Marzullo [Page 17] RFC 1301 Multicast Transport Protocol February 1992 master. Figure 4 shows a timing diagram of a token pass. Increasing time is towards the bottom of the figure. In this figure, process A has a token, and process B requests a token when there are no free tokens. A master B "A" multicasts data | | "B" requests |\ | | transmit token | \ | /| | \ | / | | \ | / | "A" multicasts data | \ | / | "B" retransmits w/eom set |\ \| / | token request | \ \V /| | \ |\ / | | \ | V / | | \ | / | | \| / | | \V | | |\ | | | V | | |\ | Master assigns | | \ | token to "B" | | \ | | | \ | | | \ | | | V| | | | | | /| "B" multicasts | | / | data | | / | | | / | | | / | | |/ | | / | | /| | | V | | | | | Figure 4. Acquiring the token Token packets, like other control packets, do not consume sequence numbers. Hence, the master must be able to use another mechanism to determine whether multiple token[request] from a single member are actually requests for a separate token, or are a retransmission of a token[request]. To carry out this obligation, the master and the members must have an implicit understanding of each other's state. Armstrong, Freier & Marzullo [Page 18] RFC 1301 Multicast Transport Protocol February 1992 0 7 8 15 16 23 24 31 ---------------------------------------------------------- ----- | protocol | packet | type | client | | | version | type | modifier | channel | | ---------------------------------------------------------- | | | | | source connection identifier | | ---------------------------------------------------------- | | | | | destination connection identifier | ---------------------------------------------------------- transport | | header | message acceptance criteria | ---------------------------------------------------------- | | | | | heartbeat | | ---------------------------------------------------------- | | | | | | window | retention | | ---------------------------------------------------------- ----- | | | | | | | TSAPs of all networks | | represented in the web | data | membership | | | | | | | ---------------------------------------------------------- ----- Figure 5. token packet Assume that the token, as viewed by the master, has three states: idle The token is not currently assigned. Specifically the message number that it defines is not represented in the current message acceptance vector. pending The token has been assigned by the master via a token[confirm] packet, but the master has not yet seen any data packets to indicate that the from the producing member received the notification. busy The token has been assigned and the master has seen data packets carrying the assigned message number. The message comprised by those packets is still represented in the message acceptance vector. Furthermore, a token that is not idle also has associated with its Armstrong, Freier & Marzullo [Page 19] RFC 1301 Multicast Transport Protocol February 1992 state the TSAP of the process that owns (or owned) the token. Based on this state, the master will respond to any process that has a token in pending state with a reassignment of that token. This is based on the assumption that the original token[confirm] was not received by the requesting process. The only other possibility is that the process did receive the token and tr