Measuring Distributed Durations with Stable Errors

Transcription

1 Measuring Distributed Durations with Stable Errors António Casimiro Pedro Martins Paulo Veríssimo Luís Rodrigues Faculdade de Ciências da Universidade de Lisboa Bloco C5, Camo Grande, Lisboa, Portugal Abstract The round-tri duration measurement technique is fundamental to solve many roblems in asynchronous distributed systems. In essence, this technique rovides the means for reading remote clocks with a known and bounded error. Therefore, it is used as a fundamental building block in several clock synchronization algorithms. In general, the technique can be used to imlement duration measurement services, such as the one of the Timely Comuting Base model. In this aer we roose a new technique to measure distributed durations that minimizes the measurement error and is able to kee this error almost stable. The new technique can be used to imrove the recision of remote clock reading in certain situations. We rovide a rotocol that imlements this new technique and we resent some evaluation results. The results clearly show that our solution is indeed better than existing ones. 1 Introduction The roblem of remote clock reading has always been one of the fundamental roblems in distributed comuting, deriving from the need of synchronizing the clocks of all the nodes in the system. Most roosed clock synchronization rotocols require the reading of remote clocks. They are built on the rincile that a node can synchronize its clock with the clock of a remote node if it knows the value dislayed by the remote clock at a given instant. In ractice, however, since remote clocks cannot be read instantaneously, when the local clock is synchronized to the remote one, they will already be aart. The difference corresonds to the time elased between the two events of reading the remote clock, and setting the local one. It is therefore necessary to take this time into account when synchronizing the clocks, which requires This work was artially suorted by the EC, through roject IST (CORTEX), and by the FCT, through the Large-Scale Informatic Systems Laboratory and rojects Praxis/P/EEI/12160/1998 (MI- CRA) and Praxis/P/EEI/14187/1998 (DEAR-COTS). the use of some measurement technique. We say that reading a remote clock is a distributed action, which takes a certain amount of time, which we call a distributed duration. Generically, a message exchanged between any two nodes in a distributed system can be considered a distributed action, to which is associated a certain real time duration that may be estimated with a bounded error. The ability to measure distributed durations with bounded and small errors is crucial not only for clock synchronization, but more generally in any distributed environment with timeliness requirements. In this aer we roose a new technique for measuring distributed durations (or for reading remote clocks), that rovides imroved measurement errors when comared with the other existing techniques. In articular, it is able to rovide better results than the original round-tri duration measurement technique [6] and some of its successors [9, 1]. We refer to this new technique as the Imroved Round-Tri Technique (IMP) and roose a rotocol that imlements it. This rotocol (and imlicitly the imroved technique) has been used, in articular, in the construction of the Duration Measurement service of the Timely Comuting Base (TCB) model [14]. The TCB model rovides a generic aroach to the roblem of artial synchrony, and the Duration Measurement service is one of the basic services that it rovides in order to satisfy a wide range of alications with timeliness requirements. The exeriments that we erformed using imlementations of both the original and the imroved round-tri techniques have clearly confirmed that it is ossible to obtain better results with the latter. We observed that unlike the revious existing solutions, the roosed solution is able to deliver almost stable measurement errors, which only increase due to the drift rate of local clocks. The rest of the aer is organized as follows. In the next section we refer to related work in the area. The basic round-tri duration measurement technique is briefly resented in Section 3, just before we exlain how it can be imroved, in Section 4. The comlete rotocol is resented in Section 5 and the evaluation results aear in Section 6. Finally, Section 7 concludes the aer.

2 2 Related Work q T 1 The measurement of distributed durations is a generic roblem of asynchronous distributed systems, which has been addressed in the context of other more secific roblems, such as achieving clock synchronization or ensuring timely communication. The variety of algorithms and solutions for clock synchronization that have been roosed during more than a decade is considerable (see surveys in [2] and [13]). Of all these algorithms, we are articularly interested in the category of robabilistic algorithms, in which it is necessary to obtain estimations of remote clock values, measuring the duration of reading actions. The seminal aer of Cristian about robabilistic clock synchronization [6] has first formally resented the round-tri duration measurement technique, on which several other works have built thereafter [1, 7, 9]. Probabilistic estimation methods have also been roosed in [3] and [12] and of articular relevance is the Network Time Protocol (NTP), widely used in the internet, that also uses a round-tri based method to obtain estimations of remote clock values [11]. These clock synchronization algorithms exloit the fact that message delivery delays in existing asynchronous networks, although exhibiting ossibly very high delays, are tyically around a small value, near the lower bound. Provided that a sufficient number of messages is transmitted, it is highly robable that some of those messages are fast messages, allowing good estimates of remote clock values. But as we show in this aer, these estimates can in certain situations be imroved using the new technique we roose. Therefore, our contribution can lead to better robabilistic clock synchronization algorithms. On the other hand, some services need to ensure the best ossible estimation for each message delivery delay, indeendently of tyical robabilistic distributions and observation intervals. This is the case of the Duration Measurement service, secified in the context of the Timely Comuting Base (TCB) model [14]. We contribute with a new technique that is more aroriate to imlement this service than reviously existing ones. 3 The Round-Tri Technique The round-tri duration measurement technique roosed by Cristian [6], used to read remote clock values, basically consists in the following. When a rocess Ô wants to read the clock of some rocess Õ, it measures on its local clock the round-tri delay elased between the sending of a request message Ñ ½ to Õ, and the arrival of the rely Ñ (see Figure 1). This delay rovides an uer bound for the time Ñ took to travel from Õ to Ô and allows to bound the reading error of Õ s clock. T 0 m 1 m 2 t min tmax (m 2 ) t min (m 2 ) T 2 Figure 1. Round-Tri duration measurement using Cristian s technique. Assuming the minimum message transmission delay to be Ø Ñ Ò, and the maximum drift rate of local clocks to be, the real time duration for the transmission delay of Ñ, Ø Ñ µ, can be bounded as follows (the suerscrit ÊÌ stands for the Round-Tri method): Ñ Ü Ñ µ Ì Ì ¼ µ ½ µ Ø Ñ Ò (1) Ñ Ò Ñ µ Ø Ñ Ò (2) Therefore, the transmission delay of Ñ can be estimated as the midoint of this interval, with an associated error equivalent to half of the interval. The result is: Ñ µ Ì Ì ¼ µ ½ µ ÊÌ Ñ µ Ì Ì ¼ µ ½ µ (3) Ø Ñ Ò (4) In the above equations, relative to the examle of Figure 1, rocess Õ sends Ñ immediately when it receives Ñ ½. However, note that in real settings one has to take into account the rocessing time sent by Õ to generate the rely. Therefore, a more generic aroach consists in assuming that any request message Ñ sent from Ô to Õ can be used to estimate Ø Ñ µ (see Figure 2). The estimation of Ø Ñ µ using any message air Ñ Ñ, requires that rocess Ô knows all send and receive timestams for that air, that is, Ì Ë, Ì Ê, Ì ½ and Ì. In the examle illustrated in Figure 2 there is a message Ñ that is used instead of Ñ ½ to estimate Ø Ñ µ. Although the minimum ossible value for the transmission delay of Ñ is always Ø Ñ Ò, the equation for Ñ Ü Ñ µ can now be written as follows: Ñ Ü Ñ µ Ì Ì Ë µ ½ µ Ì ½ Ì Ê µ ½ µ Ø Ñ Ò (5) This also affects the exressions that are used to estimate Ø Ñ µ: Ñ µ Ì Ì Ë µ ½ µ Ì ½ Ì Ê µ ½ µ (6)

3 q m k-1 T S T R T S m k m 1 m 2 T 0 T R (T 1 -T R )(1-ρ) T 1 (T 2 -T S )(1+ρ) Figure 2. Choosing the otimal message air in the round-tri duration measurement technique. ÊÌ Ñ µ Ì Ì Ë µ ½ µ Ì ½ Ì Ê µ ½ µ T 2 Ø Ñ Ò (7) Now assume that there is another message Ñ ½ that has also been sent by Ô to Õ before Ñ. The basic round-tri duration measurement technique simly rooses to use the most recent message air (in this case Ñ Ñ ) to estimate Ø Ñ µ. However, it is ossible to make an otimization which consists in using the message air ( Ñ ½ Ñ or Ñ Ñ ) that rovides the best estimation. This otimization has been resented in [9], as well as the criteria to decide which message air is the otimal one. A similar result has also been resented in [1]. The criteria to decide which messages are best for estimation uroses is alied whenever a new message is received. For instance, when rocess Õ receives message Ñ it has to decide whether Ñ is better than Ñ ½ for the urose of estimating the transmission delay of a subsequent message sent to Ô (Ñ in this examle). This is done by comaring Ì Ë and Ì Ê with Ì ¼ Ë and Ì ¼ Ê. The condition for using Ñ instead of Ñ ½ is the following: udate: Ì Ë ÌË ¼ µ ½ µ Ì Ê ÌÊ ¼ µ ½ µ (8) 4 Achieving a Stable Error Given a message air Ñ ½ Ñ, we have seen that with the round-tri duration measurement technique, Ñ Ò Ñ µ is always Ø Ñ Ò. In fact, since no assumtion whatsoever is made about Ñ ½, no restriction is made for the uer bound of Ø Ñ ½ µ and therefore the lower bound for Ø Ñ µ can be the lowest ossible, that is, Ø Ñ Ò. However, since a value for Ñ Ü Ñ ½µ has already been determined by rocess Õ when it sends Ñ (see Figure 2), this value could be sent to rocess Ô along with Ñ, making the latter able to ossibly restrict the uer bound for Ø Ñ ½ µ. In this manner, the lower bound for Ø Ñ µ could eventually be set to a value higher than Ø Ñ Ò and, in consequence, the interval of variation of Ø Ñ µ would be reduced, yielding a more accurate estimation of Ø Ñ µ. This simle reasoning is sufficient to rovide the intuition for the roosed imroved technique. The basic idea is to estimate the transmission delay of received messages using not only the send and receive timestams for the message air, but also the estimated value for the delay of the first message of the air. As we will see, the only drawback of the roosed imrovement is that it requires more information to be transmitted between rocesses than in revious round-tri based solutions. Figure 3 illustrates the fundamental relations that are used in this imroved duration measurement technique. q T S T 1 m 1 m 2 t min t(m 1 )-ε(m 1 ) T R t(m 1 )+ε(m 1 ) t max (m 2 ) t min (m 2 ) T 2 Figure 3. Round-Tri duration measurement using the imroved technique. To estimate the transmission delay of some message Ñ sent from rocess Õ to Ô, another message Ñ ½ must have been reviously sent from Ô to Õ. The round-tri delay of Ñ ½ Ñ can be used to determine the uer bound for Ø Ñ µ. It is also necessary to subtract the minimum (real) time sent by Õ before sending Ñ and the minimum ossible delay of Ñ ½, yielding the following equation: Ñ Ü Ñ µ Ì Ì Ë µ ½ µ Ì ½ Ì Ê µ ½ µ Ø Ñ ½ µ Ñ ½ µµ (9) This equation is very similar to that of the original round-tri technique (equation (5)). The difference is that now the last term deends on the estimation of Ø Ñ ½ µ, with Ø Ñ ½ µ Ñ ½ µ ossibly higher than Ø Ñ Ò. Note that Ø Ñ ½ µ Ñ ½ µ cannot be lower than Ø Ñ Ò, which guarantees that Ñ Ü Ñ µ is not higher than Ñ Ü Ñ µ. Although it may seem that Ø Ñ Ü Ñ µ can now be lower, we show that this is unfortunately not true (see Aendix A). The uer bound of a measured duration is always the same no matter which technique, the original or the imroved one, is used. The lower bounds, however, can be different. The hysical lower bound for Ø Ñ µ is obviously Ø Ñ Ò. But it might also be higher than that, deending on the estimation of Ø Ñ ½ µ. Taking the lowest ossible (real) time value for the round-tri duration, and subtracting the maximum (real) time sent by Õ before sending Ñ and the maximum ossible delay of Ñ ½, we obtain the following:

4 Ñ Ò Ñ µ Å Ø Ñ Ò Ì Ì Ë µ ½ µ Ì ½ Ì Ê µ ½ µ Ø Ñ ½ µ Ñ ½ µ (10) The exressions for the estimated transmission delay of Ñ follow directly from (9) and (10), assuming that the lower bound for Ø Ñ µ is higher than Ø Ñ Ò : Ñ µ Ø Ñ Ü Ñ µ Ø Ñ Ò Ñ µ Ì Ì Ë µ Ì ½ Ì Ê µ Ø Ñ ½ µ ÁÅÈ Ñ µ Ø Ñ Ü Ñ µ Ø Ñ Ò Ñ µ Ì Ì Ë µ Ì ½ Ì Ê µ Ñ ½ µ (11) (12) However, if we use the above equations when the lower bound of Ø Ñ µ is Ø Ñ Ò, the result is that Ñ µ ÁÅÈ Ñ µ will be lower than Ø Ñ Ò, which is obviously imossible. If this haens, then we can obtain the correct estimation for Ø Ñ µ making the following simle transformation: Ø ¼ÁÅÈ Ñ µ Ø Ñ Ü Ñ µ Ø Ñ Ò Ñ µ ØÁÅÈ Ñ µ ÁÅÈ Ñ µ Ø Ñ Ò ¼ÁÅÈ Ñ µ Ø Ñ Ü Ñ µ Ø Ñ Ò Ñ µ ØÁÅÈ Ñ µ ÁÅÈ Ñ µ Ø Ñ Ò (13) (14) Note that we have done these transformations simly using the knowledge that Ø Ñ Ü Ñ µ Ø Ñ µ Ñ µ and that Ø Ñ Ò Ñ µ Ø Ñ Ò. Note also that in equation (12) it is quite evident how the estimation error is ket almost constant. In fact, each time a new estimation is made, the error just increases by an amount corresonding to the drift of local clocks during the (tyically small) intervals of the roundtri measurement. This means that the error kees increasing, after each consecutive measurement, until a message is received for which the estimated lower bound is Ø Ñ Ò. Then, equations (13) and (14) will bring the error again to a lower value. In the most extreme case, if we could assume a erfect clock with ¼, the error would never increase and would just be reduced uon the recetion of faster messages. The imroved technique described so far requires more information to be exchanged between rocesses than the original round-tri technique. Besides the timestams Ì Ë, Ì Ê and Ì ½, the sender of Ñ also has to rovide Ø Ñ ½ µ and Ñ ½ µ. Furthermore, each rocess also has to kee more information than before, namely Ø Ñµ, ѵ, ÌË Ñ and Ì Ê Ñ for the best message Ñ received from each other rocess. These extra requirements are the trade-off for achieving best estimations. q A m 1 B C m 2 D E t(m 1 ),ε(m 1 ) t(m 2 ),ε(m 2 ) F mk Figure 4. Comaring messages Ñ ½ and Ñ. Just like in the round-tri technique, every received message is a otential best message. Hence, uon the recetion of a new message it is necessary to determine whether this message is better than the currently best one. The criteria to consider a message better than another is strictly related to its otential to roagate smaller estimation errors. An old message with a very small associated error can be better than a new message with a large error. The exact exression that must be used to comare two received messages is resented below (see Figure 4 for reference). In Aendix A we rove the correctness of this exression. Udate: Ñ µ µ µ Ñ ½ µ (15) Examle: The imact of using the imroved technique instead of the original one is easily observed when the real transmission delay of a message is visibly higher than that of revious messages. q m 1 m 2 m 3 m 4 Figure 5. Examle of imroved transmission delay estimation.

5 For instance, in the examle of Figure 5 there is a message, Ñ, that is slower than the revious ones. When using the original round-tri technique to estimate Ø Ñ µ, the estimation error will be nearly half of the round-tri delay of the air Ñ Ñ. On the other hand, when using the imroved round-tri this error is inherited from the estimation error of Ø Ñ µ, which is at most half of the round-tri delay of the air Ñ Ñ, clearly smaller than that of air Ñ Ñ. The occurrence of messages with transmission delays higher than normal is thus comletely irrelevant for the estimation errors obtained with the imroved technique. The error is ket almost stable (see Figure 9). The reader should note another interesting effect than can only be observed when the imroved technique is used. Since the technique allows the best errors to be reserved from a message to the consecutive one, the occurrence of a single message air of two fast messages is sufficient to establish a small error that will be used in all subsequent estimations. This can be articularly interesting for systems where the resource utilization and the delays can be ket constantly high during long eriods of time. 5 The Protocol In this section we describe a rotocol that uses the imroved round-tri duration measurement technique. This rotocol has been imlemented as art of the distributed duration measurement service of our TCB rototye, develoed for the Real-Time Linux oerating system [15] (see Section 6). The exlanation is divided in three arts for simlicity. We first describe the constants and global variables that are used in the rotocol (Figure 6). Then we describe the main loo of the rotocol, resented in Figure 7 and finally we exlain the more functional oerations executed by the rotocol, deicted in Figure 8. // Constants // ÑÝ Id of executing rocess // Maximum drift rate of local hardware clocks // Ø Ñ Ò Minimum message delay // Global Variables // ËÌ, ÊÌ, Ä, ÊÊ are arrays with entries for each rocess // ËÌ Ôµ Send Timestam of best message received from // ÊÌ Ôµ Receive Timestam of best message received from // Ä Ôµ Delay of best message received from // ÊÊ Ôµ Error of best message received from // Ð Ñ Estimated delay for received message Ñ // ÖÖ Ñ Error of estimation for received message Ñ // Global Function // ص Current hardware clock value Figure 6. Constants and global variables. Each rocess Ô runs an instantiation of this duration measurement service. They all have a different ÑÝ value, but and Ø Ñ Ò are the same in all instances. Since it is necessary to kee informations regarding the best message received from every rocess, we use arrays ËÌ, ÊÌ, Ä and ÊÊ to store this information. The size of these arrays is Æ, where Æ is the number of rocesses in the system. Variables Ð Ñ and ÖÖ Ñ kee the estimated delay and its associated error, that are returned to the user at the end of the main loo (Figure 7). We assume that it is ossible to read the local clock value using function ص, which returns ositive timestams. task Distributed Duration Measurement for all do ËÌ Ôµ ½; ÊÌ Ôµ ¼; Ä Ôµ ½; ÊÊ Ôµ ½; end do loo when request to send Ñ using Ø timestam do broadcast Ñ Ø ËÌ ÊÌ Ä ÊÊ ; end do when Ñ Ø Ñ ËÌ Ô ÊÌ Ô Ä Ô ÊÊ Ô received from Ô do ÖØ Ñ Øµ; if ÊÌ Ô ÑÝ µ ¼ first message (Ô, Ø Ñ ÖØ Ñ); else if Ä Ô ÑÝ µ ½ second message (Ô, Ø Ñ ÖØ Ñ ËÌ Ô ÊÌ Ô); else normal message (Ô, Ø Ñ ÖØ Ñ ËÌ Ô ÊÌ Ô Ä Ô ÊÊ Ô); end if deliver Ñ with ( Ð Ñ ÖÖ Ñ); end do end loo end task Figure 7. Pseudo code for the main loo. The main body of the rotocol has an initialization block, followed by an execution loo. When the rotocol starts, no messages have yet been received from other rocesses. Hence, we assume to have received imaginary initialization messages at instant ¼, with send timestam ½, and with infinite delays and errors. In the main loo we define two ossible entry oints, corresonding to alication requests to send messages and to messages received from the network. In this rotocol the send timestam Ø is rovided by the alication since this is imosed by the interface of the TCB duration measurement service (see [14] for a comlete descrition of this interface). The rotocol resented here assumes that the value rovided for Ø is higher than any of the values in ÊÌ. We use a broadcast service to avoid the need to secify a destination rocess. Nevertheless, when a message is sent it is necessary to include the four arrays and also the send timestam. When no message is transmitted during a long eriod, the estimation error of received messages tends to increase due to the drift rate of local clocks. Therefore, in the imlementation of the duration measurement service

6 we added an additional action to force eriodic (service secific) message transmissions. These eriodic messages can be viewed as synchronization messages that revent the errors to increase indefinitely. This action is not resented here since it is not strictly required by the imroved technique. The second entry oint corresonds to messages received from other rocesses. Uon the recetion of a message a receive timestam is immediately obtained. Then, the message is rocessed accordingly to its tye. Messages can be of three logical tyes: First messages When the service at Ô initiates, it will start receiving messages from the other rocesses. These messages do not contain any information about messages sent by Ô to the other rocesses. They are simly first messages that will be used to estimate the delay of subsequent messages. They are identified by having ÊÌ Ô ÑÝ µ ¼, which means that rocess Ô has never received a message from rocessor ÑÝ. Second messages After sending its first message Ñ to all other rocesses, rocess Ô will start receiving messages that have already been sent after their senders have received Ñ. It is thus ossible to estimate the delay for these second messages using the original round-tri technique. They can be identified because Ä Ô ÑÝ µ ½, that is, Ô has received a message from rocessor ÑÝ but has not been able to estimate its delay. Normal messages All other messages are received in a state that allows the imroved technique to be alied. Therefore we call them normal messages. It is ossible to receive several messages of the same tye, and each message tye is rocessed by a different function. However, all the rocessing functions assign values to Ð Ñ and ÖÖ Ñ, which are returned to the alication, along with the received message, in the end of the loo. Each of the message rocessing functions does two things: a) it estimates the transmission delay of messages; b) it udates, if necessary, the array entry of the rocess from which the message has been received. Estimating the message transmission delay of first messages is not ossible, since there is not way to establish an uer bound for this delay. Therefore, the first message() function simly assigns the ½ value to Ð Ñ and ÖÖ Ñ. To determine if a new first message is better than a revious one, it is necessary to aly equation (8) of the original roundtri technique. This is so because first messages will be aired with second messages to estimate the delay of the latter, and the original round-tri technique will be alied. The udate info() function is simly used to store the new information in the local arrays. udate info (Ô, Ð Ñ ÖÖ Ñ Ø Ñ ÖØ Ñ) begin ËÌ Ôµ Ø Ñ; ÊÌ Ôµ ÖØ Ñ; Ä Ôµ Ð Ñ; ÊÊ Ôµ ÖÖ Ñ; end first message (Ô, Ø Ñ ÖØ Ñ) begin Ð Ñ ½; ÖÖ Ñ ½; if Ø Ñ ËÌ Ôµµ ½ µ ÖØ Ñ ÊÌ Ôµµ ½ µ udate info (Ô, Ð Ñ ÖÖ Ñ Ø Ñ ÖØ Ñ); end if end second message (Ô, Ø Ñ ÖØ Ñ ËÌ Ô ÊÌ Ô) begin Ð Ñ ÖØ Ñ ËÌ Ô ÑÝ µµ ½ µ Ø Ñ ÊÌ Ô ÑÝ µµ ½ µ ØÑ Ò µ ; ÖÖ Ñ Ð Ñ; if ÖÖ Ñ ÊÊ Ôµ Ø Ñ ËÌ Ôµµ ÖØ Ñ ÊÌ Ôµµ udate info (Ô, Ð Ñ ÖÖ Ñ Ø Ñ ÖØ Ñ); end if end normal message (Ô, Ø Ñ ÖØ Ñ ËÌ Ô ÊÌ Ô Ä Ô ÊÊ Ô) begin Ð Ñ ÖØ Ñ ËÌ Ô ÑÝ µµ ØÑ ÊÌ Ô ÑÝ µµ Ä Ô ÑÝ µ; ÖÖ Ñ ÊÊ Ô ÑÝ µ ÖØÑ ËÌ Ô ÑÝ µµ Ø Ñ ÊÌ Ô ÑÝ µµ; if Ð Ñ ÖÖ Ñ ØÑ Ò ÓÖÖ Ø Ð Ñ Ð Ñ ÖÖÑ ØÑ Ò µ ; ÓÖÖ Ø ÖÖ Ñ Ð Ñ ÖÖÑ Ø Ñ Ò µ ; Ð Ñ ÓÖÖ Ø Ð Ñ; ÖÖ Ñ ÓÖÖ Ø ÖÖ Ñ; end if if ÖÖ Ñ ÊÊ Ôµ Ø Ñ ËÌ Ôµµ ÖØ Ñ ÊÌ Ôµµ udate info (Ô, Ð Ñ ÖÖ Ñ Ø Ñ ÖØ Ñ); end if end Figure 8. Message rocessing functions. As just said, the transmission delay of second messages is estimated with the original round-tri technique (equations (6) and (7)). However, the udate decision is now based on the imroved technique rule (exression (15). Finally, the normal message() function rocesses every other message, fully using the imroved round-tri technique. Ð Ñ and ÖÖ Ñ are obtained using equations (11) and (12), and eventually equations (13) and (14), if condition Ð Ñ ÖÖ Ñ Ø Ñ Ò evaluates to true. The udate art is equal to the one of function second message(). 6 Evaluation Results A distributed duration measurement service using the rotocol just described, has been imlemented as art of our RT-Linux TCB rototye [4]. Therefore, we were able to

7 Delay (ms) 4 3,5 3 2,5 2 1,5 1 0,5 0 Measurements in sequencial time IMP Error Uer bound RT Error Figure 9. Measurement uer bounds and errors using the imroved (IMP) and the original round-tri (RT) techniques. evaluate the effectiveness of the roosed imroved roundtri technique when comared to its original version. The tests were erformed in our distributed systems laboratory, using Pentium based PCs connected to a 10 Mbit Ethernet LAN. To enhance the recision of our evaluation results, we have also used a secial measurement tool that we develoed, the Event Timestaming Tool [10], to externally measure the transmission delay of messages. Since our aim was to comare both techniques, we simly needed to generate sequences of messages airs, for which only two machines were necessary. Given that the TCB duration measurement service only imlements the imroved technique, for the comarisons we had to extend the service with an imlementation of the original version of the round-tri technique, exactly as described in [9]. Both imlementations run in arallel, using exactly the same inut values, that is, the same send and receive timestams. They both outut airs of Ð Ñ ÖÖ Ñ values that are used for the comarison. The exeriment consisted in the execution of two alications, a client and a server, resectively concerned with eriodically sending a message, and relying to every received message. We have configured the client to send a message every 500ms. Therefore, in all the figures resented bellow the samles deicted in the X axis corresond to consecutive measurements obtained with intervals of about 500ms. Note that these intervals are affected not only by the jitter of the scheduling delay (of the client s sending task), but also by the jitter of the overall round-tri delay. The secific content of the messages is irrelevant for the exeriment. Both alications were imlemented on to of the TCB duration measurement service. Their only functionality, besides sending and receiving messages, was to store the returned air of measurement values in a local file. We had to set the values of some constants used in the rotocol. The drift rate was set to ½¼ (tyical values range from ½¼ to ½¼ [8]) and we assumed the minimum message transmission delay to be zero (Ø Ñ Ò ¼). The size of the arrays was set to a value larger than two (the number of used rocessors). Relatively to the conditions of the environment during the exeriment, in articular the network and system loads, we have forced a scenario with almost no activity (tyically idle), interleaved with short eriods of intensive network utilization. This unstable behavior is erfectly visible in the figures resented below and, as exected, allows to clearly observing the imrovements obtained with the roosed technique. To overload the network we simly used the Unix ing command, with the flood otion enabled. The exeriment was reeated several times, just to comare the results and verify their coherence. However, for the uroses of this aer we have just selected a reresentative set of results, one that exhibits some eriods of instability during which the differences between the two techniques can be observed. Note that the imact of this imrovement in real alications will deend on the articular alication and on the frequency and duration of these unstable eriods. In Figure 9 we comare the error values achieved by both techniques in a series of consecutive measurements. The uer bound values are also reresented to rovide an idea of the system behavior during the exeriment. Higher uer bounds tyically corresond to messages transmitted during eriods of more intensive traffic. The most imortant result that may be observed in this figure is the almost stable error achieved by the imroved technique (IMP Error), in contrast with the variable error achieved by the other technique (RT Error). This result clearly confirms our exectations. Note that the RT Error line closely follows the Uer bound line, as dictated by the original round-tri technique (in this case, since Ø Ñ Ò was

8 Frequency Estimation error (µs) IMP technique RT technique Figure 10. Distribution of estimation errors. set to zero, RT Error is always half of the Uer bound). If non-zero values were assumed for Ø Ñ Ò, this would simly shift all the lines down. To more clearly observe, and comare, the disersion of the measurement errors achieved with the two techniques, we analyzed the frequency of each observed error value, and resent the result in Figure 10. Error values with no occurrences are obviously not deicted, which means that the X axis scale is irregular. The higher estimation error observed with the IMP technique was near 490 s, while with the RT technique we observed several error values above 500 s. This result shows that the imroved technique can ossibly be used to construct a distributed duration measurement service that ensures a bounded measurement error. Such a service can be very useful in network monitoring systems, as exlained in [5], since it allows the establishment of accuracy bounds for the observations. The accuracy of the estimations obtained with the imroved technique can be observed in Figure 11. In this figure we comare the real delay of message transmissions, with the delays that were estimated using both techniques. During the eriods of stability, when the message transmission delays are low, both techniques seem to rovide accurate estimations. However, when a higher delay occurs, only the IMP technique is able to rovide accurate estimations. A final note on the imact of increased message size incurred in the IMP technique. It is true that instead of two, now four timestams must be sent for each rocess receiving some timed message. But the fundamental roblem is scalability, and this one is common to both aroaches. In ractice, if there is an uer bound, say Æ, for the number of rocesses allowed in a system using the RT technique, this bound must be reduced to Æ when using the IMP technique. 7 Conclusions We have resented a new technique to measure the duration of distributed actions and to read remote clocks. With this technique it is ossible to achieve a lower reading error of remote clocks than with the well know round-tri duration measurement technique roosed by Cristian. Therefore, we have referred to it as the Imroved Round-Tri Technique. We described in detail the intuition that leads to this imroved technique, and we rovided the equations that should be used to obtain the imroved results. Besides reducing the error, and erhas imortantly, this technique kees the estimation errors almost constant, indeendently of the real duration of the action that is being measured. This is an interesting feature for any distributed duration measurement service. We also resented and described a rotocol that imlements the imroved technique, which we have used in the imlementation of the distributed duration measurement service of the Timely Comuting Base (TCB). We were able to conduct an exeriment to evaluate the effectiveness of the roosed technique. The results have shown a few interesting features of the imroved technique: it is indeed able to rovide readings with almost stable errors; it revents measurement errors from deending on the measured delay (which might make them extremely high); its estimated delays are much closer to the real delays than the ones obtained with the original round-tri technique. These ositive characteristics render the imroved round-tri duration measurement technique extremely effective and efficient, in short an excellent candidate for an

9 10 8 Delay (ms) Measurements in sequential time IMP Delay Real Delay RT Delay Figure 11. Estimated delays and real (measured) delay. embedded runtime suorted service for distributed alications. This was the aroach taken in our TCB rototye. A Proofs Theorem 1: Given any message Ñ, the uer bound determined for the message delivery delay of Ñ using the imroved technique is equal to the determined uer bound using the round-tri technique, Ñ Ü Ñµ ØÊÌ Ñ Ü Ñµ. Proof: We use Figure 12 to visually guide this roof. Consider the deicted sequence of messages. We will show that the theorem is valid for message Ñ ½, and that it can be generalized to any other message. We will start by showing that Ñ ½ can be the first message in some sequence that might ossibly have Ñ Ü Ñ Ü all revious messages in the sequence are guaranteed to have the same uer bound using any of the techniques and we will show that in site of this ossibility the uer bounds are nevertheless equal. Since the same reasoning can recursively be alied to the first subsequent message for which it might also be ossible to have, the theorem can be generalized for every subsequent message. Ñ Ü ØÊÌ Ñ Ü q S m k-2 m k-1 m k m k+1 R A B C D E Figure 12. Uer bound reservation for Ñ ½. F Let us assume that the message air Ñ Ñ ½ is the one used to determine the uer bounds (Ñ has been considered by Ô the best message so far). The exressions that rovide the uer bounds for Ñ ½ follow from (5) and (9): Ñ Ü Ñ ½µ µ ½ µ µ ½ µ Ø Ñ Ò Ñ Ü Ñ ½µ µ ½ µ µ ½ µ For Ø Ñ µ Ñ Ü Ñ ½µ Ñ µ Ø Ñ Ò it immediately follows that Ñ Ü Ñ ½µ. We are left with the ossibility that Ø Ñ µ Ñ µµ Ø Ñ µ Ñ µ Ø Ñ Ò (16) In this case it is reasonable to believe that Ñ Ü Ñ ½µ may be lower than Ñ Ü Ñ ½µ, which we will rove to be untrue. Now observe that for inequality (16) to be ossible, it is necessary to consider the existence of a message air receding Ñ in the same sequence. We assume this message air to be Ñ ½ Ñ, as deicted in Figure 12. Furthermore, we assume these two messages to be the first ones in the sequence, so that Ñ ½ is clearly the first message for which Ñ Ü ØÊÌ Ñ Ü may be ossible. With these assumtions it is also clear that the theorem holds for all messages receding Ñ ½. Let us write the uer bound for Ø Ñ ½ µ (alying (5)): Ø Ñ Ü Ñ ½ µ ˵ ½ µ ʵ ½ µ Ø Ñ Ò (17) Given that Ñ Ò Ñ µ Ø Ñ µ Ñ µ Ø Ñ Ò, we can use (10) to write the following: Ñ Ò Ñ µ Ø Ñ Ò µ ½ µ µ ½ µ Ø Ñ ½ µ Ñ ½ µµ Ø Ñ Ò µ ½ µ µ ½ µ Ø Ñ Ü Ñ ½ µ Ø Ñ Ò

10 µ ½ µ µ ½ µ ˵ ½ µ ʵ ½ µ Ø Ñ Ò Ø Ñ Ò Êµ ½ µ ˵ ½ µ Ø Ñ Ò Ø Ñ Ò Ëµ ½ µ ʵ ½ µ On the other hand, from (8) we know that Ñ is only considered a best message than Ñ if ˵ ½ µ ʵ ½ µ. So we must conclude that our initial assumtion that the message air Ñ Ñ ½ can be used to determine the uer bound of Ñ ½ is in contradiction with the fact exressed in (16). We must therefore rove that the theorem still holds when the message air Ñ Ñ ½ is used to obtain Ñ Ü Ñ ½µ. In this case we have: Ñ Ü Ñ ½µ ˵ ½ µ ʵ ½ µ Ø Ñ Ò which we must comare with Ñ Ü Ñ ½µ: Ñ Ü Ñ ½µ µ ½ µ µ ½ µ Ø Ñ µ Ñ µµ µ ½ µ µ ½ µ ʵ ½ µ ˵ ½ µ Ø Ñ Ò µ ˵ ½ µ ʵ ½ µ Ø Ñ Ò It follows that Ñ Ü Ñ ½µ, which comletes our roof. Theorem 2: Given any two messages, Ñ ½ and Ñ, the former sent at and received at with estimation error of ѽ, and the latter sent at and received at with estimation error of Ñ, Ñ is considered to be best for the accuracy of the imroved round-tri technique if Ñ Ñ½ µ µ Proof: To comare messages Ñ ½ and Ñ we analyze their imact on the estimation of Ø Ñ µ for a subsequent message Ñ (see Figure 4). The best message is the one that allows Ø Ñ µ to be estimated with a smaller error Ñ µ. Alying (12) to the round-tri airs Ñ ½ Ñ and Ñ Ñ we obtain the following: Ñ Ü Ñ ½µ ѽ Ñ µ Ñ ½ µ µ µ Ñ ½ µ µ µ µ µ Ñ Ñ µ Ñ µ µ µ Hence, Ñ is better than Ñ ½ if: Ñ Ñ µ ѽ Ñ µ µ µ Ñ µ µ µ References Ñ ½ µ µ µ µ µ Ñ µ Ñ ½ µ µ µ [1] G. Alari and A. Ciuffoletti. Imlementing a robabilistic clock synchronization algorithm. Journal of Real-Time Systems, 13(1):25 46, July [2] E. Anceaume and I. Puaut. Performance evaluation of clock synchronization algorithms. Technical Reort PI-1208, IRISA, Camus de Beaulieu, Rennes Cedex, France, October [3] K. Arvind. Probabilistic clock synchronization in distributed systems. IEEE Transactions on Parallel and Distributed Systems, 5(5): , May [4] A. Casimiro, P. Martins, and P. Veríssimo. How to build a timely comuting base using real-time linux. In Proceedings of the 2000 IEEE Intl. Worksho on Factory Communication Systems, ages , Porto, Portugal, Setember [5] A. Casimiro and P. Veríssimo. Using the timely comuting base for deendable qos adatation. In Proceedings of the 20th IEEE Symosium on Reliable Distributed Systems, New Orleans, USA, October IEEE Comuter Society Press. [6] F. Cristian. Probabilistic clock synchronization. Distributed Comuting, 3(3): , [7] F. Cristian and C. Fetzer. Probabilistic internal clock synchronization. In Proceedings of the 13th Symosium on Reliable Distributed Systems, ages 22 31, Dana Point, California, USA, October IEEE Comuter Society Press. [8] F. Cristian and C. Fetzer. The timed asynchronous distributed system model. IEEE Transactions on Parallel and Distributed Systems, ages , June [9] C. Fetzer and F. Cristian. A fail-aware datagram service. In Proceedings of the 2nd Annual Worksho on Fault-Tolerant Parallel and Distributed Systems, Geneva, Switzerland, Aril [10] P. Martins and A. Casimiro. Event timestaming tool: a simle c based kernel to timestam distributed events. DI/FCUL TR 00 4, Deartment of Comuter Science, University of Lisbon, July [11] D. L. Mills. Internet time synchronization: the network time rotocol. IEEE Transactions on Communications, 39(10): , October [12] A. Olson and K. G. Shin. Probabilistic clock synchronization in large distributed systems. IEEE Transactions on Comuters, 43(9): , Setember [13] F. B. Schneider. Understanding rotocols for Byzantine clock synchronization. Technical Reort TR , Cornell University, Det. of Comuter Science, Uson Hall, Ithaca, NY 14853, August [14] P. Veríssimo, A. Casimiro, and C. Fetzer. The timely comuting base: Timely actions in the resence of uncertain timeliness. In Proceedings of the International Conference on Deendable Systems and Networks, ages , New York City, USA, June IEEE Comuter Society Press. [15] V. Yodaiken and M. Barabanov. A real-time linux. In Proceedings of the USENIX conference, htt://rtlinux.cs.nmt.edu/rtlinux/aers/usenix.s.gz.