CMD MDS Recovery DLD

Size: px

Start display at page:

Download "CMD MDS Recovery DLD"

Ami Poole
5 years ago
Views:

1 CMD MDS Recovery DLD Mike Pershin February 6, Introduction This document describes recovery changes in CMD. 2 Requirements The CMD environment requires the reviewed recovery due to major changes in functionality. The new recovery design covers the following issues: recovery cases occurred due to cross-ref situations (MDS-MDS recovery); land all recovery fixes from CMD2 project; changes in recovery logic due to FID introducing; changes due to moving network things to the MDT level; changes in recovery API due to layered nature of the new MDS. 3 Functional specification 3.1 Recovery process in the new MDS The recovery starts as usual - while setup process, the MDT reads last_rcvd file. If it s size is not zero then recovery starts by setting Ñ Ø¹ Ñ Ø Ö ÓÚ Ö Ò ½. After that connecting clients will send the replay request. The new MDS consist of several layers and almost all of them have own part in recovery process. There are several stages in recovery process: 1. Network recovery. MDT receives the replay requests and handles them as usual 1

2 3.2 Transaction handling in new MDS 3 FUNCTIONAL SPECIFICATION (a) the ØÖ Ò ÒÓ value is taken from request. This value is used for operation and is returned back; (b) Ñ Ø Ò µ receives the request and processes them in order of ØÖ Ò ÒÓ. 2. Clients finish recovery and target_finish_recovery() calls Ñ ÔÓ ØÖ ÓÚ µ on MDS OBD. The same method is defined for Ù Ú so it invoke postrecovery process in all MDS layers; 3. MDD does LOV synchronize and orphan handling in Ñ ÔÓ ØÖ ÓÚ µ 3.2 Transaction handling in new MDS The layering nature of the new MDS implies that transactions can be done in MDD layer only. But other layers like MDT and CMM need to write persistent data in several cases. The transaction callbacks were designed to provide such capability. There are three callbacks that every layer can register and use - start, stop and commit one. During start callback it is possible to increase the number of blocks involved in transaction. The stop callback is need to write any additional persistent data in the current transaction. The commit callback is used in cases where some data should be updated when transaction is committed. The recovery in CMD uses all of these callbacks. Each transaction in new MDS has a context - the set of values which are pre-allocated by every layers if needed. This context is initialized when transaction starts and finished when it is committed. The transaction context is used to store any values which exists while transaction exists ÌÖ Ò ÒÓ handling The ØÖ Ò ÒÓ variable is used to track transactions, every new transaction has own ØÖ Ò ÒÓ number which is returned to the client. In new MDT layer the ØÖ Ò ÒÓ handling is encapsulated in transaction callbacks. 1. In the stop callback: (a) if Ñ Ø Ø Ö Ò Ó contain the ØÖ Ò ÒÓ ÆÍÄÄ already then this is replay case and that ØÖ Ò ÒÓ will be used; (b) otherwise the Ñ Ø¹ Ø ØÖ Ò ÒÓ value is taken as ØÖ Ò ÒÓ and increased; 2. the taken ØÖ Ò ÒÓ is stored in ØÖÙØ Ñ Ø Ø Ö Ò Ó for further usage and is stored also in ØÖÙØ Ñ Ø ØÜÒ Ò Ó for commit callback; 3. when the request handling is finished in MDT, the reply is prepared and ØÖ Ò ÒÓ is taken from Ñ Ø Ø Ö Ò Ó µ back and written to the reply; 4. when transaction is committed the MDT commit callback takes ØÖ Ò ÒÓ value from Ñ Ø ØÜÒ Ò Ó µ and uses it to update Ñ Ø¹ Ø ÓÑÑ ØØ value if needed. 2

3 3.3 Reconstruction 3 FUNCTIONAL SPECIFICATION LAST_RCVD update The LAST_RCVD file should be updated for recovery. This should be done at the end of any new transaction and the stop callback is natural place for this. The MDT stop callback reads and updates the LAST_RCVD file in the same manner as before. Also, the start callback is used to reserve enough space in transaction handle to do Ä ËÌ Ê Î update. The layered MDS introduces one problem with LAST_RCVD update. The result code should be written into record but there is no last result code at the time of calling the stop_callback and also that result code should be passed from thread to the callback somehow. The result code will be passed through ØÜÒ Ý in the Ñ ØÖ Ò ØÓÔ µ method. The stop callback will use this value while updating the LAST_RCVD. This implies that the rc code from MDD operation shouldn t be changed after writing to the LAST_RCVD. Note: In some cases, e.g. open() the result code after transaction is stopped could be modified by consequent operations like mdt_mfd_open(), etc. This is ok, while reconstruct we will repeat these steps again and result code will be updated in the same manner. The important thing with storing rc is that it is result of operation which change persistent data and cannot be repeated once done, that rc is needed to know persistent state. All other operations can be just repeated Last committed transaction update In normal operation the last_committed value is managed in the MDT commit callback. If committed transno is higher than mdt->last_committed value then last_committed = transno. The last committed value is returned to the client after that, so client can purge the committed requests from replay queue. 3.3 Reconstruction The MDT contains the methods for reconstruction of the requests. During reconstruction in the all cases the Ä ËÌ Ê Î is read and result code is got from it. This is enough for most reconstruction cases, but some needs more work to be done: create, setattr - get attributes. If object is remote one - return ¹ Ê ÅÇÌ to the client, so it will request the attributes from remote server. open - most complex case for reconstruction. Intent disposition value is saved also in Ä ËÌ Ê Î and is used with result code while reconstruction. If file was created while open, the its attributes and lov data are returned. The open() recovery is a matter of separated DLD. unlink, link, rename, close - return only result of operation from Ä ËÌ Ê Î 3

4 3.4 Replay changes 4 USE CASES 3.4 Replay changes While replaying the stop callback will use ØÖ Ò ÒÓ from the request. This is the only difference with normal case FID and replay Replay functionality on the MDS looks similar to the usual operations in the most cases and reuse the usual methods greatly because of FID design. The open/create replay becomes simpler due to the fact that client always knows FID for open/create operation so there is no separate code-path for replay operations. 4 Use cases All use cases are intended to 11/17 tests. The cluster is started up and clients runs multiple applications. One of the nodes is failed and other nodes/clients shouldn t be affected. Due to requirements of test 17 the recovery time should be not more than 5 minutes in each case. 4.1 Client failure Client is failed and normal operations should continue. Other clients and servers should receive no errors. The major areas in MDS recovery in that case are closing all opened files from that client and reading LAST_RCVD records upon reconnection. 4.2 Singe MDS failure in CMD One MDS is failed, recovered and normal operations continue. All aspects of recovery can happens here - resent/reconstruct, replay. This is the major use case for MDS recovery. 4.3 OST failure OST is failed and recovery is started, clients should get no errors. 4

5 5 LOGIC SPECIFICATION 5 Logic specification 5.1 Recovery process Starting the recovery Ø Ø ÒØ Ñ Ø Ò Ø ÒØ Ø ÓÒ Ø ØÖÙØ Ù ÓÒØ ÜØ ØÜØ ØÖÙØ Ñ Ø Ú Ñ Ø ÙÒ Ò ÓÒ Ø ÖÚ Þ µ ØÖÙØ Ñ Ø ÖÚ Ö Ø Ñ ²Ñ Ø¹ Ñ Ø Ñ ØÖÙØ Ñ Ø ÒØ Ø Ñ ØÖÙØ Ó Ú Ó Ñ Ø¹ Ñ Ø Ñ ÚºÑ Ù Úº Ó Ó Ø Ó ¼ ÒØ Ü ÒØ Ö ¼ ÆÌÊ» Ï Ò Û Ó Ò Å Ë ÙØ ÓÛÒ Û Ú Ø Ø ØÖ Ò ÒÓ ÒØÓ Ø Öº Á Û Ò ÒØ Û Ø Ö Ø ØÖ Ò ÒÓ Ú Ù Ø Ò Ø Ó ÒØ Ñ Ý Ò Ö ÓÚ ÖÝ ÓÒ º» Ç ÄÄÇ ÈÌÊ Ñ µ Ñ µ Ê ÌÍÊÆ Ö ¹ ÆÇÅ Åµ ÓÖ Ü ¼ Ó ¾ ØÓ ÔÙ Ñ ¹ Ñ ÒØ Ø ÖØµ Ó Ø ÖÚ Þ Ü µ Ù Ø ØÖ Ò ÒÓ ØÖÙØ Ó ÜÔÓÖØ ÜÔ ØÖÙØ Ñ Ø ÜÔÓÖØ Ø Ñ Ó ¾ ØÓ ÔÙ Ñ ¹ Ñ ÒØ Ø ÖØµ Ü ½ ØÓ ÔÙ Ñ ¹ Ñ ÒØ Þ µ Ö Ñ Ø¹ Ñ Ø Ø ÖÚ ¹ Ó Ó Ý ÓÔ ¹ Ó Ö ØÜØ Ñ Ø¹ Ñ Ø Ø ÖÚ Ñ Þ Ó Ñ µ ²Ó µ Ö Þ Ó Ñ µµ Ö ¼ Ö ¼µ Ö ¹ ÍÄÌ Öµ ÊÊÇÊ ÖÖÓÖ Ö Ò Å Ë ± Ü ± Ó ±Ù Ö ± Ò Ä ËÌ Ê Î Ü Ó Öµ Ö» Ö ÖÖÓÖ ÓÙ Ò³Ø Ù Ø ÖØÙÔ ØÓ» Ñ ¹ Ñ ÙÙ ¼ ³ ¼³µ Í ÁÆ Ç ÔÔ Ò Þ ÖÓ ÒØ Ø Ó Ø ± Ò 5

6 5.1 Recovery process 5 LOGIC SPECIFICATION Üµ ÓÒØ ÒÙ Ø ØÖ Ò ÒÓ ØÓ ÔÙ Ñ ¹ Ñ Ø ØÖ Ò ÒÓµ» Ì ÜÔÓÖØ Ö Ò ÙÔ Ý Ñ Ø Ó ÓÒÒ Ø µ Ó Ø Ý Ò ØÓ Ø ÙÔ Ö ÜÔÓÖØ Ñ Ø Ó ÓÒÒ Ø µ Ó º» Í À Ê ÎÊÆ ÄÁ ÆÌ ÙÙ ± Ü ± Ö ÄÈÍ ÖÚ Ö ÄÈÍ Ü ÄÈÍ Ò Ñ ¹ Ñ ÙÙ Ü Ø ØÖ Ò ÒÓ ØÓ ÔÙ Ñ ¹ Ñ Ø ØÖ Ò ÒÓµ ØÓ ÔÙ Ñ ¹ Ñ Ø Ü µµ ÜÔ Ò Û ÜÔÓÖØ Ó ØÖÙØ Ó ÙÙ µñ ¹ Ñ ÙÙ µ ÁË ÊÊ ÜÔµµ ÇÌÇ ÖÖ ÒØ Ö ÈÌÊ ÊÊ ÜÔµµ Ñ ² ÜÔ¹ ÜÔ Ñ Ø Ø Ñ ¹ Ñ Ñ Ñ Ö Ñ Ø ÒØ ØÜØ Ñ Ø Ñ Üµ Ä ËË ÊÌ Ö ¼ Ö ± Ò Öµ» Ò³Ø Ü Ø Ò» ÜÔ¹ ÜÔ Ö Ô Ý Ò ½ ÜÔ¹ ÜÔ ÓÒÒ Ø Ò ¼ Ó ¹ Ó Ö ÓÚ Ö ÒØ Ó ¹ Ó Ñ Ü Ö ÓÚ Ö ÒØ ÜÔÓÖØ ÔÙØ ÜÔµ Í ÇÌÀ Ê ÒØ Ø Ü ± Ø ØÖ Ò ÒÓ ÄÈÍ Ò Ü Ø ØÖ Ò ÒÓµ Ô Ò Ó ²Ñ Ø¹ Ñ Ø ØÖ Ò ÒÓ Ó µ Ø ØÖ Ò ÒÓ Ñ Ø¹ Ñ Ø Ø ØÖ Ò ÒÓµ Ñ Ø¹ Ñ Ø Ø ØÖ Ò ÒÓ Ø ØÖ Ò ÒÓ Ô Ò ÙÒÓ ²Ñ Ø¹ Ñ Ø ØÖ Ò ÒÓ Ó µ ÖÖ ÒØ Ç Ê ÈÌÊ Ñ µ Ê ÌÍÊÆ Öµ Ø Ø ÒØ Ñ Ø Ò Ø ÖÚ Ö Ø ÓÒ Ø ØÖÙØ Ù ÓÒØ ÜØ ØÜØ ØÖÙØ Ñ Ø Ú Ñ Øµ ØÖÙØ Ñ Ø ÖÚ Ö Ø Ñ ²Ñ Ø¹ Ñ Ø Ñ ØÖÙØ Ñ Ø ÒØ Ø Ñ ÆÍÄÄ ØÖÙØ Ó Ú Ó Ñ Ø¹ Ñ Ø Ñ ÚºÑ Ù Úº Ó Ó Ø Ó ¼ ÙÒ Ò ÓÒ Ø ÖÚ Þ ¼ Ù ÑÓÙÒØ ÓÙÒØ ÒØ Ü ÒØ Ö 6

7 5.1 Recovery process 5 LOGIC SPECIFICATION ØÖÙØ Ñ Ø Ø Ö Ò Ó Ò Ó ØÖÙØ Ø Ó Ø Ø Ñ Ø¹ Ñ Ø Ø ÖÚ ØÖÙØ Ù ØØÖ ÆÌÊ ººº Ø ÖÚ Þ ¹ Þ Ø ÖÚ Þ ¼µ Ä ÇÆËÇÄ Ï ÊÆ ± Ò Û Ò Ø Þ Ò Ò Ó ¹ Ó Ò Ñ µ Ñ ÑÔÝ Ñ ¹ Ñ ÙÙ Ó ¹ Ó ÙÙ ºÙÙ Þ Ó Ñ ¹ Ñ ÙÙ µµ Ñ ¹ Ñ Ø ØÖ Ò ÒÓ ¼ ÑÓÙÒØ ÓÙÒØ Ñ ¹ Ñ ÑÓÙÒØ ÓÙÒØ ¼ Ñ ¹ Ñ ÖÚ Ö Þ ÔÙ ØÓ ¾ ÄÊ Ë ÊÎ Ê ËÁ µ Ñ ¹ Ñ ÒØ Ø ÖØ ÔÙ ØÓ ¾ ÄÊ ÄÁ ÆÌ ËÌ ÊÌµ Ñ ¹ Ñ ÒØ Þ ÔÙ ØÓ ½ ÄÊ ÄÁ ÆÌ ËÁ µ Ñ ¹ Ñ ØÙÖ ÖÓÓÑÔ Ø ÔÙ ØÓ ¾ Ç ÊÇ ÇÅÈ Ì ÄÇÎÇ ÂÁ µ Ñ ¹ Ñ ØÙÖ ÒÓÑÔ Ø ÔÙ ØÓ ¾ Ç ÁÆ ÇÅÈ Ì Å Ì Ç ÁÆ ÇÅÈ Ì ÇÅÅÇÆ ÄÊµ Ö Ø¹ Ó Ó Ý ÓÔ ¹ Ó Ö ØÜØ Ø Ñ Þ Ó Ñ µ ²Ó µ Ö Þ Ó Ñ µµ Ö ¼ Ö ¼µ Ö ¹ ÍÄÌ Öµ ÊÊÇÊ ÖÖÓÖ Ö Ò Å Ë ± Ö ± Ò Ä ËÌ Ê Î Öµ ÇÌÇ ÓÙØ Öµ ØÖÑÔ Ñ ¹ Ñ ÙÙ Ó ¹ Ó ÙÙ ºÙÙ µ ¼µ Ä ÇÆËÇÄ ÊÊÇÊ ÌÖÝ Ò ØÓ Ø ÖØ Ç ± Ù Ò Ø ÛÖÓÒ ± º Ï Ö Ø» Ú» ÒÑ ÒØ Ö ÖÖ Ò Ò Ó ¹ Ó ÙÙ ºÙÙ Ñ ¹ Ñ ÙÙ µ ÇÌÇ ÓÙØ Ö ¹ ÁÆÎ Äµ ÑÓÙÒØ ÓÙÒØ ØÓ ÔÙ Ñ ¹ Ñ ÑÓÙÒØ ÓÙÒØµ ººº Ñ Ø Ò Ø ÒØ Ø ØÜØ Ñ Ø Ø ÖÚ Þ µ Ó ¹ Ó Ø ÓÑÑ ØØ Ñ Ø¹ Ñ Ø Ø ØÖ Ò ÒÓ Ó ¹ Ó Ö ÓÚ Ö ÒØ µ Ï ÊÆ Ê ÇÎ Ê ÖÚ ± ± Ö ÓÚ Ö ÒØ Ø ØÖ Ò ÒÓ ÄÈÍ Ò Ó ¹ Ó Ò Ñ 7

8 5.1 Recovery process 5 LOGIC SPECIFICATION Ó ¹ Ó Ö ÓÚ Ö ÒØ Ñ Ø¹ Ñ Ø Ø ØÖ Ò ÒÓµ Ó ¹ Ó Ò ÜØ Ö ÓÚ ÖÝ ØÖ Ò ÒÓ Ó ¹ Ó Ø ÓÑÑ ØØ ½ Ó ¹ Ó Ö ÓÚ Ö Ò ½ Ó ¹ Ó Ö ÓÚ ÖÝ Ø ÖØ ÍÊÊ ÆÌ Ë ÇÆ Ë» ÇÒÝ Ù ÓÖ ÔÖÓ Ø ØÙ» Ó ¹ Ó Ö ÓÚ ÖÝ Ò Ó ¹ Ó Ö ÓÚ ÖÝ Ø ÖØ Ç Ê ÇÎ Ê ÌÁÅ ÇÍÌ Recovery starts if the Ó ¹ Ó Ö ÓÚ Ö ÒØ is set in 1 until recovery will finish and Ó ÔÓ ØÖ ÓÚ µ will be invoked. ¼º The Ó ¹ Ó Ö ÓÚ Ö Ò Post-recovery New method for Ù Ú is defined - Ù ÔÓ Ø Ö ÓÚ ÖÝ µ. ØÖÙØ Ù Ú ÓÔ Ö Ø ÓÒ ØÖÙØ Ù Ó Ø Ó Ó Ø Óµ ÓÒ Ø ØÖÙØ Ù ÓÒØ ÜØ ØÜ ÓÒ Ø ØÖÙØ Ù Ó Ø Ö ØÖÙØ Ù Ú µ ÒØ Ó ÔÖÓ ÓÒ µ ÓÒ Ø ØÖÙØ Ù ÓÒØ ÜØ ØÜ ØÖÙØ Ù Ú ØÖÙØ Ù ØÖ µ ÒØ Ó Ö ÓÚ ÖÝ ÓÑÔ Ø µ ÓÒ Ø ØÖÙØ Ù ÓÒØ ÜØ ØÜ ØÖÙØ Ù Ú µ The MDD will use this to finish recovery process by doing lov synchronize and destroying the unlinked objects on OST. ÒØ Ñ Ö ÓÚ ÖÝ ÓÑÔ Ø ÓÒ Ø ØÖÙØ Ù ÓÒØ ÜØ ØÜ ØÖÙØ Ù Ú µ ØÖÙØ Ñ Ú Ñ Ù¾Ñ Ú µ ØÖÙØ Ó Ú Ó Ñ ¾ Ó Ñ µ Ö Ñ ÓÚ Ø Ò ÜØ ØÜ Ñ µ Öµ ÊÊÇÊ ± Ñ ÓÚ Ø Ò ÜØ ± Ò Ó ¹ Ó Ò Ñ Öµ ÇÌÇ ÓÙØ Öµ Ö Ñ ÒÙÔ ÙÒ Ò Ó ØÜ Ñ µ Ó ÒÓØ Ý Ó ¹ ÙºÑ ºÑ Ó Ó ÆÍÄÄ Ó ¹ Ó ÝÒ Ö ÓÚ Ç ÆÇÌÁ Ë Æ ÆÇÆ ÄÇ Ã 8

9 5.2 Handling the ØÖ Ò ÒÓ value 5 LOGIC SPECIFICATION Ç ÆÇÌÁ Ë Æ ÆÍÄÄµ Ê ÌÍÊÆ Öµ OSD will use that method to invoke orphans cleanup on bottom filesystem. ÒØ Ó Ö ÓÚ ÖÝ ÓÑÔ Ø ÓÒ Ø ØÖÙØ Ù ÓÒØ ÜØ ØÜ ØÖÙØ Ù Ú µ ØÖÙØ Ó Ú Ó Ù¾Ó Ú µ ÒØ Ö» Ö ÓÚ ÖÝ ÓÒ Ó Ö ¹ÓÔ Ò Ö ÓÒ ÓÔ Ò ÓÖÔ Ò Ö Ô ÒÒ Ò ÓÒÝ ÒÓÒ¹ÓÔ Ò ÓÖÔ Ò Û Ø» Ö ÓÖÔ Ò ÒÙÔ ºººµ Ö ØÙÖÒ Ö 5.2 Handling the ØÖ Ò ÒÓ value Ø Ø ÒØ Ñ Ø ØÜÒ ØÓÔ ÓÒ Ø ØÖÙØ Ù ÓÒØ ÜØ ØÜ ØÖÙØ Ø Ú Ú ØÖÙØ Ø Ò ØÜÒ ÚÓ ÓÓ µ ØÖÙØ Ñ Ø Ú Ñ Ø ÓÓ ØÖÙØ Ñ Ø ØÜÒ Ò Ó ØÜÒ ØÖÙØ Ñ Ø Ø Ö Ò Ó ÑØ ÒØ Ö» ØÖ Ò ÒÓ Ò ØÛÓ ÓÒØ ÜØ ¹ ÓÖ ÓÑÑ Ø Ò ÓÖ Ø Ö» ØÜÒ Ù ÓÒØ ÜØ Ý Ø ²ØÜÒ¹ Ø ØÜ ²Ñ Ø ØÜÒ Ýµ ÑØ Ù ÓÒØ ÜØ Ý Ø ØÜ ²Ñ Ø Ø Ö Ýµ Ô Ò Ó ²Ñ Ø¹ Ñ Ø ØÖ Ò ÒÓ Ó µ ÑØ ¹ ÑØ ØÖ Ò ÒÓ ¼µ ÑØ ¹ ÑØ ØÖ Ò ÒÓ Ñ Ø¹ Ñ Ø Ø ØÖ Ò ÒÓ» Ö Ô Ý» ÑØ ¹ ÑØ ØÖ Ò ÒÓ Ñ Ø¹ Ñ Ø Ø ØÖ Ò ÒÓµ Ñ Ø¹ Ñ Ø Ø ØÖ Ò ÒÓ ÑØ ¹ ÑØ ØÖ Ò ÒÓ Ô Ò ÙÒÓ ²Ñ Ø¹ Ñ Ø ØÖ Ò ÒÓ Ó µ» Ú ØÖ Ò ÒÓ ÓÖ Ø ÓÑÑ Ø» ØÜÒ ¹ ØÜ ØÖ Ò ÒÓ ÑØ ¹ ÑØ ØÖ Ò ÒÓ» ÍÔ Ø Ø ÖÚ Ö ÓÖ Û Ø Ø Ø ØÖ Ò Ø ÓÒ Ø» Ö Ñ Ø ÙÔ Ø Ø ÖÚ ÑØ Ú Ø Ò µ 9

10 5.3 Updating the Ä ËÌ Ê Î 5 LOGIC SPECIFICATION Ö ØÙÖÒ Ö» ÓÑÑ Ø Ù ØÓ ÙÔ Ø Ø ÓÑÑ Ø Ú Ù» Ø Ø ÒØ Ñ Ø ØÜÒ ÓÑÑ Ø ÓÒ Ø ØÖÙØ Ù ÓÒØ ÜØ ØÜ ØÖÙØ Ø Ú Ú ØÖÙØ Ø Ò ØÜÒ ÚÓ ÓÓ µ ØÖÙØ Ñ Ø Ú Ñ Ø ÓÓ ØÖÙØ Ó Ú Ó Ñ ¾Ù Ú ²Ñ Ø¹ Ñ Ø Ñ Úµ¹ Ó ØÖÙØ Ñ Ø ØÜÒ Ò Ó ØÜ ØÜ Ù ÓÒØ ÜØ Ý Ø ²ØÜÒ¹ Ø ØÜ ²Ñ Ø ØÜÒ Ýµ Ô Ò Ó ²Ñ Ø¹ Ñ Ø Ø ÓÑÑ ØØ Ó µ ØÜ ¹ ØÜ ØÖ Ò ÒÓ Ñ Ø¹ Ñ Ø Ø ÓÑÑ ØØ µ Ñ Ø¹ Ñ Ø Ø ÓÑÑ ØØ ØÜ ¹ ØÜ ØÖ Ò ÒÓ Ô Ò ÙÒÓ ²Ñ Ø¹ Ñ Ø Ø ÓÑÑ ØØ Ó µ ÔØÖÔ ÓÑÑ Ø Ö Ô Ó µ Ô Ò ÙÒÓ ²Ñ Ø¹ Ñ Ø Ø ÓÑÑ ØØ Ó µ Í À ± ØÖ Ò ÒÓ ÄÈ ÓÑÑ ØØ Ò Ó ¹ Ó Ò Ñ ØÜ ¹ ØÜ ØÖ Ò ÒÓµ Ö ØÙÖÒ ¼ 5.3 Updating the Ä ËÌ Ê Î ÒÙÑ Å Ì Ì Æ Ä ËÌ Ê Î Ê ÁÌË» Ö Ø ÓÖ Ø ÖÚ ÙÔ Ø» Ø Ø ÒØ Ñ Ø ØÜÒ Ø ÖØ ÓÒ Ø ØÖÙØ Ù ÓÒØ ÜØ ØÜ ØÖÙØ Ø Ú Ú ØÖÙØ ØÜÒ Ô Ö Ñ Ô Ö Ñ ÚÓ ÓÓ µ Ô Ö Ñ¹ ØÔ Ö Ø Å Ì Ì Æ Ä ËÌ Ê Î Ê ÁÌË Ö ØÙÖÒ ¼ Ø Ø ÒØ Ñ Ø Ö Ø ÖÚ ØÖÙØ Ñ Ø Ø Ö Ò Ó Ò Ó ØÖÙØ Ñ Ø ÒØ Ø Ñ Ó Ø Ó µ ØÖÙØ Ñ Ø Ú Ñ Ø Ò Ó¹ ÑØ Ñ Ø ÒØ Ö Ö Ñ Ø¹ Ñ Ø Ø ÖÚ ¹ Ó Ó Ý ÓÔ ¹ Ó Ö Ò Ó¹ ÑØ ØÜØ Ñ Ø¹ Ñ Ø Ø ÖÚ Ñ Þ Ó Ñ µ 10

11 5.3 Updating the Ä ËÌ Ê Î 5 LOGIC SPECIFICATION Ó µ Ö Þ Ó Ñ µµ Ö ¼ Ö ¼µ Ö ¹ ÍÄÌ Ö ØÙÖÒ Ö Ø Ø ÒØ Ñ Ø ÛÖ Ø Ø ÖÚ ØÖÙØ Ñ Ø Ø Ö Ò Ó Ò Ó ØÖÙØ Ñ Ø ÒØ Ø Ñ Ó Ø Ó ØÖÙØ Ø Ò Ø µ ØÖÙØ Ñ Ø Ú Ñ Ø Ò Ó¹ ÑØ Ñ Ø ÒØ Ö Ö Ñ Ø¹ Ñ Ø Ø ÖÚ ¹ Ó Ó Ý ÓÔ ¹ Ó ÛÖ Ø Ò Ó¹ ÑØ ØÜØ Ñ Ø¹ Ñ Ø Ø ÖÚ Ñ Þ Ó Ñ µ Ó Ø µ Ö Þ Ó Ñ µµ Ö ¼ Ö ¼µ Ö ¹ ÍÄÌ Ö ØÙÖÒ Ö ÒØ Ñ Ø ÙÔ Ø Ø ÖÚ ØÖÙØ Ñ Ø Ø Ö Ò Ó Ò Ó ØÖÙØ Ø Ú Ø ØÖÙØ Ø Ò Ø µ ØÖÙØ Ñ Ø Ú Ñ Ø Ò Ó¹ ÑØ Ñ Ø ØÖÙØ ÔØÖÔ Ö ÕÙ Ø Ö Õ Ñ Ø Ò Ó Ö Õ Ò Óµ ØÖÙØ Ñ Ø ÜÔÓÖØ Ø Ñ ²Ö Õ¹ ÖÕ ÜÔÓÖØ¹ ÜÔ Ñ Ø Ø ØÖÙØ Ñ Ø ÒØ Ø Ñ Ñ ¹ Ñ Ñ Ó Ø Ó ÒØ ÖÖ ¾ Ö Ø ¹ Ø Ö ÙØ ÆÌÊ» Ø ÜÔÓÖØ Ö Ý Ò Û Ú ÒÓ Ø ÖÚ ÓØ» Ö Õ¹ ÖÕ ÜÔÓÖØ¹ ÜÔ µ Ï ÊÆ ÓÑÑ Ø ØÖ Ò Ø ÓÒ ÓÖ ÓÒÒ Ø ÒØ ± Ö ± Ò Ö Õ¹ ÖÕ ÜÔÓÖØ¹ ÜÔ ÒØ ÙÙ ºÙÙ Öµ Ö ¼µ Ö ¹ ÆÇÌ ÇÆÆ Ê ÌÍÊÆ Öµ Ó Ñ ¹ Ñ Ö Ó ÓÛÒ ²Ñ Ø¹ Ñ Ø Ñ Ó µ Ñ ¹ Ñ Ø ØÖ Ò ÒÓ ÔÙ ØÓ Ò Ó¹ ÑØ ØÖ Ò ÒÓµ 11

12 5.4 Resent and reconstruction 5 LOGIC SPECIFICATION Ñ ¹ Ñ Ø Ü ÔÙ ØÓ Ö Õ¹ ÖÕ Ü µ Ñ ¹ Ñ Ø Ö ÙØ ÔÙ ØÓ ¾ Öµ Ñ ¹ Ñ Ø Ø ÔÙ ØÓ ¾ ÓÔ Ø µ Ó ¼µ ÊÊÇÊ ÒØ Ü ± Ó Ø ± Ò Ñ ¹ Ñ Ö Ü Ó µ ÖÖ ¹ ÁÆÎ Ä ÖÖ Ñ Ø ÛÖ Ø Ø ÖÚ Ò Ó Ñ ²Ñ ¹ Ñ Ö Ó Ø µ ÙÔ ²Ñ Ø¹ Ñ Ø Ñ Ó µ Ê ÌÍÊÆ ÖÖµ 5.4 Resent and reconstruction Reconstruction methods are the same as in old MDS but they are using new API Generic reconstruct Most of the operations, e.g. link(), unlink(), rename() need only generic reconstruct: ÚÓ Ñ Ø Ö ÓÒ ØÖÙØ Ò Ö ØÖÙØ Ù ÓÒØ ÜØ ØÜØ ØÖÙØ ÔØÖÔ Ö ÕÙ Ø Ö Õµ ØÖÙØ Ñ Ø ÜÔÓÖØ Ø Ñ ²Ö Õ¹ ÖÕ ÜÔÓÖØ¹ ÜÔ Ñ Ø Ø Ñ Ø Ö Õ ÖÓÑ Ñ Ö Õ Ñ ¹ Ñ Ñ µ ÚÓ Ñ Ö Õ ÖÓÑ Ñ ØÖÙØ ÔØÖÔ Ö ÕÙ Ø Ö Õ ØÖÙØ Ñ Ø ÒØ Ø Ñ µ Í Ê É À Ö Õ Ö ØÓÖ Ò ØÖ Ò ÒÓ ÄÈ» Ø ØÙ ± Ñ ¹ Ñ Ø ØÖ Ò ÒÓ Ñ ¹ Ñ Ø Ö ÙØµ Ö Õ¹ ÖÕ Ö ÔÑ ¹ ØÖ Ò ÒÓ Ö Õ¹ ÖÕ ØÖ Ò ÒÓ Ñ ¹ Ñ Ø ØÖ Ò ÒÓ Ö Õ¹ ÖÕ Ö ÔÑ ¹ Ø ØÙ Ö Õ¹ ÖÕ Ø ØÙ Ñ ¹ Ñ Ø Ö ÙØ Ñ Ø Ó Ö Õµ reconstruct_create(), reconstruct_setattr() Ø Ø ÚÓ Ö ÓÒ ØÖÙØ Ö ÒØ Ö Ø ØÖÙØ Ñ Ø Ø Ö Ò Ó Ò Óµ ØÖÙØ ÔØÖÔ Ö ÕÙ Ø Ö Õ Ñ Ø Ò Ó Ö Õ Ò Óµ ØÖÙØ Ñ Ø ÜÔÓÖØ Ø Ñ ²Ö Õ¹ ÖÕ ÜÔÓÖØ¹ ÜÔ Ñ Ø Ø 12

13 5.4 Resent and reconstruction 5 LOGIC SPECIFICATION ØÖÙØ Ñ Ø Ú Ñ Ø Ò Ó¹ ÑØ Ñ Ø ØÖÙØ Ñ Ø Ó Ø ØÖÙØ Ñ Ø Ó Ý Ó Ý Ñ Ø Ö Õ ÖÓÑ Ñ Ö Õ Ñ ¹ Ñ Ñ µ Ö Õ¹ ÖÕ Ø ØÙ µ Ö ØÙÖÒ» ÒÓ ÖÖÓÖ Ó Û Ö Ø Û Ø Ö ÕÙ Ø» Ñ Ø Ó Ø Ò Ò Ó¹ ÑØ ØÜØ Ñ Ø Ò Ó¹ ÑØ ÖÖºÖÖ ¾µ Ä ËË ÊÌ ÁË ÊÊ µµ Ó Ý Ö Õ Ô Ù ÖÚ Ö Ø ² Ò Ó¹ ÑØ Ô ²ÊÅ Å Ì Ç µ Ö ÑÓ ØØÖ Ø ØÜØ Ñ Ø Ó Ø µ ² Ò Ó¹ ÑØ ØØÖµ Ö ¹ Ê ÅÇÌ µ» Ó Ø Û Ö Ø ÓÒ Ö ÑÓØ ÖÚ Ö» Ó Ý¹ Ú Ç Å Å Ë Ñ Ø Ô ØØÖ¾ Ó Ý Ó Ý ² Ò Ó¹ ÑØ ØØÖºÑ ØØÖ Ò Ó¹ ÑØ ÖÖºÖÖ ¾µ Ñ Ø Ó Ø ÔÙØ Ò Ó¹ ÑØ ØÜØ µ Ø Ø ÚÓ Ö ÓÒ ØÖÙØ Ö ÒØ Ø ØØÖ ØÖÙØ Ñ Ø Ø Ö Ò Ó Ò Óµ ØÖÙØ ÔØÖÔ Ö ÕÙ Ø Ö Õ Ñ Ø Ò Ó Ö Õ Ò Óµ ØÖÙØ Ñ Ø ÜÔÓÖØ Ø Ñ ²Ö Õ¹ ÖÕ ÜÔÓÖØ¹ ÜÔ Ñ Ø Ø ØÖÙØ Ñ Ø Ú Ñ Ø Ò Ó¹ ÑØ Ñ Ø ØÖÙØ Ñ Ø Ó Ø Ó ØÖÙØ Ñ Ø Ó Ý Ó Ý Ñ Ö Õ ÖÓÑ Ñ Ö Õ Ñ ¹ Ñ Ñ µ Ö Õ¹ ÖÕ Ø ØÙ µ Ö ØÙÖÒ Ó Ý Ö Õ Ô Ù ÖÚ Ö Ø ² Ò Ó¹ ÑØ Ô ²ÊÅ Å Ì Ç µ Ó Ñ Ø Ó Ø Ò Ò Ó¹ ÑØ ØÜØ Ñ Ø Ò Ó¹ ÑØ ÖÖºÖÖ ½µ Ä ËË ÊÌ ÁË ÊÊ Ó µµ ÑÓ ØØÖ Ø ØÜØ Ñ Ø Ó Ø Ó µ ² Ò Ó¹ ÑØ ØØÖµ Ñ Ø Ô ØØÖ¾ Ó Ý Ó Ý ² Ò Ó¹ ÑØ ØØÖºÑ ØØÖ Ò Ó¹ ÑØ ÖÖºÖÖ ½µ» ÓÒ³Ø Ö ØÙÖÒ ÇËÌ¹ Ô ØØÖ ÙØ Û Ò³Ø Ù Ø Ø Ø Ñ» Ö ¹ ÙÖ ØØÖº Ú ² ÌÌÊ ËÁ µ Ó Ý¹ Ú Ç Å ÄËÁ Ç Å Ä ÄÇ ÃË Ö ¹ ÙÖ ØØÖº Ú ² ÌÌÊ ÅÌÁÅ ÌÌÊ ÅÌÁÅ Ë Ìµµ Ó Ý¹ Ú Ç Å ÄÅÌÁÅ Ö ¹ ÙÖ ØØÖº Ú ² ÌÌÊ ÌÁÅ ÌÌÊ ÌÁÅ Ë Ìµµ Ó Ý¹ Ú Ç Å Ä ÌÁÅ Ñ Ø Ó Ø ÔÙØ Ò Ó¹ ÑØ ØÜØ Ó µ 13

14 6 STATE MANAGEMENT 6 State management 6.1 State invariants FID is invariant so recovery become simpler because uses the same FID as ordinary operations did. 14

Version-Based Recovery DLD

Version-Based Recovery DLD Mike Pershin 2008-04-15 1 Requirements 1. a server only replays a client update request when the update affects exactly the same versions of objects as it did in the original