TwoCommit

Problem

There can occur situations, when during connection commit blobbers will have different states. Less than enough to recover data blobbers can receive write marker and commit connection S1, the rest of bobbers can stay with previous uncommitted state S0 (broken). If a client will decide to repair such allocation, there won't be enough bobbers to restore committed state and there is no way to rollback, since blobbers have already merged changes.

It can happen, when blobbers will crash during connection commit, or client will send commit only partially to blobbers.

Solution

To provide a way to get back to previous correct state (S0), even if it is committed on part of the network, we introduce double commit approach. We add new state of the filesystem "pre-committed" S1 (in yellow)

Implementation

We should assume that client-blobber interactions (data upload, connection_commit) are decoupled from blobber-blockchain-validator interactions (write_marker submit, challenge complete). It is achieved mainly by using chains of writemarkers during completing challenges, which in order allows blobbers not to store all the versions of allocations, but use the latest one and do not store any earlier versions.

Direct flow

• user uploads 2 files, f1 and f2
• user commits them with writemarker W1
• files are moved to precommit directory with version 1
• user uploads f3 and deletes f2
• user commits with writemarker W2
• f1 and f2 are moved to filestore with version 1
• f3 and delete of f2 (REF table record) are moved to the precommit dir with version 2
• user uploads f4
• user commits with W3
• f2 is deleted in filestore, f3 is added changing fs version to 2
• f4 is moved to precommit changing the version to 3

Rollback flow

• user uploads 2 files, f1 and f2
• user commits them with writemarker W1
• files are moved to precommit directory with version 1
• user uploads f3 and deletes f2
• user commits with writemarker W2
• f1 and f2 are moved to filestore with version 1
• f3 and delete of f2 (REF table record) are moved to the precommit dir with version 2
• user uploads f4
• user rollbacks with empty writemarker W2e
• precommit dir is discarded as well as temp dir and allocation gets back to version 1
• f3 is aploaded and committed with writemarker W3

Allocation versioning

We need to introduce a common version of allocation to be able to sync the state of each blobber. We will use writemarker.timestamp for that purpose. User should issue writemarkers all with exactly the same timestamp when committing the write, the version of allocation is the timestamp of corresponding writemarker to precommit directory.

##Empty writemarker

type WriteMarker struct {
	AllocationRoot         string           `json:"allocation_root"` // the root of previous allocation
	PreviousAllocationRoot string           `json:"prev_allocation_root"` // the root of the same previous allocation
	FileMetaRoot           string           `json:"file_meta_root"` // the same as previous
	AllocationID           string           `json:"allocation_id"` 
	Size                   int64            `json:"size"` //is zero
	BlobberID              string           `json:"blobber_id"`
	Timestamp              common.Timestamp `json:"timestamp"` //the same as timestamp of reverted writemarker
	ClientID               string           `json:"client_id"`
	Signature              string           `json:"signature"`
}

Empty writemarker is used to rollback the writemarker of current precommit directory and discard those changes.

The timestamp of empty writemarker should be exactly the same as the version of precommit directory user wants to rollback. After the rollback version of commit directory is used (empty writemarker is skipped)

Empty writemarker is committed to blockchain as usual writemarker.

Challenge protocol

Writemarker is used as a proof of rollback, so instead of writemarker chains validator and blobber should operate on wrietmarker trees, pretty simple though. The only situation when trees will be used is when rolled back allocation root is challenged, in that case validator provides following proof:

This tree shows to validator that blobber doesn't have allocation of version 1 locally, also it doesn't have any direct siblings of it, because it was rolled back, instead it has allocation version of n that was built on common ancestor root 0.

Pre-committed section

At the moment changes are stored to getAllocTempDir (tmp) directory. We will add new directory getAllocPreCommitDir (precommit). After changes are pre-committed they are moved to this directory from getAllocTempDir. It will allow us to store S1 (pic) in the different isolated sandbox.

Pre-commit event

It is the main part of double-commit protocol, we pre-commit on CommitWrite, but instead of connectionObj.ApplyChanges we move changes to getAllocPreCommitDir dir.

CommitWrite

When client commits write blobber does several actions:

• commit previously pre-committed data with connectionObj.ApplyChanges and moving them from getAllocPreCommitDir to filesystem as before on commit
• pre-commit current data that is speculative, move it to getAllocPreCommitDir from getAllocTempDir

##Rollback We add Rollback, to be able to rollback pre-committed state S1(pic) to committed S0(pic) state. We do not delete pre-committed changes though, since these changes could be already committed to blockchain and challenged, instead we store them locally for some time, but remove from getAllocPreCommitDir to some other temp dir. These changes won't be used anyhow in the future, only to complete challenges.

Data manipulation

/v1/file and /v1/dir uses pre-committed state to manipulate it, so instead of merged state before, we will use files from connectionObj.ApplyChanges section to build state upon