VUFS specification
Index of specification:
Block it is elementary unity in ariADDna storage. Block contain local file content of 0-5МB parts.
On cloud resources blocks save on path: <root of storage>/ariaddna-storage/<file uuid>/<block uuid>.dat
Block has next fields:
- version - type: string
- block uuid - type: string
- block number - type: long
- file uuid - type: string
- payload - type: byte[]
- timestamp - type: long
- size - type: long
version - version for format of block, template: <release version>.<major version>.<minor version>:
- release version - increment when build new codebase with large changes (it changes can crash backward compatibility) examples: change format, deprecated features.
- major version - increment when build new codebase with large changes (it changes can't crash backward compatibility) example: bugfix.
- minor version - increment when build new codebase with large changes (it changes can crash backward compatibility) examples: refactoring, add features in release.
It's version equals in block, metafile, metatable and VUFS specification (Current version is 0.0.6).
block uuid - UUID ID of block (the block uuid is part of blocks name, duplicate in this field for protect to rename file on cloud) when we detect rename we run grab files in directory process and restore correct name of file. UUID it's: hash(block payload + file uuid), VUFS work as CopyOnWrite filesystem, then all blocks will be immutable. CoW helping make fast snapshot. Blocks can't delete from cloud resource if this block include to any snapshot. Also this hash (contain in UUID) use for error control in payload. But restore of payload work over resource allocation strategy on cloud resources.
For calculate hash block need use fast algorithm. Variant hash algorithm: (need investigation what more performance and collision effect)
- xxHash (https://cyan4973.github.io/xxHash/) (have info about mass collision)
- Murmur3F (https://github.com/greenrobot/essentials/blob/master/java-essentials/src/main/java/org/greenrobot/essentials/hash/Murmur3F.java) https://ipfs.io/ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1mXWo6uco/wiki/MurmurHash.html
file uuid - uuid of file in VUFS (it's uuid equals name directory where save block files on cloud resources) include of this block, use for restore the name of file directory (case rename directory contain blocks files on cloud resource). If we detect rename directory we run grab files in directory (if structure metafile in metatable is corrupt) and set as name directory most common value.
block number - sequence number block of file, the first number is 0.
payload - contain local file content of 0-5МB parts.
timestamp - timestamp write/modify block on VUFS. size - size of filed "payload".
Block save on disk or cloud use serialization for class Block. Variant serialization algorithm: (need investigation what more performance)
- FAST: https://github.com/RuedigerMoeller/fast-serialization
- Krio: https://blog.hazelcast.com/kryo-serializer/, https://github.com/EsotericSoftware/kryo#quickstart.
Metafile it's one of many vertex acyclic graph and on local filesystem metafile equals local files and directory.
Metafile has next fields:
- version - type: string
- file uuid - type: string
- parent file uuid - type: string
- child files uuid [] - type: set
- blocks uuid [] - type: list
- acl - (only in version more 1.1.0)
- properties - type: map<string,string>
- allocate strategy - type: enum
- blocks allocate [] - type: map<block uuid, cloud uuid>
version - version for format of Metafile, template: <release version>.<major version>.<minor version>.
file uuid - global UUID of file on VUFS, the ariADDna storage network can't has two metafile with one UUID.
parent file uuid - UUID of parent metafile (parent directory).
child files uuid [] - set of files UUID contain in this metafile (if this metafile is directory).
blocks uuid [] - ordered array of blocks UUI, order block always equals block number in the local file.
acl - Access Control List (not implemented in version 1.0.0 and make by default owner file as current user for process).
properties - standard properties of file/directory (creation time, modification time, ...), implemented as map<string,string>. So in this map contain info about of file's type on local storage: key="typeOfFs" value= from enum ("FILE_TXT" for text file, "FILE_BIN" for binary file, "FILE_CRYPT" for encrypt file, "DIR" for directory)
allocate strategy - block allocation strategy on cloud resources inherited for parent metafile. User can set block allocation strategy only for metafile (directory) location on root VUFS.
blocks allocate [] - allocation blocks map on cloud resource (implemented as ConcurrentHashMap<block uuid, cloud uuid>)
Metatable it's kernel structure of VUFS and implementing as acyclic graph. VUFS can have many metatables, but only one of them can be master table. The max count of metatable need set in system configuration parameter. Metatable implementing as Copy-On-Write structure.
Metatable has next fileds:
- version - type: string
- type - type: enum
- metafiles metafile [] - type:
set<metafile> - last update timestamp - type: long
version - version for format of Metatable, template: <release version>.<major version>.<minor version>.
type - flag of type metatable, can has value: master or snapshot.
metafiles metafile [] - not ordered set of metafile, it's Copy-On-Write structure (implementing on CopyOnWriteArraySet<E> from JDK).
last update timestamp - last timestamp update metatable.
Snapshot it's metatable too, but has in type field value equals "snapshot". Because metatable use CoW structure for make snaphot you can get iterator and safe it as separate map in snapshot metatable.
For snapshot field last update timestamp has value timestamp when snapshot has be created. All metatable with type snapshot should be use only read-only mode.
All metatable in periodic run synchronization on all client devises and ariADDna server. For it mechanism use for make restore when metatable is corrupt.
In this case metafile don't include to any snapshot.
In this case metafile include in one or more snapshot.
It's simple method use overwrite all blocks incude in metafile.
It's method upload only chages blocks. In this case new block equals old block exist in metatable and not include to any snapshot.
It's method upload only chages blocks. In this case new block not equals old block exist in metatable and old block include to some snapshot.
It's method upload only chages blocks. In this case new block not equals old block exist in metatable and old block not include to any snapshot.
In this case block include to mastertable or any other snapshot.
In this case block not include to mastertable or any other snapshot.
Goal: stores for set of metatables: master and snapshots. And management current state of filesystem.
Service has public methods:
- create new empty metafile in current master metatable
- get exists metafile from current master metatable
- add new block by uuid to metafile
- delete exists block by uuid from metafile
- modification properties exists metafile
- get properties exists metafile
- modification strategy exists metafile
- get strategy exists metafile
- add exists metafile to metatable
- delete exists metafile from metatable
- make snapshot of master metatable
- set metatable as current master (case first init metatable or rollback on snapshot)
- get exists snapshots
- get current master metatable
- get allocation for blocks by uuid
- set allocation for blocks by uuid
- is block by uuid include to snapshots
- get count snapshots include block by uuid
- is block by uuid include to master metatable
- delete snapshot
- add metafile as child to parent metafile
- delete child metafile from parent metafile
Note: one block can include to one metafile.
Goal: run and management synchronization process
Service has public methods:
- handle change on local filesystem
- pre-commit new block location to metafile (temporary save set of location for some blocks)
private methods:
- parse changes (return 3 collection added, modified and deleted local files)
- get next local file from changes
- is local file has flag directory (in Java directory it is type File too)
- get child local file for directory from change
Note: example algorithm add new file to VUFS: Called from Local Service when local service found changes on VUFS Get changes from Local Service. Add/Update/Remove metafile for each local file in got changes. Get first block for each local file (run Block Manipulation service) Run Cloud Transfer service for move block on clouds. Repeat for next block in file (see sequence diagram "create new file")
Goal: Management blocks from convert local file to VUFS and vise versa.
Service has public methods:
- get next block from local file
- concatenation set of blocks to one local file
- get next block uuid from exists metafile
- get block uuid by block number from exists metafile
- is block exists on current master metatable (calculate hash from block and equals with hash filed in exists in metatable blocks)
private methods:
- create next block from local file (block can from 0 to max size is set in configuration, now it is 5MB. local size write in filed "size" of Block structures.)
Goal: Management resource policy of use cloud resource. This service store setting of max usage and current usage of cloud resource.
Service has public methods:
- get max limit of use cloud resource
- set max limit of use cloud resource (after set limit need call AllocateService.handleChangeLimits)
- get free resource (return collection available free space of each clouds)
- update state of use cloud resource
Goal: Resource allocation strategy management (use data from resource policy service and resource allocation strategy from VUFS), calculating location for next block (where is will be upload next block).
Service should has public methods:
- get location for new block
- get location for exists block by uuid
- commit location for block by uuid
- calculating location
private methods:
- calculation location (base on resource strategy use and available space on clouds)
Note: Called before Cloud Transfer service upload each next block Call Create VUFS service for write block location in blocks allocate map.
Goal: Collected statistic and export updates to other services.
Service should has public methods:
- update statistics (call services where need update some counters)
private methods:
- collected statistics on store
Note: Called after Cloud Transfer service upload/download blocks and generate statistic. Get statistic for client and call Resource Policy service for update state of use cloud resource
Goal: Upload/download blocks and generate statistic about use cloud resource and network quality.
Service should has public methods:
- upload block on cloud location
- download block by uuid from cloud location
- delete block by uuid on cloud location
private methods:
- make folder for metafile on cloud location
- delete folder for metafile on cloud location
- get operation statistics
Note: Call Allocate service for get next target cloud resource.
In the cloud for store blocks use path: <root of storage>/ariaddna-storage/<file uuid>/<block uuid>.dat
For example:
if VUFS contain:
/
|
- my_file.txt (file uuid: 0111-093j-j198-1zn9)
|
- block-1 (block uuid: 8937-24jh-24jq-23we)
|
- ...
|
- block-n (block uuid: 8345-ad6h-56hj-2erg)
then GoogleDrive contain:
/
|
- ariaddna-storage/
|
- 0111-093j-j198-1zn9/
|
- 8937-24jh-24jq-23we.dat
|
...
|
- 8345-ad6h-56hj-2erg.dat
Goal: Management changes on local store and mapping local path to VUFS path
Service should has public methods:
- set mapping local path to VUFS path
- get VUFS path for local path
- get local path for VUFS path
- handle changes on FS