tfgrid/info_tfgrid
30
3
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Activity
1a5e76c82d
info_tfgrid/collections/tech/qsss/qss_filesystem.md

3.2 KiB
Raw Blame History

Quantum Safe Filesystem

A redundant filesystem, can store PB's (millions of gigabytes) of information.

Unique features:

  • Unlimited scalability (many petabytes)
  • Quantum Safe:
    • No farmer knows what the data is
    • Even a quantum computer cannot decrypt the data
  • Data can't be lost
    • Protection for datarot, data will autorepair
  • Data is kept forever (data does not get deleted)
  • Data is dispersed over multiple sites
    • Even if the sites go down the data will not be lost
  • Up to 10x more efficient than storing on classic storage cloud systems
  • Can be mounted as filesystem on any OS or any deployment system (OSX, Linux, Windows, Docker, Kubernetes etc.)
  • Compatible with ± all data workloads (not high performance data driven workloads like a database)
  • Self-healing: when a node or disk is lost, the storage system can get back to the original redundancy level
  • Helps with compliance for regulations like GDPR (as the hosting facility has no view on what is stored: information is encrypted and incomplete)
  • Hybrid: can be installed onsite, public and private
  • Read-write caching on encoding node (the front end)

Mount Any Files In Your Storage Infrastructure

The QSFS is a mechanism to mount any file system (in any format) on the grid, in a quantum secure way.

This storage layer relies on 3 primitives:

  • 0-db is the storage engine. It is an always append database, which stores objects in an immutable format. It allows history to be kept out-of-the-box, good performance on disk, low overhead, easy data structure and easy backup (linear copy and immutable files).

  • 0-stor-v2 is used to disperse the data into chunks by performing 'forward-looking error-correcting code' (FLECC) on it and send the fragments to safe locations. It takes files in any format as input, encrypts the file with AES based on a user-defined key, then FLECC-encodes the file and spreads out the result to multiple 0-DBs. The number of generated chunks is configurable to make it more or less robust against data loss through unavailable fragments. Even if some 0-DBs are unreachable, you can still retrieve the original data, and missing 0-DBs can even be rebuilt to have full consistency. It is an essential element of the operational backup.

  • 0-db-fs is the filesystem driver which uses 0-DB as a primary storage engine. It manages the storage of directories and metadata in a dedicated namespace and file payloads in another dedicated namespace.

Together they form a storage layer that is quantum secure: even the most powerful computer can't hack the system because no single node contains all of the information needed to reconstruct the data.

This concept scales forever, and you can bring any file system on top of it:

  • S3 storage
  • any backup system
  • an ftp-server
  • IPFS and Hypercore distributed file sharing protocols

Architecture

By using our filesystem inside a Virtual Machine or Kubernetes, the cloud user can deploy any storage application on top e.g. Minio for S3 storage, OwnCloud as online fileserver.

Any storage workload can be deployed on top of the zstor.