tfgrid/info_tfgrid_manual
28
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

...

Browse Source
This commit is contained in:
despiegk
2024-02-23 06:07:41 +03:00
commit a83f4cd5e8
1963 changed files with 56869 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
tosort/grid_deployment/grid_deployment.md Normal file
View File

@@ -0,0 +1,13 @@
# Grid Deployment
The TFGrid whole source code is open-source and instances of the grid can be deployed by anyone thanks to the distribution of daily grid snapshots of the complete ThreeFold Grid stacks.
> Note: This currently works for devnet only
## Table of Contents
- [TFGrid Stacks](./tfgrid_stacks.md)
- [Full VM Grid Deployment](./grid_deployment_full_vm.md)
- [Guardians of the Grid](./tfgrid_guardians.md)
> [Check the TFGrid Backup](https://dashboard.bknd1.ninja.tf/)

137
tosort/grid_deployment/grid_deployment_full_vm.md Normal file
View File

@@ -0,0 +1,137 @@
<h1> Grid Deployment on a Full VM </h1>
<h2>Table of Contents</h2>
- [Introduction](#introduction)
- [Prerequisites](#prerequisites)
- [DNS Settings](#dns-settings)
- [DNS Verification](#dns-verification)
- [Prepare the VM](#prepare-the-vm)
- [Launch the Script](#launch-the-script)
- [Access the Grid Services](#access-the-grid-services)
- [Manual Commands](#manual-commands)
- [Update the Deployment](#update-the-deployment)
***
## Introduction
We present the steps to deploy a development network (devnet) instance of the TFGrid on a full VM.
## Prerequisites
For this guide, you will need to deploy a full VM on the ThreeFold Grid with at least the following minimum specs:
- IPv4
- IPv6
- 32GB of RAM
- 1000 GB of SSD
- 8 vcores
After deploying the full VM, take note of the IPv4 and IPv6 addresses to properly set the DNS records and then SSH into the VM.
## DNS Settings
You need to set an A record for the IPv4 address and an AAAA record for the IPv6 address with a wildcard subdomain.
The following table explicitly shows how to set the A and AAAA records for your domain.
| Type | Host | Value |
| ---- | ---- | -------------- |
| A | \* | <ipv4_address> |
| AAAA | \* | <ipv6_address> |
### DNS Verification
You can use tools such as [DNSChecker](https://dnschecker.org/) or [dig](https://linux.die.net/man/1/dig) on a terminal to check if the DNS propagadation is complete.
## Prepare the VM
- Download the ThreeFold Tech `grid_deployment` repository
```
git clone https://github.com/threefoldtech/grid_deployment
cd grid_deployment/docker-compose/devnet
```
- Generate a TFChain node key with `subkey`
```
echo .subkey_devnet >> .gitignore
../subkey generate-node-key > .nodekey_devnet
cat .nodekey_devnet
```
- Create and the set environment variables file
```
cp .secrets.env-example .secrets.env
```
- Adjust the environment file
```
nano .secrets.env
```
- To adjust the `.secrets.env` file, take into account the following:
- **DOMAIN**
- Write your own domain (e.g. example.com)
- **TFCHAIN_NODE_KEY**
- Write the output of the command `cat .nodekey_devnet`
- **ACTIVATION_SERVICE_MNEMONIC**
- Write the seed phrase of an account on devnet with at least 10 TFT in the wallet
- **GRID_PROXY_MNEMONIC**
- Write the seed phrase of an account on devenet with at least 10 TFT in the wallet and a registered twin ID\*
> \*Note: If you've created an account using the ThreeFold Dashboard on devnet, the twin ID is automatically registered.
## Launch the Script
Once you've prepared the VM, you can simply run the script to install the grid stack and deploy it online.
```
sh install_grid_bknd.sh
```
This will take a few minutes since you are downloading the whole devnet grid snapshots.
## Access the Grid Services
Once you've deployed the grid stack online, you can access the different grid services by usual the usual subdomains:
```
dashboard.your.domain
metrics.your.domain
tfchain.your.domain
graphql.your.domain
relay.your.domain
gridproxy.your.domain
activation.your.domain
stats.your.domain
```
## Manual Commands
Once you've run the install script, you can deploy manually the grid stack with the following command:
```
docker compose --env-file .secrets.env --env-file .env up -d
```
You can also check if the environment variables are properly set:
```
docker compose --env-file .secrets.env --env-file .env config
```
If you want to see the output during deployment, remove `-d` in the command above as follows:
```
docker compose --env-file .secrets.env --env-file .env up
```
This can be helpful to troubleshoot errors.
## Update the Deployment
Go into the correct folder for the network your deploying for, e.g. devnet, and run the following commands
```
git pull -r
docker compose --env-file .secrets.env --env-file .env up -d
```
If you're interested about hosting your own instance of the grid to strenghten the ThreeFold ecosystem, make sure to read the next section, [Guardians of the Grid](./tfgrid_guardians.md).

206
tosort/grid_deployment/snapshots.md Normal file
View File

@@ -0,0 +1,206 @@
<h1>Snapshots for Grid Backend Services</h1>
<h2>Table of Contents</h2>
- [Introduction](#introduction)
- [Prerequisites](#prerequisites)
- [Services](#services)
- [ThreeFold Public Snapshots](#threefold-public-snapshots)
- [Deploy the Services with Scripts](#deploy-the-services-with-scripts)
- [Start All the Services](#start-all-the-services)
- [Stop All the Services](#stop-all-the-services)
- [Create the Snapshots](#create-the-snapshots)
- [Expose the Snapshots with Rsync](#expose-the-snapshots-with-rsync)
- [Create the Service Configuration File](#create-the-service-configuration-file)
- [Start the Service](#start-the-service)
***
## Introduction
To facilitate deploying grid backend services, we provide snapshots to significantly reduce sync time. This can be setup anywhere from scratch. Once all services are synced, one can use the scripts to create snapshots automatically.
## Prerequisites
There are a few prerequisites to properly run the ThreeFold services.
- [Docker engine](../computer_it_basics/docker_basics.md#install-docker-desktop-and-docker-engine)
- [Rsync](../computer_it_basics/file_transfer.md#rsync)
## Services
There are 3 grid backend services that collect enough data to justify creating snapshots:
- ThreeFold blockchain - TFChain
- Graphql - Indexer
- Graphql - Processor
## ThreeFold Public Snapshots
ThreeFold hosts all available snapshots at: [https://bknd.snapshot.grid.tf/](https://bknd.snapshot.grid.tf/). Those snapshots can be downloaded with rsync:
- Mainnet:
```
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshots/tfchain-mainnet-latest.tar.gz .
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshots/indexer-mainnet-latest.tar.gz .
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshots/processor-mainnet-latest.tar.gz .
```
- Testnet:
```
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshotstest/tfchain-testnet-latest.tar.gz .
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshotstest/indexer-testnet-latest.tar.gz .
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshotstest/processor-testnet-latest.tar.gz .
```
- Devnet:
```
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshotsdev/tfchain-devnet-latest.tar.gz .
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshotsdev/indexer-devnet-latest.tar.gz .
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshotsdev/processor-devnet-latest.tar.gz .
```
Let's now see how to use those snapshots to run the services via scripts.
## Deploy the Services with Scripts
You can deploy the 3 individual services using known methods such as [Docker](https://manual.grid.tf/computer_it_basics/docker_basics.html). To facilitate the process, scripts are provided that run the necessary docker commands.
The first script creates the snapshots, while the second and third scripts serve to start and stop all services.
You can use the start script to start all services and then set a cron job to execute periodically the snapshot creation script. This will ensure that you always have the latest version available on your server.
### Start All the Services
You can start all services by running the provided scripts.
- Download the script.
- Main net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/mainnet/startall.sh
```
- Test net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/testnet/startall.sh
```
- Dev net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/devnet/startall.sh
```
- Set the permissions of the script
```
chmod +x startall.sh
```
- Run the script to start all services via docker engine.
```
./startall.sh
```
### Stop All the Services
You can stop all services by running the provided scripts.
- Download the script.
- Main net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/mainnet/stopall.sh
```
- Test net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/testnet/stopall.sh
```
- Dev net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/devnet/stopall.sh
```
- Set the permissions of the script
```
chmod +x stopall.sh
```
- Run the script to stop all services via docker engine.
```
./stopall.sh
```
### Create the Snapshots
You can set a cron job to execute a script running rsync to create the snapshots and generate logs at a given interval.
- First download the script.
- Main net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/mainnet/create_snapshot.sh
```
- Test net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/testnet/create_snapshot.sh
```
- Dev net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/devnet/create_snapshot.sh
```
- Set the permissions of the script
```
chmod +x create_snapshot.sh
```
- Make sure to a adjust the snapshot creation script for your specific deployment
- Set a cron job
```
crontab -e
```
- Here is an example of a cron job where we execute the script every day at 1 AM and send the logs to `/var/log/snapshots/snapshots-cron.log`.
```sh
0 1 * * * sh /opt/snapshots/create-snapshot.sh > /var/log/snapshots/snapshots-cron.log 2>&1
```
## Expose the Snapshots with Rsync
We use rsync with a systemd service to expose the snapshots to the community.
### Create the Service Configuration File
To setup a public rsync server, create and edit the following file:
`/etc/rsyncd.conf`
```sh
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsync.log
port = 34873
max connections = 20
exclude = lost+found/
transfer logging = yes
use chroot = yes
reverse lookup = no
[gridsnapshots]
path = /storage/rsync-public/mainnet
comment = THREEFOLD GRID MAINNET SNAPSHOTS
read only = true
timeout = 300
list = false
[gridsnapshotstest]
path = /storage/rsync-public/testnet
comment = THREEFOLD GRID TESTNET SNAPSHOTS
read only = true
timeout = 300
list = false
[gridsnapshotsdev]
path = /storage/rsync-public/devnet
comment = THREEFOLD GRID DEVNET SNAPSHOTS
read only = true
timeout = 300
list = false
```
### Start the Service
Start and enable via systemd:
```sh
systemctl start rsync
systemctl enable rsync
systemctl status rsync
```
If you're interested about hosting your own instance of the grid to strenghten the ThreeFold ecosystem, make sure to read the next section, [Guardians of the Grid](./tfgrid_guardians.md).

22
tosort/grid_deployment/tfgrid_guardians.md Normal file
View File

@@ -0,0 +1,22 @@
<h1> Guardians of the Grid </h1>
<h2>Table of Contents</h2>
- [Introduction](#introduction)
- [Become a Guardian](#become-a-guardian)
***
## Introduction
[Backups of the TFGrid](./snapshots.md) are available on websites such as [ThreeFold Ninja backup](https://dashboard.bknd1.ninja.tf/) and anyone can host an instance of the ThreeFold Grid. Concretely, this means that you can participate in the ThreeFold ecosystem by hosting your own instance of the complete set of all ThreeFold services on your own domain!
Trusted guardians of the ThreeFold Grid can then be listed in the ThreeFold Manual, thus providing to users around the world different instances of the ThreeFold Grid in multiple locations, ensuring a resilient system that is highly decentralized and truly open-source.
## Become a Guardian
You can become a guardian of the ThreeFold Grid by deploying your own instance of the full ThreeFold stack (main, test and dev networks) on your own domain.
Information concerning trusted guardians of the ThreeFold Grid can then be displayed in the ThreeFold ecosystem, e.g. TF Manual, TF Forum, and more. Then, grid users can use any of the different grid instances to interact with the TFGrid. This cooperative system allows for a strong grid ecosystem that is resilient and decentralized by design.
Join the discussions on the [ThreeFold Forum](https://forum.threefold.io/).

39
tosort/grid_deployment/tfgrid_stacks.md Normal file
View File

@@ -0,0 +1,39 @@
<h1> TFGrid Stacks </h1>
<h2>Table of Contents</h2>
- [Introduction](#introduction)
- [Concrete Example](#concrete-example)
- [Advantages](#advantages)
- [Run Your Own Stack](#run-your-own-stack)
***
## Introduction
ThreeFold is an open-source project and anyone can run the full stack of the TFGrid in a totally decentralized manner. In practice, this means that anyone can grab a docker compose file shared by ThreeFold of the TFGrid stacks and run an instance of the grid services on their own domain.
This means that you could host your own instance of the ThreeFold Dashboard at `dashboard.yourdomain.com` that would serve your own instance of the complete TFGrid stack! Users could then access the ThreeFold Dashboard via your own domain.
The process is actually very straightforward and we even provide a script to streamline the process.
> Note: This currently works for devnet only. Soon it will also be possible to deploy stacks of the test and main networks. Stay tuned.
## Concrete Example
For example, there is a TFGrid backend at this URL: [https://dashboard.bknd1.ninja.tf/](https://dashboard.bknd1.ninja.tf/).
## Advantages
Setting such instances of the TFGrid ensures resiliency and decentralization of the ThreeFold ecosystem.
As a very concrete example, image that one instance of the Dashboard goes offline, `dashboard.grid.tf`, then users could still access the Dashboard from another instance. The more users of the TFGrid deploy their own instance, the more resilient the grid becomes.
The overall ThreeFold ecosystem becomes more resilient to failures of individual nodes.
## Run Your Own Stack
To set your own instance of the TFGrid, you can download a snapshot of the grid and deploy the TFGrid services with Docker. We even provide scripts to quicken the whole process!
Read more about snapshots in the [next section](./grid_deployment_full_vm.md).