tfgrid/projectmycelium
11
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

init projectmycelium

Browse Source
This commit is contained in:
mik-tf
2025-09-01 21:37:01 -04:00
commit b41efb0e99
319 changed files with 128160 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
docs/ops/method/current/README.md Normal file
View File

@@ -0,0 +1,17 @@
# Current Method for Ops Running the Project Mycelium
## Introduction
This short index is for operators/SREs who deploy and run the marketplace.
## Quick links
- [Deployment Method](./deployment_method.md)
- [Architecture & Roadmap](../../../dev/design/current/projectmycelium-roadmap.md)
- [Dev design method](../../../dev/method/current/design_method.md)
- [DevOps overview](../../../devops.md)
## What you'll need
- Access to dev/prod environments (hosts or cluster)
- Required secrets/config (Stripe, PostgREST, DB)

230
docs/ops/method/current/automated-deployment.md Normal file
View File

@@ -0,0 +1,230 @@
# Automated Deployment Guide
This guide explains how to use the automated deployment system for the Project Mycelium.
## Prerequisites
- SSH access to `root@info.ourworld.tf` (passwordless SSH key setup recommended)
- Git repository with `main` and `development` branches
- Gitea credentials for deployment (username and personal access token)
- `.env.deploy` file configured with your Gitea credentials
## Quick Start
1. Copy the deployment environment template:
```bash
cp .env.deploy.example .env.deploy
```
2. Edit `.env.deploy` with your Gitea credentials:
```bash
GITEA_USER=your_username
GITEA_TOKEN=your_personal_access_token
```
3. Setup development environment:
```bash
make setup-dev
```
## Available Commands
### First-Time Setup Commands
#### Setup Development Environment
```bash
make setup-dev
```
This will:
1. Validate `.env.deploy` file exists and has proper credentials
2. Copy development deployment script to server
3. Copy development service configuration
4. Copy deployment credentials to server
5. Setup monitoring for the development service
6. Start the `tf-marketplace-dev` service
#### Setup Production Environment
```bash
make setup-prod
```
This will:
1. Validate `.env.deploy` file exists and has proper credentials
2. Copy production deployment script to server
3. Copy production service configuration
4. Copy deployment credentials to server
5. Setup monitoring for the production service
6. Start the `tf-marketplace-prod` service
7. Restart Caddy to load configurations
#### Setup Both Environments
```bash
make setup
```
Sets up both development and production environments in sequence.
### Regular Update Commands
#### Update Development Environment
```bash
make update-dev
```
This will:
1. SSH to the server
2. Navigate to development repository path
3. Checkout the `development` branch
4. Pull latest changes
5. Restart the `tf-marketplace-dev` service
#### Update Production Environment
```bash
make update-prod
```
This will:
1. SSH to the server
2. Navigate to production repository path
3. Checkout the `main` branch
4. Pull latest changes
5. Restart the `tf-marketplace-prod` service
### Monitoring Commands
#### Check Service Status
```bash
make status
```
Shows the status of both marketplace services.
#### View Service Logs
```bash
make logs
```
Shows logs from both development and production services.
### Local Development Commands
#### Build and Run Locally
```bash
make build
```
Builds and runs the application locally (checks/generates `.env` file automatically).
#### Generate Secret Key
```bash
make key
```
Generates a new `SECRET_KEY` in the `.env` file.
#### Get Help
```bash
make help
```
Shows all available commands with descriptions.
## Deployment Workflow
### Initial Setup Workflow
1. Copy `.env.deploy.example` to `.env.deploy`
2. Edit `.env.deploy` with your Gitea credentials
3. Run `make setup-dev` for development environment
4. Run `make setup-prod` for production environment (or `make setup` for both)
### Development Workflow
1. Make changes in your local development branch
2. Push changes to `development` branch in gitea
3. Run `make update-dev` to deploy to development environment
### Production Workflow
1. Create PR from `development` to `main` in gitea
2. Review and merge the PR
3. Run `make update-prod` to deploy to production environment
## Server Configuration
The deployment assumes the following server setup:
- **Development Repository Path**: `/root/code/git.ourworld.tf/tfgrid_research/dev/projectmycelium`
- **Production Repository Path**: `/root/code/git.ourworld.tf/tfgrid_research/prod/projectmycelium`
- **Development Service**: `tf-marketplace-dev`
- **Production Service**: `tf-marketplace-prod`
- **Zinit Configuration**: Service configs stored in `/etc/zinit/`
- **Deployment Scripts**: Stored in `/etc/zinit/cmds/`
## Troubleshooting
### SSH Connection Issues
Ensure you have SSH key access to `root@info.ourworld.tf`:
```bash
ssh root@info.ourworld.tf 'echo "Connection successful"'
```
### Service Not Starting
Check service logs:
```bash
make logs
```
Or check directly on the server:
```bash
ssh root@info.ourworld.tf 'zinit log tf-marketplace-prod'
ssh root@info.ourworld.tf 'zinit log tf-marketplace-dev'
```
### Missing .env.deploy File
If you see errors about missing `.env.deploy` file:
1. Copy the template: `cp .env.deploy.example .env.deploy`
2. Edit the file with your Gitea credentials
3. Ensure `GITEA_USER` and `GITEA_TOKEN` are properly set
### Invalid Gitea Credentials
If setup fails due to invalid credentials:
1. Verify your Gitea username and personal access token
2. Ensure the token has appropriate repository access permissions
3. Update `.env.deploy` with correct credentials
### Service Status Issues
Check service status:
```bash
make status
```
This will show if services are running, stopped, or have errors.
## Manual Deployment
If you need to deploy manually, the commands are:
### Development
```bash
ssh root@info.ourworld.tf 'cd /root/code/git.ourworld.tf/tfgrid_research/dev/projectmycelium && git checkout development && git pull && zinit restart tf-marketplace-dev'
```
### Production
```bash
ssh root@info.ourworld.tf 'cd /root/code/git.ourworld.tf/tfgrid_research/prod/projectmycelium && git checkout main && git pull && zinit restart tf-marketplace-prod'
```
## Environment Files
### .env.deploy
Required for deployment setup. Contains Gitea credentials:
```bash
GITEA_USER=your_username
GITEA_TOKEN=your_personal_access_token
```
### .env (Local Development)
Generated automatically by `make build` or `make key`. Contains:
- `SECRET_KEY` - Auto-generated secure key
- Other application configuration variables
## Service Management
The deployment uses zinit for service management:
- **Development Service**: `tf-marketplace-dev`
- **Production Service**: `tf-marketplace-prod`
- **Configuration Files**:
- `/etc/zinit/tf-marketplace-dev.yaml`
- `/etc/zinit/tf-marketplace-prod.yaml`
- **Deployment Scripts**:
- `/etc/zinit/cmds/tf-marketplace-dev.sh`
- `/etc/zinit/cmds/tf-marketplace-prod.sh`

308
docs/ops/method/current/deployment.md Normal file
View File

@@ -0,0 +1,308 @@
# Basic Deployment Guide for Project Mycelium
This guide provides a simplified approach to deploying the Project Mycelium on a server, using `cargo run` directly.
> **🚀 For automated deployment, see [Automated Deployment Guide](./automated-deployment.md)**
We show the steps for the main branch, for development branch, simply add _dev, -dev accordingly and checkout into development.
## Prerequisites
- Linux server with:
- Git installed
- Rust toolchain installed
- Caddy web server installed
- zinit service manager installed
- Root or access
## Step 1: Clone the Repository
We'll clone the repository to a structured directory path:
```bash
# Create directory structure
mkdir -p /root/code/git.ourworld.tf/tfgrid_research/
cd /root/code/git.ourworld.tf/tfgrid_research/
# Clone the repository
git clone https://git.ourworld.tf/tfgrid_research/projectmycelium
cd projectmycelium
git checkout main
```
## Step 2: Create a zinit Service
Create a zinit service to run and manage the application:
1. First create the service script:
```bash
mkdir -p /etc/zinit/cmds
nano /etc/zinit/cmds/tf-marketplace.sh
```
Add the following content:
```bash
#!/bin/bash
cd /root/code/git.ourworld.tf/tfgrid_research/projectmycelium
git checkout main
exec /root/.cargo/bin/cargo run --release --bin projectmycelium -- --port 9999
```
Make the script executable:
```bash
chmod +x /etc/zinit/cmds/tf-marketplace.sh
```
2. Create the zinit service definition:
```bash
nano /etc/zinit/tf-marketplace.yaml
```
Add the following content:
```yaml
exec: "/bin/bash -c /etc/zinit/cmds/tf-marketplace.sh"
```
## Step 3: Configure Caddy
Create or update your Caddyfile to serve the application:
```bash
# Edit the Caddyfile
nano /root/code/github/despiegk/env_web/ourworld/ovh1_web_current/caddy/Caddyfile
```
- Add the Caddy file in Caddyfile
```
import threefold_marketplace.caddy
```
- Create `threefold_marketplace.caddy` then enter the following. Adjust your domain as needed.
```
threefold.pro {
reverse_proxy localhost:9999 {
header_up Host {host}
header_up X-Real-IP {remote}
header_up X-Forwarded-Proto {scheme}
}
}
```
## Step 4: Start Services with zinit
- Monitor the zinit service
```
zinit monitor tf-marketplace-prod
```
- Start the service
```bash
# Start the marketplace service
zinit start tf-marketplace-prod
# Restart Caddy to load new configuration
zinit restart caddy
```
## Automated Deployment with Makefile
For easier deployment, you can use the provided Makefile targets with separate dev and prod environments:
### First-Time Setup
1. **Set up deployment credentials:**
```bash
# Create deployment credentials file
cp .env.deploy.example .env.deploy
# Edit deployment credentials
nano .env.deploy
# Add your Gitea credentials:
GITEA_USER=your_username
GITEA_TOKEN=your_personal_access_token
```
2. **Deploy to server (choose one):**
**Option A: Setup development only:**
```bash
make setup-dev
```
**Option B: Setup production only:**
```bash
make setup-prod
```
**Option C: Setup both dev and prod:**
```bash
make setup
```
### Clean Environment Structure
The deployment system uses industry-standard environment separation:
```
/root/code/git.ourworld.tf/tfgrid_research/
├── dev/projectmycelium/ # Development branch (port 9998)
└── prod/projectmycelium/ # Production branch (port 9999)
```
**Benefits:**
- ✅ **No build conflicts**: Each environment has its own build cache
- ✅ **Independent deployments**: Dev and prod can be updated separately
- ✅ **Parallel compilation**: Both can build simultaneously
- ✅ **Clean separation**: Different branches, ports, and domains
### What each setup does:
**setup-dev:**
- Sets up development environment only
- Clones to `dev/projectmycelium/`
- Uses development branch
- Runs on port 9998 (dev.threefold.pro)
**setup-prod:**
- Sets up production environment only
- Clones to `prod/projectmycelium/`
- Uses main branch
- Runs on port 9999 (threefold.pro)
- Restarts Caddy for production
**setup (both):**
- Runs both dev and prod setups
- Complete environment setup
### Regular Deployments
**Simple Updates (recommended):**
```bash
# Update development branch (just git pull + restart)
make update-dev
# Update production branch (just git pull + restart)
make update-prod
```
**What happens:**
- SSH to server
- Navigate to repository directory
- Git pull latest changes
- Restart the service
**Monitoring:**
```bash
# Check service status
make status
# View service logs
make logs
# Get help with all commands
make help
```
**Manual Alternative:**
```bash
# Development
ssh root@info.ourworld.tf
cd /root/code/git.ourworld.tf/tfgrid_research/dev/projectmycelium
git checkout development && git pull
zinit restart tf-marketplace-dev
# Production
ssh root@info.ourworld.tf
cd /root/code/git.ourworld.tf/tfgrid_research/prod/projectmycelium
git checkout main && git pull
zinit restart tf-marketplace-prod
```
### Creating a Gitea Personal Access Token
1. Go to https://git.ourworld.tf/user/settings/applications
2. Click "Generate New Token"
3. Give it a descriptive name (e.g., "Project Mycelium Deployment")
4. Select minimal permissions: **repository read access only**
5. Copy the generated token and add it to your local `.env` file as `GITEA_TOKEN`
### Environment File Structure
**Two separate environment files for clean separation:**
**Local `.env.deploy` (deployment credentials only):**
```bash
# Gitea credentials for deployment
GITEA_USER=your_username
GITEA_TOKEN=your_personal_access_token
```
**Server `.env` (application secrets, auto-generated):**
```bash
# Application secret (automatically generated on server)
SECRET_KEY=auto_generated_secret_key
```
**Clean Separation:**
- **Deployment credentials**: Only in `.env.deploy`, used for git authentication
- **Application secrets**: Generated automatically on server in cloned repo
- **No mixing**: Deployment and application concerns completely separated
## Updating the Application
To update the application after making changes:
```bash
# Go to the repository directory
cd /root/code/git.ourworld.tf/tfgrid_research/projectmycelium
# Pull the latest changes
git checkout main
git pull
# Restart the application service
zinit restart tf-marketplace-prod
```
## Monitoring the Application
You can monitor the application status with these commands:
```bash
# Check if the application is running
zinit list
# View application logs
zinit log tf-marketplace-prod
# Monitor logs in real-time
tail -f /var/log/zinit/tf-marketplace-prod.log
```
## Troubleshooting
### Application Not Starting
Check for errors in the application logs:
```bash
zinit log tf-marketplace-prod
```
### Connection Refused
Make sure the application is running and listening on the correct port:
```bash
ss -tuln | grep 9999
```

109
docs/ops/method/current/deployment_method.md Normal file
View File

@@ -0,0 +1,109 @@
# Project Mycelium Ops Method
## Introduction
We present simple CLI tools to deploy the Project Mycelium.
## Tools
You can use quick utility tools gits from ucli and pal from scottyeager (see https://github.com/ucli-tools, https://github.com/scottyeager/pal)
- When any step is confirmed to work, save as version management control
```
git add . && pal /commit --yolo && git push
```
- To push as a PR when many commits create a whole step/task in the roadmap/todo
```
gits pr create --title 'Update' --base development && gits pr merge --pr-number $(gits pr-latest)
```
- To update the website from e.g. development using Makefile
```
make update-dev
```
- This will ssh into the server hosting the marketplace on dev and prod (dev.threefold.pro, threefold.pro)
## Prerequisites
- git, make
- Docker or Podman (if containerizing locally)
- SSH access to deployment hosts (dev/prod)
- kubectl/helm if deploying to Kubernetes
- Config/secrets available (Stripe keys, PostgREST endpoint, DB credentials)
## Environments
- Dev: dev.threefold.pro
- Prod: threefold.pro
## Configuration
- Environment variables (examples):
- STRIPE_SECRET_KEY, STRIPE_PUBLISHABLE_KEY
- POSTGREST_URL
- DATABASE_URL or connection parameters
- APP_ENV (development|production)
- Provide via .env files, system service env files, or Kubernetes Secrets as appropriate.
## Deployment Flows
- Local
- Build/run the app locally; verify basic UI and flows.
- Dev
- Use: `make update-dev` (as provided above)
- Verify deployment (see Verification)
- Prod
- Follow the same deployment pattern as dev against prod host or use corresponding prod target if defined.
## Verification
- Health: open the site on the target environment (dev.threefold.pro or threefold.pro)
- Basic smoke:
- Load homepage and key pages
- Add item to cart
- Run a test checkout with Stripe test keys (on dev)
## Rollback
- Identify last known good commit/tag
- Revert: `git revert <commit>` or checkout the tag/commit
- Redeploy using the same steps (e.g., `make update-dev`)
## Operations
- Logs: inspect application/server logs on target hosts; if running on Kubernetes, use `kubectl logs`
- Monitoring: confirm metrics/alerts if configured
- Backups: ensure regular DB and config backups and test restore procedures
## Troubleshooting
- Missing/incorrect env vars → verify configuration section
- SSH failures → validate keys and host reachability
- Stripe errors → confirm test vs live keys per environment
## Standard git equivalents
- Save work (equivalent to pal):
```
git add -A && git commit -m "<message>" && git push
```
- Create a PR: push your branch and open a PR on your Git hosting provider targeting `development` (or relevant branch)
## Checklists
- Pre-deploy: correct branch, env config present, backup taken (if needed)
- Post-deploy: site loads, smoke tests pass, logs clean of new errors
## References
- Dev design method: `../../../dev/method/current/design_method.md`
- DevOps overview: `../../../devops.md`
- Architecture & Roadmap: `../../../dev/design/current/projectmycelium-roadmap.md`
```mermaid
flowchart LR
Local[Local] --> Dev[Dev]
Dev --> Prod[Prod]
Dev -. smoke/verify .-> Dev
Prod -. checks .-> Prod
```

36
docs/ops/method/current/shortcuts.md Normal file
View File

@@ -0,0 +1,36 @@
# Devops Shortcuts
## Prerequisites
- Use fish for shell completion
- Use [gits](https://github.com/ucli-tools/gits) and [pal](https://github.com/scottyeager/pal) for quicker operations
## Devops Workflow Summary
- Create a branch
```
gits new development-refactoring
```
- Commit changes and push
- First time
```
git add . && pal /commit -y && git push --set-upstream origin development-refactoring
```
- After first time
```
git add . && pal /commit -y && git push
```
- Create PR and merge (e.g. base development branch is protected from direct push)
```
gits pr create --title 'Update' --base development && gits pr merge --pr-number $(gits pr-latest)
```
- Deploy Changed to Online App
```
gits pull development && make update-dev
```
- More info
- List all available commands
```
make help
```
- Read about [automated deployment](./automated-deployment.md)