herocode/baobab
Archived
13
0
Fork 0
Code Issues 8 Pull Requests Actions Packages Projects Releases Wiki Activity

end to end job management support

Browse Source
This commit is contained in:
Timur Gordon
2025-07-30 08:36:55 +02:00
parent 7d7ff0f0ab
commit 32c2cbe0cc
20 changed files with 2686 additions and 442 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
interfaces/websocket/server/Cargo.toml
View File

@@ -44,7 +44,8 @@ redis = { workspace = true }
uuid = { workspace = true }
tokio = { workspace = true }
chrono = { workspace = true }
rhai_dispatcher = { path = "../../../../rhailib/src/dispatcher" } # Corrected relative path
hero_dispatcher = { path = "../../../core/dispatcher" }
hero_job = { path = "../../../core/job" }
thiserror = { workspace = true }
heromodels = { path = "../../../../db/heromodels" }

69
interfaces/websocket/server/README.md
View File

@@ -1,29 +1,11 @@
# `server`: The Circles WebSocket Server
# `server`: The Hero WebSocket Server
The `server` crate provides a secure, high-performance WebSocket server built with `Actix`. It is the core backend component of the `circles` ecosystem, responsible for handling client connections, processing JSON-RPC requests, and executing Rhai scripts in a secure manner.
An OpenRPC WebSocket Server to interface with the [cores](../../core) of authorized circles.
## Features
- **`Actix` Framework**: Built on `Actix`, a powerful and efficient actor-based web framework.
- **WebSocket Management**: Uses `actix-web-actors` to manage each client connection in its own isolated actor (`CircleWs`), ensuring robust and concurrent session handling.
- **JSON-RPC 2.0 API**: Implements a JSON-RPC 2.0 API for all client-server communication. The API is formally defined in the root [openrpc.json](../../openrpc.json) file.
- **Secure Authentication**: Features a built-in `secp256k1` signature-based authentication system to protect sensitive endpoints.
- **Stateful Session Management**: The `CircleWs` actor maintains the authentication state for each client, granting or denying access to protected methods like `play`.
- **Webhook Integration**: Supports HTTP webhook endpoints for external services (Stripe, iDenfy) with signature verification and script execution capabilities.
## Core Components
### `spawn_circle_server`
This is the main entry point function for the server. It configures and starts the `Actix` HTTP server and sets up the WebSocket route with path-based routing (`/{circle_pk}`).
### `CircleWs` Actor
This `Actix` actor is the heart of the server's session management. A new instance of `CircleWs` is created for each client that connects. Its responsibilities include:
- Handling the WebSocket connection lifecycle.
- Parsing incoming JSON-RPC messages.
- Managing the authentication state of the session (i.e., whether the client is authenticated or not).
- Dispatching requests to the appropriate handlers (`fetch_nonce`, `authenticate`, and `play`).
- [OpenRPC Specification](openrpc.json) defines the API.
- There are RPC Operations specified to authorize a websocket connection.
- Authorized clients can execute Rhai scripts on the server.
- The server uses the [supervisor] to dispatch [jobs] to the [workers].
## Authentication
@@ -34,43 +16,6 @@ The server provides a robust authentication mechanism to ensure that only author
For a more detailed breakdown of the authentication architecture, please see the [ARCHITECTURE.md](docs/ARCHITECTURE.md) file.
## Webhook Integration
The server also provides HTTP webhook endpoints for external services alongside the WebSocket functionality:
- **Stripe Webhooks**: `POST /webhooks/stripe/{circle_pk}` - Handles Stripe payment events
- **iDenfy Webhooks**: `POST /webhooks/idenfy/{circle_pk}` - Handles iDenfy KYC verification events
### Webhook Features
- **Signature Verification**: All webhooks use HMAC signature verification for security
- **Script Execution**: Webhook events trigger Rhai script execution via the same Redis-based system
- **Type Safety**: Webhook payload types are defined in the `heromodels` library for reusability
- **Modular Architecture**: Separate handlers for each webhook provider with common utilities
For detailed webhook architecture and configuration, see [WEBHOOK_ARCHITECTURE.md](WEBHOOK_ARCHITECTURE.md).
## How to Run
### As a Library
The `server` is designed to be used as a library by the `launcher`, which is responsible for spawning a single multi-circle server instance that can handle multiple circles via path-based routing.
To run the server via the launcher with circle public keys:
```bash
cargo run --package launcher -- -k <circle_public_key1> -k <circle_public_key2> [options]
```
The launcher will start a single `server` instance that can handle multiple circles through path-based WebSocket connections at `/{circle_pk}`.
### Standalone Binary
A standalone binary is also available for development and testing purposes. See [`cmd/README.md`](cmd/README.md) for detailed usage instructions.
```bash
# Basic standalone server
cargo run
# With authentication and TLS
cargo run -- --auth --tls --cert cert.pem --key key.pem
```
cargo run

73
interfaces/websocket/server/examples/circle_auth_demo.rs Normal file
View File

@@ -0,0 +1,73 @@
use std::collections::HashMap;
use hero_websocket_server::ServerBuilder;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
env_logger::init();
// Define circles and their members
let mut circles = HashMap::new();
// Circle "alpha" with two members
circles.insert("alpha".to_string(), vec![
"04a1b2c3d4e5f6789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789ab".to_string(),
"04b2c3d4e5f6789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcd".to_string(),
]);
// Circle "beta" with one member
circles.insert("beta".to_string(), vec![
"04c3d4e5f6789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef".to_string(),
]);
// Circle "gamma" with three members
circles.insert("gamma".to_string(), vec![
"04d4e5f6789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef01".to_string(),
"04e5f6789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123".to_string(),
"04f6789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef012345".to_string(),
]);
// Build server with circle-based authentication
let server = ServerBuilder::new()
.host("127.0.0.1")
.port(8080)
.redis_url("redis://localhost:6379")
.with_auth()
.circles(circles)
.build()?;
println!("Starting WebSocket server with circle-based authentication...");
println!("Available circles:");
for (circle_id, members) in &server.circles {
println!(" Circle '{}' has {} members:", circle_id, members.len());
for (i, member) in members.iter().enumerate() {
println!(" Member {}: {}...{}", i + 1, &member[..10], &member[member.len()-10..]);
}
}
println!("\nTo connect to a specific circle, use URLs like:");
println!(" ws://127.0.0.1:8080/ws/alpha (for circle 'alpha')");
println!(" ws://127.0.0.1:8080/ws/beta (for circle 'beta')");
println!(" ws://127.0.0.1:8080/ws/gamma (for circle 'gamma')");
println!("\nAuthentication flow:");
println!("1. Connect to WebSocket URL for specific circle");
println!("2. Call 'fetch_nonce' method to get a nonce");
println!("3. Sign the nonce with your private key");
println!("4. Call 'authenticate' with your public key and signature");
println!("5. Server will verify signature AND check circle membership");
println!("6. Only members of the target circle will be authenticated");
// Start the server
let (task_handle, server_handle) = server.spawn_circle_server()?;
println!("\nServer started! Press Ctrl+C to stop.");
// Wait for the server to complete
match task_handle.await {
Ok(Ok(())) => println!("Server stopped successfully"),
Ok(Err(e)) => eprintln!("Server error: {}", e),
Err(e) => eprintln!("Task join error: {}", e),
}
Ok(())
}

9
interfaces/websocket/server/src/builder.rs
View File

@@ -13,6 +13,7 @@ pub struct ServerBuilder {
enable_auth: bool,
enable_webhooks: bool,
circle_worker_id: String,
circles: HashMap<String, Vec<String>>,
}
impl ServerBuilder {
@@ -28,6 +29,7 @@ impl ServerBuilder {
enable_auth: false,
enable_webhooks: false,
circle_worker_id: "default".to_string(),
circles: HashMap::new(),
}
}
@@ -72,6 +74,11 @@ impl ServerBuilder {
self.enable_webhooks = true;
self
}
pub fn circles(mut self, circles: HashMap<String, Vec<String>>) -> Self {
self.circles = circles;
self
}
pub fn build(self) -> Result<Server, TlsConfigError> {
Ok(Server {
@@ -87,8 +94,10 @@ impl ServerBuilder {
circle_worker_id: self.circle_worker_id,
circle_name: "default".to_string(),
circle_public_key: "default".to_string(),
circles: self.circles,
nonce_store: HashMap::new(),
authenticated_pubkey: None,
dispatcher: None,
})
}
}

220
interfaces/websocket/server/src/config.rs Normal file
View File

@@ -0,0 +1,220 @@
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::fs;
use std::path::Path;
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ServerConfig {
/// Server host address
#[serde(default = "default_host")]
pub host: String,
/// Server port
#[serde(default = "default_port")]
pub port: u16,
/// Redis connection URL
#[serde(default = "default_redis_url")]
pub redis_url: String,
/// Enable authentication
#[serde(default)]
pub auth: bool,
/// Enable TLS/WSS
#[serde(default)]
pub tls: bool,
/// Path to TLS certificate file
pub cert: Option<String>,
/// Path to TLS private key file
pub key: Option<String>,
/// Separate port for TLS connections
pub tls_port: Option<u16>,
/// Enable webhook handling
#[serde(default)]
pub webhooks: bool,
/// Circles configuration - maps circle names to lists of member public keys
#[serde(default)]
pub circles: HashMap<String, Vec<String>>,
}
impl Default for ServerConfig {
fn default() -> Self {
Self {
host: default_host(),
port: default_port(),
redis_url: default_redis_url(),
auth: false,
tls: false,
cert: None,
key: None,
tls_port: None,
webhooks: false,
circles: HashMap::new(),
}
}
}
impl ServerConfig {
/// Load configuration from a JSON file
pub fn from_file<P: AsRef<Path>>(path: P) -> Result<Self, ConfigError> {
let content = fs::read_to_string(path.as_ref())
.map_err(|e| ConfigError::FileRead(path.as_ref().to_path_buf(), e))?;
let config: ServerConfig = serde_json::from_str(&content)
.map_err(|e| ConfigError::JsonParse(e))?;
config.validate()?;
Ok(config)
}
/// Save configuration to a JSON file
pub fn to_file<P: AsRef<Path>>(&self, path: P) -> Result<(), ConfigError> {
let content = serde_json::to_string_pretty(self)
.map_err(|e| ConfigError::JsonSerialize(e))?;
fs::write(path.as_ref(), content)
.map_err(|e| ConfigError::FileWrite(path.as_ref().to_path_buf(), e))?;
Ok(())
}
/// Validate the configuration
pub fn validate(&self) -> Result<(), ConfigError> {
// Validate TLS configuration
if self.tls && (self.cert.is_none() || self.key.is_none()) {
return Err(ConfigError::InvalidTlsConfig(
"TLS is enabled but certificate or key path is missing".to_string()
));
}
// Validate that circles are not empty if auth is enabled
if self.auth && self.circles.is_empty() {
return Err(ConfigError::InvalidAuthConfig(
"Authentication is enabled but no circles are configured".to_string()
));
}
Ok(())
}
/// Create a sample configuration file
pub fn create_sample() -> Self {
let mut circles = HashMap::new();
circles.insert(
"example_circle".to_string(),
vec![
"0x1234567890abcdef1234567890abcdef12345678".to_string(),
"0xabcdef1234567890abcdef1234567890abcdef12".to_string(),
]
);
Self {
host: "127.0.0.1".to_string(),
port: 8443,
redis_url: "redis://127.0.0.1/".to_string(),
auth: true,
tls: false,
cert: Some("cert.pem".to_string()),
key: Some("key.pem".to_string()),
tls_port: Some(8444),
webhooks: false,
circles,
}
}
}
#[derive(Debug, thiserror::Error)]
pub enum ConfigError {
#[error("Failed to read config file {0}: {1}")]
FileRead(std::path::PathBuf, std::io::Error),
#[error("Failed to write config file {0}: {1}")]
FileWrite(std::path::PathBuf, std::io::Error),
#[error("Failed to parse JSON config: {0}")]
JsonParse(serde_json::Error),
#[error("Failed to serialize JSON config: {0}")]
JsonSerialize(serde_json::Error),
#[error("Invalid TLS configuration: {0}")]
InvalidTlsConfig(String),
#[error("Invalid authentication configuration: {0}")]
InvalidAuthConfig(String),
}
// Default value functions
fn default_host() -> String {
"127.0.0.1".to_string()
}
fn default_port() -> u16 {
8443
}
fn default_redis_url() -> String {
"redis://127.0.0.1/".to_string()
}
#[cfg(test)]
mod tests {
use super::*;
use tempfile::NamedTempFile;
#[test]
fn test_config_serialization() {
let config = ServerConfig::create_sample();
let json = serde_json::to_string_pretty(&config).unwrap();
let deserialized: ServerConfig = serde_json::from_str(&json).unwrap();
assert_eq!(config.host, deserialized.host);
assert_eq!(config.port, deserialized.port);
assert_eq!(config.circles.len(), deserialized.circles.len());
}
#[test]
fn test_config_file_operations() {
let config = ServerConfig::create_sample();
let temp_file = NamedTempFile::new().unwrap();
// Test writing
config.to_file(temp_file.path()).unwrap();
// Test reading
let loaded_config = ServerConfig::from_file(temp_file.path()).unwrap();
assert_eq!(config.host, loaded_config.host);
assert_eq!(config.circles.len(), loaded_config.circles.len());
}
#[test]
fn test_config_validation() {
let mut config = ServerConfig::default();
// Valid config should pass
assert!(config.validate().is_ok());
// TLS enabled without cert/key should fail
config.tls = true;
assert!(config.validate().is_err());
// Fix TLS config
config.cert = Some("cert.pem".to_string());
config.key = Some("key.pem".to_string());
assert!(config.validate().is_ok());
// Auth enabled without circles should fail
config.auth = true;
assert!(config.validate().is_err());
// Add circles
config.circles.insert("test".to_string(), vec!["pubkey".to_string()]);
assert!(config.validate().is_ok());
}
}

10
interfaces/websocket/server/src/handler.rs
View File

@@ -31,6 +31,16 @@ impl actix::StreamHandler<Result<ws::Message, ws::ProtocolError>> for Server {
self.handle_whoami(req.params, client_rpc_id, ctx)
}
"play" => self.handle_play(req.params, client_rpc_id, ctx),
"create_job" => self.handle_create_job(req.params, client_rpc_id, ctx),
"start_job" => self.handle_start_job(req.params, client_rpc_id, ctx),
"run_job" => self.handle_run_job(req.params, client_rpc_id, ctx),
"get_job_status" => self.handle_get_job_status(req.params, client_rpc_id, ctx),
"get_job_output" => self.handle_get_job_output(req.params, client_rpc_id, ctx),
"get_job_logs" => self.handle_get_job_logs(req.params, client_rpc_id, ctx),
"list_jobs" => self.handle_list_jobs(req.params, client_rpc_id, ctx),
"stop_job" => self.handle_stop_job(req.params, client_rpc_id, ctx),
"delete_job" => self.handle_delete_job(req.params, client_rpc_id, ctx),
"clear_all_jobs" => self.handle_clear_all_jobs(req.params, client_rpc_id, ctx),
_ => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),

999
interfaces/websocket/server/src/job_handlers.rs Normal file
View File

@@ -0,0 +1,999 @@
use crate::Server;
use actix::prelude::*;
use actix_web_actors::ws;
use hero_dispatcher::{Dispatcher, ScriptType};
use serde_json::{json, Value};
use std::time::Duration;
const TASK_TIMEOUT_DURATION: Duration = Duration::from_secs(30);
#[derive(serde::Serialize)]
struct SuccessResult {
success: bool,
}
#[derive(serde::Serialize)]
struct JobResult {
job_id: String,
}
#[derive(serde::Serialize)]
struct JsonRpcResponse {
jsonrpc: String,
result: Option<Value>,
error: Option<JsonRpcError>,
id: Value,
}
#[derive(serde::Serialize)]
struct JsonRpcError {
code: i32,
message: String,
data: Option<Value>,
}
impl Server {
pub fn handle_create_job(
&mut self,
params: Value,
client_rpc_id: Value,
ctx: &mut ws::WebsocketContext<Self>,
) {
// For now, create_job is the same as run_job
self.handle_run_job(params, client_rpc_id, ctx);
}
pub fn handle_start_job(
&mut self,
params: Value,
client_rpc_id: Value,
ctx: &mut ws::WebsocketContext<Self>,
) {
if self.enable_auth && !self.is_connection_authenticated() {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: "Authentication required".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
let job_id = match params.get("job_id").and_then(|v| v.as_str()) {
Some(id) => id.to_string(),
None => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32602,
message: "Missing required parameter: job_id".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
};
let dispatcher = match self.dispatcher.clone() {
Some(d) => d,
None => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32603,
message: "Internal error: dispatcher not available".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
};
let client_rpc_id_clone = client_rpc_id.clone();
let fut = async move {
dispatcher.start_job(&job_id).await
};
ctx.spawn(
fut.into_actor(self)
.map(move |res, _act, ctx_inner| match res {
Ok(_) => {
let result = SuccessResult { success: true };
let resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: Some(serde_json::to_value(result).unwrap()),
error: None,
id: client_rpc_id_clone.clone(),
};
ctx_inner.text(serde_json::to_string(&resp).unwrap());
}
Err(e) => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: format!("Failed to start job: {}", e),
data: None,
}),
id: client_rpc_id_clone,
};
ctx_inner.text(serde_json::to_string(&err_resp).unwrap());
}
})
.timeout(TASK_TIMEOUT_DURATION)
.map(move |res, _act, ctx_inner| {
if res.is_err() {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: "Request timed out".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx_inner.text(serde_json::to_string(&err_resp).unwrap());
}
}),
);
}
pub fn handle_get_job_status(
&mut self,
params: Value,
client_rpc_id: Value,
ctx: &mut ws::WebsocketContext<Self>,
) {
if self.enable_auth && !self.is_connection_authenticated() {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: "Authentication required".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
let job_id = match params.get("job_id").and_then(|v| v.as_str()) {
Some(id) => id.to_string(),
None => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32602,
message: "Missing required parameter: job_id".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
};
let dispatcher = match self.dispatcher.clone() {
Some(d) => d,
None => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32603,
message: "Internal error: dispatcher not available".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
};
let client_rpc_id_clone = client_rpc_id.clone();
let fut = async move {
dispatcher.get_job_status(&job_id).await
};
ctx.spawn(
fut.into_actor(self)
.map(move |res, _act, ctx_inner| match res {
Ok(status) => {
let resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: Some(json!(status)),
error: None,
id: client_rpc_id_clone.clone(),
};
ctx_inner.text(serde_json::to_string(&resp).unwrap());
}
Err(e) => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: format!("Failed to get job status: {}", e),
data: None,
}),
id: client_rpc_id_clone,
};
ctx_inner.text(serde_json::to_string(&err_resp).unwrap());
}
})
.timeout(TASK_TIMEOUT_DURATION)
.map(move |res, _act, ctx_inner| {
if res.is_err() {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: "Request timed out".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx_inner.text(serde_json::to_string(&err_resp).unwrap());
}
}),
);
}
pub fn handle_list_jobs(
&mut self,
_params: Value,
client_rpc_id: Value,
ctx: &mut ws::WebsocketContext<Self>,
) {
if self.enable_auth && !self.is_connection_authenticated() {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: "Authentication required".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
let dispatcher = match self.dispatcher.clone() {
Some(d) => d,
None => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32603,
message: "Internal error: dispatcher not available".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
};
let client_rpc_id_clone = client_rpc_id.clone();
let fut = async move {
dispatcher.list_jobs().await
};
ctx.spawn(
fut.into_actor(self)
.map(move |res, _act, ctx_inner| match res {
Ok(jobs) => {
let resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: Some(json!(jobs)),
error: None,
id: client_rpc_id_clone.clone(),
};
ctx_inner.text(serde_json::to_string(&resp).unwrap());
}
Err(e) => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: format!("Failed to list jobs: {}", e),
data: None,
}),
id: client_rpc_id_clone,
};
ctx_inner.text(serde_json::to_string(&err_resp).unwrap());
}
})
.timeout(TASK_TIMEOUT_DURATION)
.map(move |res, _act, ctx_inner| {
if res.is_err() {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: "Request timed out".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx_inner.text(serde_json::to_string(&err_resp).unwrap());
}
}),
);
}
pub fn handle_run_job(
&mut self,
params: Value,
client_rpc_id: Value,
ctx: &mut ws::WebsocketContext<Self>,
) {
if self.enable_auth && !self.is_connection_authenticated() {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: "Authentication required".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
let circle_pk = match params.get("circle_pk").and_then(|v| v.as_str()) {
Some(pk) => pk.to_string(),
None => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32602,
message: "Missing required parameter: circle_pk".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
};
let script_content = match params.get("script_content").and_then(|v| v.as_str()) {
Some(script) => script.to_string(),
None => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32602,
message: "Missing required parameter: script_content".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
};
let dispatcher = match self.dispatcher.clone() {
Some(d) => d,
None => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32603,
message: "Internal error: dispatcher not available".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
};
let client_rpc_id_clone = client_rpc_id.clone();
let fut = async move {
dispatcher
.new_job()
.context_id(&circle_pk)
.script_type(ScriptType::RhaiSAL)
.script(&script_content)
.timeout(TASK_TIMEOUT_DURATION)
.await_response()
.await
};
ctx.spawn(
fut.into_actor(self)
.map(move |res, _act, ctx_inner| match res {
Ok(job_id) => {
let result = JobResult { job_id };
let resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: Some(serde_json::to_value(result).unwrap()),
error: None,
id: client_rpc_id_clone.clone(),
};
ctx_inner.text(serde_json::to_string(&resp).unwrap());
}
Err(e) => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: format!("Failed to run job: {}", e),
data: None,
}),
id: client_rpc_id_clone,
};
ctx_inner.text(serde_json::to_string(&err_resp).unwrap());
}
})
.timeout(TASK_TIMEOUT_DURATION)
.map(move |res, _act, ctx_inner| {
if res.is_err() {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: "Request timed out".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx_inner.text(serde_json::to_string(&err_resp).unwrap());
}
}),
);
}
pub fn handle_get_job_output(
&mut self,
params: Value,
client_rpc_id: Value,
ctx: &mut ws::WebsocketContext<Self>,
) {
if self.enable_auth && !self.is_connection_authenticated() {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: "Authentication required".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
let job_id = match params.get("job_id").and_then(|v| v.as_str()) {
Some(id) => id.to_string(),
None => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32602,
message: "Missing required parameter: job_id".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
};
let dispatcher = match self.dispatcher.clone() {
Some(d) => d,
None => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32603,
message: "Internal error: dispatcher not available".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
};
let client_rpc_id_clone = client_rpc_id.clone();
let fut = async move {
dispatcher.get_job_output(&job_id).await
};
ctx.spawn(
fut.into_actor(self)
.map(move |res, _act, ctx_inner| match res {
Ok(output) => {
let resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: Some(json!(output)),
error: None,
id: client_rpc_id_clone.clone(),
};
ctx_inner.text(serde_json::to_string(&resp).unwrap());
}
Err(e) => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: format!("Failed to get job output: {}", e),
data: None,
}),
id: client_rpc_id_clone,
};
ctx_inner.text(serde_json::to_string(&err_resp).unwrap());
}
})
.timeout(TASK_TIMEOUT_DURATION)
.map(move |res, _act, ctx_inner| {
if res.is_err() {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: "Request timed out".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx_inner.text(serde_json::to_string(&err_resp).unwrap());
}
}),
);
}
pub fn handle_get_job_logs(
&mut self,
params: Value,
client_rpc_id: Value,
ctx: &mut ws::WebsocketContext<Self>,
) {
if self.enable_auth && !self.is_connection_authenticated() {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: "Authentication required".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
let job_id = match params.get("job_id").and_then(|v| v.as_str()) {
Some(id) => id.to_string(),
None => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32602,
message: "Missing required parameter: job_id".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
};
let dispatcher = match self.dispatcher.clone() {
Some(d) => d,
None => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32603,
message: "Internal error: dispatcher not available".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
};
let client_rpc_id_clone = client_rpc_id.clone();
let fut = async move {
dispatcher.get_job_logs(&job_id).await
};
ctx.spawn(
fut.into_actor(self)
.map(move |res, _act, ctx_inner| match res {
Ok(logs) => {
let result = json!({ "logs": logs });
let resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: Some(result),
error: None,
id: client_rpc_id_clone.clone(),
};
ctx_inner.text(serde_json::to_string(&resp).unwrap());
}
Err(e) => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: format!("Failed to get job logs: {}", e),
data: None,
}),
id: client_rpc_id_clone,
};
ctx_inner.text(serde_json::to_string(&err_resp).unwrap());
}
})
.timeout(TASK_TIMEOUT_DURATION)
.map(move |res, _act, ctx_inner| {
if res.is_err() {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: "Request timed out".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx_inner.text(serde_json::to_string(&err_resp).unwrap());
}
}),
);
}
pub fn handle_stop_job(
&mut self,
params: Value,
client_rpc_id: Value,
ctx: &mut ws::WebsocketContext<Self>,
) {
if self.enable_auth && !self.is_connection_authenticated() {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: "Authentication required".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
let job_id = match params.get("job_id").and_then(|v| v.as_str()) {
Some(id) => id.to_string(),
None => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32602,
message: "Missing required parameter: job_id".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
};
let dispatcher = match self.dispatcher.clone() {
Some(d) => d,
None => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32603,
message: "Internal error: dispatcher not available".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
};
let client_rpc_id_clone = client_rpc_id.clone();
let fut = async move {
dispatcher.stop_job(&job_id).await
};
ctx.spawn(
fut.into_actor(self)
.map(move |res, _act, ctx_inner| match res {
Ok(_) => {
let resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: Some(json!(null)),
error: None,
id: client_rpc_id_clone.clone(),
};
ctx_inner.text(serde_json::to_string(&resp).unwrap());
}
Err(e) => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: format!("Failed to stop job: {}", e),
data: None,
}),
id: client_rpc_id_clone,
};
ctx_inner.text(serde_json::to_string(&err_resp).unwrap());
}
})
.timeout(TASK_TIMEOUT_DURATION)
.map(move |res, _act, ctx_inner| {
if res.is_err() {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: "Request timed out".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx_inner.text(serde_json::to_string(&err_resp).unwrap());
}
}),
);
}
pub fn handle_delete_job(
&mut self,
params: Value,
client_rpc_id: Value,
ctx: &mut ws::WebsocketContext<Self>,
) {
if self.enable_auth && !self.is_connection_authenticated() {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: "Authentication required".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
let job_id = match params.get("job_id").and_then(|v| v.as_str()) {
Some(id) => id.to_string(),
None => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32602,
message: "Missing required parameter: job_id".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
};
let dispatcher = match self.dispatcher.clone() {
Some(d) => d,
None => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32603,
message: "Internal error: dispatcher not available".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
};
let client_rpc_id_clone = client_rpc_id.clone();
let fut = async move {
dispatcher.delete_job(&job_id).await
};
ctx.spawn(
fut.into_actor(self)
.map(move |res, _act, ctx_inner| match res {
Ok(_) => {
let resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: Some(json!(null)),
error: None,
id: client_rpc_id_clone.clone(),
};
ctx_inner.text(serde_json::to_string(&resp).unwrap());
}
Err(e) => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: format!("Failed to delete job: {}", e),
data: None,
}),
id: client_rpc_id_clone,
};
ctx_inner.text(serde_json::to_string(&err_resp).unwrap());
}
})
.timeout(TASK_TIMEOUT_DURATION)
.map(move |res, _act, ctx_inner| {
if res.is_err() {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: "Request timed out".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx_inner.text(serde_json::to_string(&err_resp).unwrap());
}
}),
);
}
pub fn handle_clear_all_jobs(
&mut self,
_params: Value,
client_rpc_id: Value,
ctx: &mut ws::WebsocketContext<Self>,
) {
if self.enable_auth && !self.is_connection_authenticated() {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: "Authentication required".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
let dispatcher = match self.dispatcher.clone() {
Some(d) => d,
None => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32603,
message: "Internal error: dispatcher not available".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
};
let client_rpc_id_clone = client_rpc_id.clone();
let fut = async move {
dispatcher.clear_all_jobs().await
};
ctx.spawn(
fut.into_actor(self)
.map(move |res, _act, ctx_inner| match res {
Ok(_) => {
let resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: Some(json!(null)),
error: None,
id: client_rpc_id_clone.clone(),
};
ctx_inner.text(serde_json::to_string(&resp).unwrap());
}
Err(e) => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: format!("Failed to clear jobs: {}", e),
data: None,
}),
id: client_rpc_id_clone,
};
ctx_inner.text(serde_json::to_string(&err_resp).unwrap());
}
})
.timeout(TASK_TIMEOUT_DURATION)
.map(move |res, _act, ctx_inner| {
if res.is_err() {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: "Request timed out".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx_inner.text(serde_json::to_string(&err_resp).unwrap());
}
}),
);
}
}

192
interfaces/websocket/server/src/lib.rs
View File

@@ -3,7 +3,8 @@ use actix_web::{web, App, Error, HttpRequest, HttpResponse, HttpServer};
use actix_web_actors::ws;
use log::{info, error}; // Added error for better logging
use once_cell::sync::Lazy;
use hero_dispatcher::{DispatcherBuilder, DispatcherError};
use hero_dispatcher::{Dispatcher, DispatcherBuilder, DispatcherError};
use hero_job::{Job, JobStatus};
use rustls::pki_types::PrivateKeyDer;
use rustls::ServerConfig as RustlsServerConfig;
use rustls_pemfile::{certs, pkcs8_private_keys};
@@ -29,10 +30,13 @@ static AUTHENTICATED_CONNECTIONS: Lazy<Mutex<HashMap<Addr<Server>, String>>> =
mod auth;
mod builder;
mod config;
mod handler;
mod job_handlers;
use crate::auth::{generate_nonce, NonceResponse};
pub use crate::builder::ServerBuilder;
pub use crate::config::{ServerConfig, ConfigError};
// Re-export server handle type for external use
pub type ServerHandle = actix_web::dev::ServerHandle;
@@ -100,6 +104,64 @@ struct FetchNonceParams {
pubkey: String,
}
// Job management parameter structures
#[derive(Debug, Serialize, Deserialize)]
struct CreateJobParams {
job: Job,
}
#[derive(Debug, Serialize, Deserialize)]
struct RunJobParams {
job: Job,
}
#[derive(Debug, Serialize, Deserialize)]
struct JobIdParams {
job_id: String,
}
// Job management result structures
#[derive(Debug, Serialize, Deserialize)]
struct CreateJobResult {
job_id: String,
}
#[derive(Debug, Serialize, Deserialize)]
struct RunJobResult {
job_id: String,
output: String,
}
#[derive(Debug, Serialize, Deserialize)]
struct JobStatusResult {
status: JobStatus,
}
#[derive(Debug, Serialize, Deserialize)]
struct JobOutputResult {
output: Option<String>,
}
#[derive(Debug, Serialize, Deserialize)]
struct JobLogsResult {
logs: Option<String>,
}
#[derive(Debug, Serialize, Deserialize)]
struct ListJobsResult {
jobs: Vec<Job>,
}
#[derive(Debug, Serialize, Deserialize)]
struct SuccessResult {
success: bool,
}
#[derive(Debug, Serialize, Deserialize)]
struct ClearJobsResult {
deleted_count: usize,
}
impl Actor for Server {
type Context = ws::WebsocketContext<Self>;
@@ -142,11 +204,14 @@ pub struct Server {
pub tls_port: Option<u16>,
pub enable_auth: bool,
pub enable_webhooks: bool,
pub circle_worker_id: String,
pub circle_name: String,
pub circle_public_key: String,
/// Map of circle IDs to vectors of public keys that are members of that circle
pub circles: HashMap<String, Vec<String>>,
nonce_store: HashMap<String, NonceResponse>,
authenticated_pubkey: Option<String>,
pub dispatcher: Option<Dispatcher>,
}
impl Server {
@@ -250,33 +315,34 @@ impl Server {
client_rpc_id: Value,
ctx: &mut ws::WebsocketContext<Self>,
) {
match serde_json::from_value::<FetchNonceParams>(params) {
Ok(params) => {
let nonce_response = generate_nonce();
self.nonce_store
.insert(params.pubkey, nonce_response.clone());
let resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: Some(serde_json::to_value(nonce_response).unwrap()),
error: None,
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&resp).unwrap());
}
Err(e) => {
// Extract pubkey string directly from params
let pubkey = match params.as_str() {
Some(pk) => pk.to_string(),
None => {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32602,
message: format!("Invalid parameters for fetch_nonce: {}", e),
message: "Invalid parameters for fetch_nonce: expected string pubkey".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
return;
}
}
};
let nonce_response = generate_nonce();
self.nonce_store.insert(pubkey, nonce_response.clone());
let resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: Some(serde_json::to_value(&nonce_response.nonce).unwrap()),
error: None,
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&resp).unwrap());
}
fn handle_authenticate(
@@ -327,18 +393,41 @@ impl Server {
};
if is_valid {
self.authenticated_pubkey = Some(auth_params.pubkey.clone());
AUTHENTICATED_CONNECTIONS
.lock()
.unwrap()
.insert(ctx.address(), auth_params.pubkey);
let resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: Some(serde_json::json!({ "authenticated": true })),
error: None,
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&resp).unwrap());
// Check if the authenticated public key belongs to the circle
let is_circle_member = self.circles
.get(&self.circle_name)
.map(|members| members.contains(&auth_params.pubkey))
.unwrap_or(false);
if is_circle_member {
self.authenticated_pubkey = Some(auth_params.pubkey.clone());
AUTHENTICATED_CONNECTIONS
.lock()
.unwrap()
.insert(ctx.address(), auth_params.pubkey);
let resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: Some(serde_json::json!({ "authenticated": true })),
error: None,
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&resp).unwrap());
} else {
log::warn!("Auth failed for {}: Public key {} not a member of circle {}",
self.circle_name, auth_params.pubkey, self.circle_name);
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32003,
message: "Public key not authorized for this circle".to_string(),
data: None,
}),
id: client_rpc_id,
};
ctx.text(serde_json::to_string(&err_resp).unwrap());
ctx.stop();
}
} else {
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
@@ -459,7 +548,7 @@ impl Server {
let redis_url_clone = self.redis_url.clone();
let _rpc_id_clone = client_rpc_id.clone();
let public_key = self.authenticated_pubkey.clone();
let worker_id_clone = self.circle_worker_id.clone();
let fut = async move {
let caller_id = public_key.unwrap_or_else(|| "anonymous".to_string());
@@ -471,7 +560,7 @@ impl Server {
hero_dispatcher
.new_job()
.context_id(&circle_pk_clone)
.worker_id(&worker_id_clone)
.script_type(hero_dispatcher::ScriptType::RhaiSAL)
.script(&script_content)
.timeout(TASK_TIMEOUT_DURATION)
.await_response()
@@ -484,35 +573,16 @@ impl Server {
ctx.spawn(
fut.into_actor(self)
.map(move |res, _act, ctx_inner| match res {
Ok(task_details) => {
if task_details.status == "completed" {
let output = task_details
.output
.unwrap_or_else(|| "No output".to_string());
let result_value = PlayResult { output };
let resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: Some(serde_json::to_value(result_value).unwrap()),
error: None,
id: client_rpc_id,
};
ctx_inner.text(serde_json::to_string(&resp).unwrap());
} else {
let error_message = task_details.error.unwrap_or_else(|| {
"Rhai script execution failed".to_string()
});
let err_resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: None,
error: Some(JsonRpcError {
code: -32000,
message: error_message,
data: None,
}),
id: client_rpc_id,
};
ctx_inner.text(serde_json::to_string(&err_resp).unwrap());
}
Ok(output) => {
// The dispatcher returns the actual string output from job execution
let result_value = PlayResult { output };
let resp = JsonRpcResponse {
jsonrpc: "2.0".to_string(),
result: Some(serde_json::to_value(result_value).unwrap()),
error: None,
id: client_rpc_id,
};
ctx_inner.text(serde_json::to_string(&resp).unwrap());
}
Err(e) => {
let (code, message) = match e {