mycelium/circle_support
27
0
Fork 0
Code Issues 19 Pull requests Projects Releases Packages Wiki Activity Actions

Failed Deployments on NodeID_7976 #86

New issue
Open
opened 2026-03-30 07:57:13 +00:00 by Sherwint · 2 comments
Sherwint commented 2026-03-30 07:57:13 +00:00
Member
Copy link
  • Made a few attempts to deploy on one of Robert's nodes_ ID_7976 and it ends with this error msg on all attempts - Failed to send request to twinId 15343 with command: zos.deployment.get, payload: {"contract_id":2085855} 0 deployment not found
  • The only errors I'm seeing from Grafana is about the contract failing to decomission. Post which the logs return to normal.

image

- Made a few attempts to deploy on one of Robert's nodes_ ID_7976 and it ends with this error msg on all attempts - `Failed to send request to twinId 15343 with command: zos.deployment.get, payload: {"contract_id":2085855} 0 deployment not found` - The only errors I'm seeing from Grafana is about the contract failing to decomission. Post which the logs return to normal. ![image](/attachments/f5748700-19bf-4905-a1f8-173b9ff41e5d)
image.png
671 KiB
omarz commented 2026-03-30 08:40:55 +00:00
Member
Copy link

it looks the provisiond service hangs after this step provisiond: 2026-03-29T04:27:18Z info provisioning workload workload-id=3309-2086709-vmhyphq workload-type=zmachine for about 3 mins then it panics due to context timesout

this could be coming from different steps, setup mycelium tapdev, starting cloud-hypervisor service or takes time to download the flist from the hub. still investigating the root cause

also noticed this node doesn't have any live vms, so will need to check its history further as well

it looks the provisiond service hangs after this step `provisiond: 2026-03-29T04:27:18Z info provisioning workload workload-id=3309-2086709-vmhyphq workload-type=zmachine` for about 3 mins then it panics due to context timesout this could be coming from different steps, setup mycelium tapdev, starting cloud-hypervisor service or takes time to download the flist from the hub. still investigating the root cause also noticed this node doesn't have any live vms, so will need to check its history further as well
👍 1
omarz commented 2026-04-01 07:58:30 +00:00
Member
Copy link

noticed similar behavior on node 1610 as well, pushing a patch to mainnet with a fix mycelium/circle_ops#738

noticed similar behavior on node 1610 as well, pushing a patch to mainnet with a fix https://forge.ourworld.tf/mycelium/circle_ops/issues/738
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
mycelium/circle_support#86
No description provided.