docs: Add comprehensive troubleshooting guide for nginx-nodeport with improved test script
This commit is contained in:
mik-tf
347
examples/nginx-nodeport/TROUBLESHOOTING.md
Normal file
347
examples/nginx-nodeport/TROUBLESHOOTING.md
Normal file
@@ -0,0 +1,347 @@
|
||||
# nginx-nodeport Troubleshooting Guide
|
||||
|
||||
## Common Issues and Solutions
|
||||
|
||||
### Issue: External IPv6 Connectivity Test Failed
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
⚠️ External Mycelium IPv6 connectivity test failed
|
||||
ℹ️ This might be due to:
|
||||
ℹ️ • Firewall or network policies blocking port 30091
|
||||
ℹ️ • Pod not running on this specific node
|
||||
ℹ️ • Mycelium network not properly configured
|
||||
```
|
||||
|
||||
**Root Causes & Solutions:**
|
||||
|
||||
#### 1. Mycelium Not Running on Your Machine ⭐ Most Common
|
||||
|
||||
**Problem:** You're trying to access from a machine without Mycelium installed/running.
|
||||
|
||||
**Solution:** Install and start Mycelium on your local machine:
|
||||
|
||||
```bash
|
||||
# Download Mycelium
|
||||
wget https://github.com/threefoldtech/mycelium/releases/latest/download/mycelium-private-x86_64-unknown-linux-musl.tar.gz
|
||||
tar -xzf mycelium-private-x86_64-unknown-linux-musl.tar.gz
|
||||
sudo chmod +x mycelium-private
|
||||
sudo mv mycelium-private /usr/local/bin/mycelium
|
||||
|
||||
# Start Mycelium with peers
|
||||
sudo mycelium --peers tcp://188.40.132.242:9651 tcp://136.243.47.186:9651 tcp://185.69.166.7:9651
|
||||
```
|
||||
|
||||
**After starting Mycelium, test connectivity:**
|
||||
```bash
|
||||
# Your machine should now have a Mycelium IPv6 address
|
||||
ip addr show | grep mycelium
|
||||
|
||||
# Test connection
|
||||
NODE_IPV6="552:5984:2d97:72dc:ff0f:39ef:6ec:a48c" # Replace with your node's IP
|
||||
curl -6 "http://[$NODE_IPV6]:30091/"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### 2. Testing from Wrong Location
|
||||
|
||||
**Problem:** Testing from within cluster or wrong network segment.
|
||||
|
||||
**Solution:** Test from a machine WITH Mycelium running:
|
||||
|
||||
```bash
|
||||
# ✅ CORRECT: Test from machine with Mycelium
|
||||
your-laptop$ curl -6 "http://[node-mycelium-ipv6]:30091/"
|
||||
|
||||
# ❌ WRONG: Test from within cluster (use localhost instead)
|
||||
kubectl exec pod/nginx-nodeport-xxx -- curl localhost:8080
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### 3. Firewall Blocking NodePort
|
||||
|
||||
**Problem:** Firewall rules blocking port 30091.
|
||||
|
||||
**Check firewall:**
|
||||
```bash
|
||||
# On the worker node
|
||||
sudo iptables -L -n | grep 30091
|
||||
sudo firewall-cmd --list-ports # if using firewalld
|
||||
```
|
||||
|
||||
**Solution:** Open port 30091 on worker nodes:
|
||||
```bash
|
||||
# Using iptables
|
||||
sudo iptables -I INPUT -p tcp --dport 30091 -j ACCEPT
|
||||
|
||||
# Using firewalld
|
||||
sudo firewall-cmd --permanent --add-port=30091/tcp
|
||||
sudo firewall-cmd --reload
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### 4. externalTrafficPolicy: Local Issue
|
||||
|
||||
**Problem:** With `externalTrafficPolicy: Local`, service is only accessible on nodes where pods are running.
|
||||
|
||||
**Understanding:**
|
||||
- ✅ Pod on node A → Access via node A's IP works
|
||||
- ❌ Pod on node A → Access via node B's IP fails
|
||||
|
||||
**Solution:** Always use the IP of the node where your pod is running:
|
||||
|
||||
```bash
|
||||
# Find which node has your pod
|
||||
POD_NODE=$(kubectl get pods -l app=nginx-nodeport -o jsonpath='{.items[0].spec.nodeName}')
|
||||
echo "Pod is on: $POD_NODE"
|
||||
|
||||
# Get THAT node's Mycelium IPv6
|
||||
NODE_IPV6=$(kubectl get node "$POD_NODE" -o jsonpath='{range .status.addresses[?(@.type=="InternalIP")]}{.address}{"\n"}{end}' | grep ':' | head -1)
|
||||
echo "Use this IP: $NODE_IPV6"
|
||||
|
||||
# Test
|
||||
curl -6 "http://[$NODE_IPV6]:30091/"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### 5. IPv6 vs IPv4 Confusion
|
||||
|
||||
**Problem:** Getting both IPv4 and IPv6 addresses, causing malformed URLs.
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
curl: (3) bad range in URL position 9:
|
||||
http://[10.20.2.2 51d:3596:6cc3:81e7:ff0f:d546:3737:4c8c]:30091/
|
||||
```
|
||||
|
||||
**Solution:** Filter for IPv6 only:
|
||||
```bash
|
||||
# Correct way to get IPv6 only
|
||||
kubectl get nodes -o jsonpath='{range .items[*]}{range .status.addresses[?(@.type=="InternalIP")]}{.address}{"\n"}{end}{end}' | grep -E '^[0-9a-f]+:[0-9a-f]+:'
|
||||
|
||||
# NOT this (gets both IPv4 and IPv6)
|
||||
kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Pod Not Accessible
|
||||
|
||||
**Check pod status:**
|
||||
```bash
|
||||
kubectl get pods -l app=nginx-nodeport
|
||||
kubectl describe pod -l app=nginx-nodeport
|
||||
kubectl logs -l app=nginx-nodeport
|
||||
```
|
||||
|
||||
**Test internal connectivity:**
|
||||
```bash
|
||||
POD_NAME=$(kubectl get pods -l app=nginx-nodeport -o name | head -1)
|
||||
|
||||
# Test from within pod
|
||||
kubectl exec $POD_NAME -- curl -s http://localhost:8080/
|
||||
|
||||
# Test health endpoint
|
||||
kubectl exec $POD_NAME -- curl -s http://localhost:8080/health
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Service Not Working
|
||||
|
||||
**Check service configuration:**
|
||||
```bash
|
||||
# Verify service type
|
||||
kubectl get svc nginx-nodeport-service -o jsonpath='{.spec.type}'
|
||||
# Should output: NodePort
|
||||
|
||||
# Check NodePort assignment
|
||||
kubectl get svc nginx-nodeport-service -o jsonpath='{.spec.ports[0].nodePort}'
|
||||
# Should output: 30091
|
||||
|
||||
# Check endpoints
|
||||
kubectl get endpoints nginx-nodeport-service
|
||||
# Should show pod IP and port
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Wrong Content Displayed
|
||||
|
||||
**Update ConfigMap:**
|
||||
```bash
|
||||
# Regenerate content with current IPs
|
||||
./update-content.sh
|
||||
|
||||
# Restart deployment to pick up changes
|
||||
kubectl rollout restart deployment/nginx-nodeport
|
||||
|
||||
# Wait for rollout
|
||||
kubectl rollout status deployment/nginx-nodeport
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing Checklist
|
||||
|
||||
Run through this checklist to verify everything:
|
||||
|
||||
- [ ] **Mycelium Running:** `ip addr show | grep mycelium`
|
||||
- [ ] **Pod Running:** `kubectl get pods -l app=nginx-nodeport`
|
||||
- [ ] **Service Exists:** `kubectl get svc nginx-nodeport-service`
|
||||
- [ ] **NodePort 30091:** `kubectl get svc nginx-nodeport-service -o jsonpath='{.spec.ports[0].nodePort}'`
|
||||
- [ ] **Pod Node Located:** `kubectl get pods -l app=nginx-nodeport -o wide`
|
||||
- [ ] **Node IPv6 Found:** `kubectl get node <node-name> -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}' | grep ':'`
|
||||
- [ ] **Internal Access:** `kubectl exec <pod> -- curl -s http://localhost:8080/`
|
||||
- [ ] **External Access:** `curl -6 "http://[node-ipv6]:30091/"`
|
||||
|
||||
---
|
||||
|
||||
## Quick Validation Commands
|
||||
|
||||
```bash
|
||||
# Complete test sequence
|
||||
cd myceliumcloud-examples/examples/nginx-nodeport
|
||||
|
||||
# 1. Run automated tests
|
||||
./test-nodeport-ipv6.sh
|
||||
|
||||
# 2. Find your pod's node
|
||||
POD_NODE=$(kubectl get pods -l app=nginx-nodeport -o jsonpath='{.items[0].spec.nodeName}')
|
||||
echo "Pod is on node: $POD_NODE"
|
||||
|
||||
# 3. Get that node's Mycelium IPv6
|
||||
NODE_IPV6=$(kubectl get node "$POD_NODE" -o jsonpath='{range .status.addresses[?(@.type=="InternalIP")]}{.address}{"\n"}{end}' | grep -E '^[0-9a-f]+:[0-9a-f]+:' | head -1)
|
||||
echo "Node Mycelium IPv6: $NODE_IPV6"
|
||||
|
||||
# 4. Test from machine with Mycelium
|
||||
curl -6 "http://[$NODE_IPV6]:30091/"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Expected Behavior
|
||||
|
||||
### ✅ Working Correctly
|
||||
|
||||
When everything is working, you should see:
|
||||
|
||||
1. **Test Script Output:**
|
||||
```
|
||||
✅ nginx-nodeport pods are ready
|
||||
✅ nginx configuration is valid
|
||||
✅ Health endpoint is working
|
||||
✅ NodePort service is configured correctly
|
||||
✅ NodePort is correctly set to 30091
|
||||
```
|
||||
|
||||
2. **External Access:**
|
||||
```bash
|
||||
$ curl -6 "http://[552:5984:2d97:72dc:ff0f:39ef:6ec:a48c]:30091/"
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<title>Mycelium Cloud - Nginx NodePort Website</title>
|
||||
...
|
||||
```
|
||||
|
||||
3. **Browser Access:**
|
||||
- Navigate to: `http://[node-mycelium-ipv6]:30091/`
|
||||
- Should see: Styled website with "NODEPORT SECURE" badge
|
||||
|
||||
---
|
||||
|
||||
## Advanced Debugging
|
||||
|
||||
### Check kube-proxy
|
||||
|
||||
```bash
|
||||
# On worker node
|
||||
sudo iptables-save | grep 30091
|
||||
```
|
||||
|
||||
### View Service Routing
|
||||
|
||||
```bash
|
||||
# Check service details
|
||||
kubectl describe svc nginx-nodeport-service
|
||||
|
||||
# Check endpoints
|
||||
kubectl get endpoints nginx-nodeport-service -o yaml
|
||||
```
|
||||
|
||||
### Test from Another Pod
|
||||
|
||||
```bash
|
||||
# Create test pod
|
||||
kubectl run test-pod --image=nginx:alpine --rm -it -- sh
|
||||
|
||||
# From inside test pod
|
||||
wget -O- http://nginx-nodeport-service:8080/
|
||||
# Should work (internal cluster access)
|
||||
|
||||
# Try NodePort (won't work from inside cluster)
|
||||
wget -O- http://[node-ip]:30091/
|
||||
# May not work depending on network policies
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Still Having Issues?
|
||||
|
||||
If none of the above solutions work:
|
||||
|
||||
1. **Check Logs:**
|
||||
```bash
|
||||
kubectl logs -l app=nginx-nodeport --tail=50
|
||||
```
|
||||
|
||||
2. **Describe Resources:**
|
||||
```bash
|
||||
kubectl describe deployment nginx-nodeport
|
||||
kubectl describe service nginx-nodeport-service
|
||||
kubectl describe pod -l app=nginx-nodeport
|
||||
```
|
||||
|
||||
3. **Verify Network:**
|
||||
```bash
|
||||
# Check pod can reach internet
|
||||
kubectl exec <pod> -- ping -c 3 8.8.8.8
|
||||
|
||||
# Check pod networking
|
||||
kubectl exec <pod> -- ip addr
|
||||
kubectl exec <pod> -- ip route
|
||||
```
|
||||
|
||||
4. **Compare with hostNetwork:**
|
||||
- The [`nginx-mycelium`](../nginx-mycelium/) example uses hostNetwork
|
||||
- If that works but NodePort doesn't, it's likely a service/networking issue
|
||||
- If neither works, check Mycelium connectivity
|
||||
|
||||
---
|
||||
|
||||
## Common Questions
|
||||
|
||||
**Q: Why can't I access from any node's IP?**
|
||||
A: With `externalTrafficPolicy: Local`, you can only access via the node where the pod is running. This preserves source IP but limits accessibility.
|
||||
|
||||
**Q: Should I change to `externalTrafficPolicy: Cluster`?**
|
||||
A: You can, but you'll lose source IP preservation. Decision depends on your logging/security needs.
|
||||
|
||||
**Q: How is this different from hostNetwork?**
|
||||
A: NodePort provides network isolation and standard Kubernetes patterns. HostNetwork gives direct host access (less secure). See [compare-approaches.md](compare-approaches.md).
|
||||
|
||||
**Q: Why port 30091 and not 80?**
|
||||
A: Kubernetes NodePorts use range 30000-32767. Port 80 requires LoadBalancer or Ingress type services.
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2025-01-07
|
||||
**Related Docs:**
|
||||
- [nginx-nodeport.md](nginx-nodeport.md) - Complete guide
|
||||
- [compare-approaches.md](compare-approaches.md) - Security comparison
|
||||
- [../nginx-variants.md](../nginx-variants.md) - All nginx variants
|
||||
Reference in New Issue
Block a user