Workload Metrics Issues

Workload Metrics Not Visible in OpsRamp Portal

Issue

Workload metrics (for applications such as Nginx, MySQL, etc.) are not visible in the OpsRamp portal, even though the Kubernetes 2.0 agent pods are running.

Cause

Workload Monitoring not Enabled
- Unlike cluster metrics, workload monitoring does not start automatically with the Kubernetes 2.0 agent.
- A ConfigMap must be applied to specify which workloads to monitor.
Workload Pods not Running
- If the application workload (e.g., Nginx, MySQL, Redis) is not deployed or not running, the agent cannot collect metrics.
Missing Credentials or Incorrect Configuration
- Some workloads require credentials (e.g., database username/password) defined in the ConfigMap. If these are missing or incorrect, workload metrics will not be collected.
ConfigMap Misconfiguration
- The ConfigMap opsramp-workload-metric-user-config may be missing workload definitions or incorrectly configured.

Solution

Deploy Workload Pods
- Ensure the workload pods (e.g., nginx, mysql) are deployed and running:
```
kubectl get pods -n <namespace>
```

Apply Workload ConfigMap

Create or edit the ConfigMap to include the workloads you want to monitor:
```
kubectl apply -f opsramp-workload-metric-user-config.yaml -n <namespace>
```

Example snippet for Nginx:

workloads:
  nginx:
    enabled: true
    endpoint: "http://nginx-service:80/status"

Provide Required Credentials
- For workloads that require authentication (e.g., MySQL), update the ConfigMap with the correct credentials:
```
workloads:
  mysql:
    enabled: true
    username: <db-username>
    password: <db-password>
```

Verify ConfigMap Applied

Confirm the ConfigMap is active:

kubectl get configmap opsramp-workload-metric-user-config -n <namespace> -o yaml

Validate in Portal
- Once the pipeline is started, navigate to the Workload Monitoring section of the OpsRamp portal.
- Verify metrics for the selected workloads are visible.

Workload Metrics Troubleshooting

Common Scenarios & Impacts

Scenario	Impact	Logs to Check
ConfigMap applied with correct syntax	Otel restarts, metrics exported	None
ConfigMap has missing fields / invalid syntax	Otel restarts, shows schema validation error for that workload; other workloads continue to export metrics	configuration error
ConfigMap has wrong configuration (wrong IP/Port/Credentials)	Otel restarts, no error, but skips that workload; other workloads continue to export metrics	Error scraping metrics or Failed to scrape Prometheus endpoint

Debugging Steps

Enable Debug Mode
Make sure the agent is running with LOG_LEVEL=debug.

Check for Syntax Errors

kubectl logs <master-pod-name> | grep "configuration error"

Example:

configuration error: workload cockroach/cockroach did not conform to schema.
Error at "/port": value must be an integer

Update the config, reapply, and restart.

Check Pipelines in Worker Pod

kubectl exec -it <worker-pod-name> -- sh
cd /var/log/opsramp
cat agent.log

Look for:

Receivers, Exporters, Processors, Pipelines.
If some are missing → configuration issue.
If all are present → configuration is valid, metrics should be exported.