Introduction

OpsRamp Kubernetes 2.0 Agent allows you to collect and manage logs from your cluster. Log Management can be enabled during installation, and later modified as per your needs. This guide explains how to configure and customize log collection after the agent has been installed.

Prerequisites

1. Enable Log Management at client level
   • Enable Log Management while creating a new client
   • Enable Log Management for an existing client
2. Enable Logs during Agent installation
   
   • During installation of the Kubernetes 2.0 Agent, ensure that the Logs option is enabled.
   • For details, refer to the Agent Installation Guide.

Steps to Configure

Step 1: Get the Existing ConfigMap

kubectl get configmap opsramp-logs-user-config -n <agent-installed-namespace> -oyaml

Sample ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: "opsramp-logs-user-config"
  labels:
    app: "opsramp-logs-user-config"
  namespace: opsramp-agent
data:
  logsConfig.yaml: |
    pods:
      enable: true
      additional_labels:   
      namespaces:
        include: 
        is_include_regex: false
        exclude: 
        is_exclude_regex: false
      log_level: "Unspecified"
      masking:
        - attribute_type: ""
          attribute_key: ""
          text: ""
          placeholder: ""
    node:
      - logSource: syslog
        enable: true
        configs:
          - mount_path: ["/var/log/syslog","/var/log/messages"]         
            multiline:
              line_start_pattern: '^\s*(?P<timestamp>\w*\s*\d*\s*\d*:\d*:\d*)\s*(?P<host>[^\s]*)'
            parser_type: "regex"
            parser_expression: '^\s*(?P<timestamp>\w*\s*\d*\s*\d*:\d*:\d*)\s*(?P<host>[^\s]*)\s*(?P<syslog_tag>[^:]*):\s*(?P<message>.*)$'
            timestamp_layout_type: "strptime"
            timestamp_layout: "%b %e %H:%M:%S"
            log_level: "Unspecified"

Step 2: Edit an Existing ConfigMap

To modify an existing logs configuration, use:

kubectl edit configmap opsramp-logs-user-config -n <agent-installed-namespace>

Step 3: Update Parameters

Update the parameters as you need.

Step 4: Save & Apply Changes

The OpsRamp Agent automatically restarts the OpenTelemetry collector with the new configuration.

Configuration Options

1. Pods Logs Configuration

By default, all pod logs are collected. You can customize this behavior:

Disable pod logs

If users prefer not to collect pod logs by default, they can disable this feature by setting enable to false in the configuration. This can be done by running the following command:

pods:
  enable: false

Adding Additional Labels

The additional_labels field in the ConfigMap allows users to specify key-value pairs that will be added as attributes to each log record.

Example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: "opsramp-logs-user-config"
  labels:
    app: "opsramp-logs-user-config"
  namespace: <agent-installed-namespace>
data:
  logsConfig.yaml: |
    pods:
      enable: true
      additional_labels:  
        app: nginx
        env: staging 

Namespace Filtering

By default, the agent collects pod logs from all namespaces. To restrict log collection to specific namespaces, users need to define inclusion or exclusion rules within the opsramp-logs-user-config ConfigMap.

Example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: "opsramp-logs-user-config"
  labels:
    app: "opsramp-logs-user-config"
  namespace: <agent-installed-namespace>
data:
  logsConfig.yaml: |  
    pods: 
      enable: true
      namespaces:
        include: 
          - ^monitoring
        is_include_regex: true
        exclude: 
          - monitoring-test
        is_exclude_regex: false

• The include and exclude fields can accept either a string or a regex pattern as a namespace.
• If a user specifies a regex pattern for a namespace in the include/exclude field, the corresponding flags is_include_regex or is_exclude_regex must be set to true.
• If a user specifies a namespace that is not a regex pattern in the include/exclude field, the flags is_include_regex and is_exclude_regex must be set to false.
• For example, the agent collects logs of pods in all namespaces that start with monitoring but excludes logs from pods in the monitoring-test namespace.

Use Case 1:

Use Case 2:

Log-Level Filtering

• Log severity levels are ordered as follows: Fatal > Error > Warn > Info > Debug > Trace > Unspecified/Unknown.
• By default, the agent evaluates log severity for each entry and sends logs to the OpsRamp platform based on this hierarchy.
• Users can filter or drop logs based on their severity level by modifying the opsramp-logs-user-config ConfigMap.

Example 1: Filter logs with severity level Warn or higher (includes Warn, Fatal, and Error) and drop logs below Warn (Info, Debug, Trace, Unknown).

apiVersion: v1
kind: ConfigMap
metadata:
  name: "opsramp-logs-user-config"
  labels:
    app: "opsramp-logs-user-config"
  namespace: opsramp-agent
data:
  logsConfig.yaml: |
    pods:
      enable: true
      log_level: "Warn"

Example 2: Filter logs with severity level Info or higher (includes Info, Warn, Error, and Fatal) and drop logs below Info (Debug, Trace, Unknown).

apiVersion: v1
kind: ConfigMap
metadata:
  name: "opsramp-logs-user-config"
  labels:
    app: "opsramp-logs-user-config"
  namespace: opsramp-agent
data:
  logsConfig.yaml: |
    pods:
      enable: true
      log_level: "Info"

Masking

Masking involves obscuring sensitive or personally identifiable information (PII) in log entries to ensure compliance with data protection regulations.

Example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: "opsramp-logs-user-config"
  labels:
    app: "opsramp-logs-user-config"
  namespace: opsramp-agent
data:
  logsConfig.yaml: |
    pods:
      enable: true
      masking:
        - attribute_type: ""
          attribute_key: "k8s.namespace.name"
          text: "^ops"
          placeholder: "***"

• To mask sensitive data (e.g., namespaces starting with “ops”):
• If the namespace is opsramp-agent, it will appear as ***ramp-agent after masking.

Field Descriptions:

MaxTime & TopN

• max_time: Represents the duration from which logs are retrieved from log files. The default value is 2 hours. Users can adjust this by adding a max_time field to their logs user configuration. For example, “Setting MaxTime to 2 hours ensures only recently modified pod log files are processed, which helps reduce overhead and boost performance.”
• top_n: Determines the number of files to track when using file ordering. The default value is 500 files, with a default maxTime of 2 hours. Users can adjust this by adding a top_n field to their logs user configuration. It will collect top_n files modified within maxTime

Example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: "opsramp-logs-user-config"
  labels:
    app: "opsramp-logs-user-config"
  namespace: opsramp-agent
data:
  logsConfig.yaml: |
    pods:
      enable: true
      max_time: 10h
      top_n: 700   

2. Node Logs Configuration

By default, the agent collects node logs, such as syslogs, from all Kubernetes nodes using the opsramp-logs-user-config ConfigMap. This ConfigMap is available immediately after the agent installation.

Disable Node Logs

The following is the default configuration for syslog collection:

node:
  - logSource: syslog
    enable: true
    configs:
      - mount_path: ["/var/log/syslog","/var/log/messages"]
        parser_type: "regex"
        parser_expression: '^\s*(?P<timestamp>\w*\s*\d*...'

If you do not want to collect syslogs by default, set enable: false in the configuration by running the following command.

Parser Expression by OS Version

• For nodes running Ubuntu version 22.04:

  parser_expression: '^\s*(?P<timestamp>\w*\s*\d*\s*\d*:\d*:\d*)\s*(?P<host>[^\s]*)\s*(?P<syslog_tag>[^:]*):\s*(?P<message>.*)$' 

• For nodes running Ubuntu version 24.0x, update to:

  parser_expression: '^(?P<timestamp>\d{4}\W*\d{2}\W*\d{2}\w\d{2}:\d{2}:\d{2}.\d*\+\d{2}:\d{2})\s(?P<host>\w*\W\w*)\s(?P<syslog_tag>\w*\W\d*\W):(?P<message>.*)' 

Configurable Fields

Here are a few key points to keep in mind:

1. All fields in parser_expression will be parsed to log attributes only by default.
2. Users can customize all fields according to their preferences. Furthermore, they have the flexibility to modify the default configuration, as long as these settings are not the original defaults.
3. If a user has any custom configurations for syslog, in addition to the default settings, they can include these configurations in the ‘configs’ field as an array.

Example ConfigMap Parser Expression:

node:
  - logSource: syslog
    enable: true
    configs:
      - mount_path: ["/var/log/syslog","/var/log/messages"]         
        multiline:
          line_start_pattern: '^\s*(?P<timestamp>\w*\s*\d*\s*\d*:\d*:\d*)\s*(?P<host>[^\s]*)'
        parser_type: "regex"
        parser_expression: '^\s*(?P<timestamp>\w*\s*\d*\s*\d*:\d*:\d*)\s*(?P<host>[^\s]*)\s*(?P<syslog_tag>[^:]*):\s*(?P<message>.*)$'
        timestamp_layout_type: "strptime"
        timestamp_layout: "%b %e %H:%M:%S"
      - mount_path: ["/opt/log/syslog/0.log"]         
        multiline:
          line_end_pattern: '^\s*(?P<timestamp>\w*\s*\d*\s*\d*:\d*:\d*)\s*(?P<host>[^\s]*)'
        parser_type: "json"
        parser_expression: '^\s*(?P<timestamp>\w*\s*\d*\s*\d*:\d*:\d*)\s*(?P<host>[^\s]*)\s*(?P<syslog_tag>[^:]*):\s*(?P<message>.*)$'
        timestamp_layout_type: "strptime"
        timestamp_layout: "%b %e %H:%M:%S"

Log-level Filtering for Syslog

• The Log severity/level ordering is structured as follows:
  Fatal > Error > Warn > Info > Debug > Trace> Unspecified/Unknown
• By default, the agent will assess the log severity outlined above for each log entry and send it to the OpsRamp platform.
• If the user wants to filter/drop some logs based on severity, then the user can specify a config like below

Example:

node:
  - logSource: syslog
    enable: true
    configs:
      - mount_path: ["/var/log/syslog","/var/log/messages"]         
        multiline:
          line_start_pattern: '^\s*(?P<timestamp>\w*\s*\d*\s*\d*:\d*:\d*)\s*(?P<host>[^\s]*)'
        parser_type: "regex"
        parser_expression: '^\s*(?P<timestamp>\w*\s*\d*\s*\d*:\d*:\d*)\s*(?P<host>[^\s]*)\s*(?P<syslog_tag>[^:]*):\s*(?P<message>.*)$'
        timestamp_layout_type: "strptime"
        timestamp_layout: "%b %e %H:%M:%S"
        log_level: "warn"

MaxTime

The max_time represents the duration from which logs are retrieved from log files. The default value is 2 hours. Users can adjust this by adding a max_time field to their logs user configuration. For example, “Setting MaxTime to 2 hours ensures only recently modified pod log files are processed, which helps reduce overhead and boost performance.

Example:

node:
  - logSource: syslog
    enable: true
    configs:
      - mount_path: ["/var/log/syslog","/var/log/messages"]         
        multiline:
          line_start_pattern: '^\s*(?P<timestamp>\w*\s*\d*\s*\d*:\d*:\d*)\s*(?P<host>[^\s]*)'
        parser_type: "regex"
        parser_expression: '^\s*(?P<timestamp>\w*\s*\d*\s*\d*:\d*:\d*)\s*(?P<host>[^\s]*)\s*(?P<syslog_tag>[^:]*):\s*(?P<message>.*)$'
        timestamp_layout_type: "strptime"
        timestamp_layout: "%b %e %H:%M:%S"
        max_time: 5h

View Logs in OpsRamp Portal

Users can view logs in the OpsRamp portal from Infrastructure > Logs.

1. Default Logs Screen: Access your logs through the Logs UI.

   

2. Logs Filter: Click on +FILTER to select resource types and operators (e.g., Equals, Regex Not Contains).
3. Live Tail: Click on LIVE TAIL to view real-time logs.
4. Log Details: Click on any log entry for detailed information.

   

Once you edit the K8s ConfigMap and apply it to your cluster, the OpsRamp Agent will pick up the latest configuration changes within a few minutes. This version organizes the content into clear sections and uses straightforward language for better understanding.

Troubleshooting

If you encounter logs issues, see the Troubleshooting documentation.