Fluentd agent is setup to automatically read extra format and source configuration files from folders /etc/td-agent/conf.d/ and /etc/td-agent/source.d/.

Formatting extensions

You can use fluentd debug tool to debug your format definitions and regex builder to test your regex expressions.

To add new gateway role specific log format processing in you role playbook:

- name: upload gateway service config
  template: 
    dest: "/etc/td-agent/conf.d/service-gateway.conf"
    force: yes
    src: service-gateway.conf.j2
    owner: td-agent
    group: td-agent
  become: yes
  notify: reload td-agent

In service-gateway.conf.j2:


  <match athena.service.gateway.httpd_access_log1>
    @type rewrite_tag_filter
    rewriterule1 http_code ^2.+$ athena.service.${tag_parts[2]}.httpd_access_log1.info
    rewriterule2 http_code ^3.+$ athena.service.${tag_parts[2]}.httpd_access_log1.info
    rewriterule3 http_code ^4.+$ athena.service.${tag_parts[2]}.httpd_access_log1.warn
    rewriterule4 http_code !^[234].+$ athena.service.${tag_parts[2]}.httpd_access_log1.error
  </match>

Source extensions

To add new gateway role specific log source in your role playbook:

- name: upload gateway source config
  template: 
    dest: "/etc/td-agent/source.d/source-gateway.conf"
    force: yes
    src: source-gateway.conf.j2
    owner: td-agent
    group: td-agent
  become: yes
  notify: reload td-agent

In source-gateway.conf.j2:

<source>
  @type   tail
  @label  @mainstream
  path   /var/log/apache2/athena.log
  pos_file /var/data/fluentd/athena.log.pos
  tag athena.service.gateway.httpd_access_log1
  format /^\[(?<time>[^\]]*)\] (?<http_code>[0-9]*) (?<athena_service>\w*) (?<athena_ip>[^ ]*) (?<athena_user>[\w\.]*) "(?<athena_message>[^"]*)"$/
  time_format %d/%b/%Y:%H:%M:%S %z
</source>

<source>
  @type   tail
  @label  @mainstream
  path   /var/log/apache2/error.log
  pos_file /var/data/fluentd/athena.log.pos
  format /^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\] (?:\[pid (?<pid>[^\]]*)\])? (?:\[client (?<athena_ip>[^\]]*)\])?(?<athena_message>.*)$/
  tag athena.service.gateway.httpd_error_log1
</source>