Open in app

Sign in

Write

Sign in

Set configMap of fluentd-daemonset-elasticsearch in kubernetes

Jun Kudo
5 min readJan 2, 2019

Introduction

In AkS and other kubernetes, if you are using fluentd to transfer to Elastic Search, you will get various logs when you deploy the formula.
However, because it sometimes wanted to acquire only the logs required due to the limitation of the disk on the Elastic Search side, I made it.
https://github.com/fluent/fluentd-kubernetes-daemonset

So we will explain how to get only Log necessary for configMap.
https://github.com/jkudo/fluentd-kubernetes-daemonset

configMap

fluentd-daemonset-elasticsearch-rbac-configmap.yaml

In case of deployment.apps/nginx-deploy, you want to get log of nginx.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
data:
  fluent.conf: |
    #@include systemd.conf
    @include kubernetes.conf
    <match **>
       @type elasticsearch
       @id out_es
       @log_level info
       include_tag_key true
       host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
       port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
       scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
       ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"
       user "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
       password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"
       reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'true'}"
       logstash_prefix "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash'}"
       logstash_format true
       type_name fluentd
       <buffer>
         flush_thread_count "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_THREAD_COUNT'] || '8'}"
         flush_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_INTERVAL'] || '5s'}"
         chunk_limit_size "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_CHUNK_LIMIT_SIZE'] || '2M'}"
         queue_limit_length "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_QUEUE_LIMIT_LENGTH'] || '32'}"
         retry_max_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_RETRY_MAX_INTERVAL'] || '30'}"
         retry_forever true
       </buffer>
    </match>
  kubernetes.conf: |
    <match fluent.**>
      @type null
    </match>
    #Target Logs (ex:nginx)
    <source>
      @type tail
      @id in_tail_container_logs
      path "/var/log/containers/nginx*.log"
      pos_file "/var/log/fluentd-nginx.log.pos"
      tag "kubernetes.*"
      read_from_head true
      <parse>
        @type "json"
        time_format "%Y-%m-%dT%H:%M:%S.%NZ"
        time_type string
      </parse>
    </source>
    #<source>
    #  @type tail
    #  @id in_tail_container_logs
    #  path "/var/log/containers/*.log"
    #  pos_file "/var/log/fluentd-{log name}.log.pos"
    #  tag "kubernetes.*"
    #  read_from_head true
    #  <parse>
    #    @type "json"
    #    time_format "%Y-%m-%dT%H:%M:%S.%NZ"
    #    time_type string
    #  </parse>
    #</source>
  #systemd.conf: |
  #  <source>
  #    @type systemd
  #    @id in_systemd_kubelet
  #    matches [{ "_SYSTEMD_UNIT": "kubelet.service" }]
  #    <storage>
  #      @type local
  #      persistent true
  #      path /var/log/fluentd-journald-kubelet-cursor.json
  #    </storage>
  #    <entry>
  #      fields_strip_underscores true
  #    </entry>
  #    read_from_head true
  #    tag kubelet
  #  </source>
  #  <source>
  #    @type systemd
  #    @id in_systemd_docker
  #    matches [{ "_SYSTEMD_UNIT": "docker.service" }]
  #    <storage>
  #      @type local
  #      persistent true
  #      path /var/log/fluentd-journald-docker-cursor.json
  #    </storage>
  #    <entry>
  #      fields_strip_underscores true
  #    </entry>
  #    read_from_head true
  #    tag docker.systemd
  #  </source>
  #  <source>
  #    @type systemd
  #    @id in_systemd_bootkube
  #    matches [{ "_SYSTEMD_UNIT": "bootkube.service" }]
  #    <storage>
  #      @type local
  #      persistent true
  #      path /var/log/fluentd-journald-bootkube-cursor.json
  #    </storage>
  #    <entry>
  #      fields_strip_underscores true
  #    </entry>
  #    read_from_head true
  #    tag bootkube
  #  </source>

If you want to acquire system logs, enable systemd.conf.

Pod

fluentd-daemonset-elasticsearch-rbac.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: fluentd
  namespace: kube-system
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - namespaces
  verbs:
  - get
  - list
  - watch

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      initContainers:
      - name: config-fluentd
        image: busybox
        imagePullPolicy: IfNotPresent
        command: ["/bin/sh","-c"]
        args:
        - cp /fluentd/etc2/fluent.conf /fluentd/etc/fluent.conf;
          cp /fluentd/etc2/kubernetes.conf /fluentd/etc/kubernetes.conf;
          #cp /fluentd/etc2/systemd.conf /fluentd/etc/systemd.conf;
        volumeMounts:
        - name: config-path
          mountPath: /fluentd/etc
        - name: config-source
          mountPath: /fluentd/etc2
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1.3.0-debian-elasticsearch
        env:
          - name:  FLUENT_ELASTICSEARCH_HOST
            value: "10.240.0.4"
          - name:  FLUENT_ELASTICSEARCH_PORT
            value: "9200"
          - name: FLUENT_ELASTICSEARCH_SCHEME
            value: "http"
          # X-Pack Authentication
          # =====================
          - name: FLUENT_ELASTICSEARCH_USER
            value: "elastic"
          - name: FLUENT_ELASTICSEARCH_PASSWORD
            value: "changeme"
          - name: FLUENT_UID
            value: "0"
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: config-path
          mountPath: /fluentd/etc
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: config-source
        configMap:
          name: fluentd-config
      - name: config-path
        emptyDir: {}

If you want to acquire system logs, enable systemd.conf.

log

If you check it, you can see that the target log is tailed.

2019-01-02 03:38:22 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2019-01-02 03:38:22 +0000 [info]: [out_es] Connection opened to Elasticsearch cluster => {:host=>"10.240.0.4", :port=>9200, :scheme=>"http", :user=>"elastic", :password=>"obfuscated"}
2019-01-02 03:38:22 +0000 [info]: [out_es] Detected ES 6.x: ES 7.x will only accept `_doc` in type_name.
2019-01-02 03:38:22 +0000 [info]: using configuration file: <ROOT>
  <match fluent.**>
    @type null
  </match>
  <source>
    @type tail
    @id in_tail_container_logs
    path "/var/log/containers/nginx*.log"
    pos_file "/var/log/fluentd-nginx.log.pos"
    tag "kubernetes.*"
    read_from_head true
    <parse>
      @type "json"
      time_format "%Y-%m-%dT%H:%M:%S.%NZ"
      time_type string
    </parse>
  </source>
  <match **>
    @type elasticsearch
    @id out_es
    @log_level "info"
    include_tag_key true
    host "10.240.0.4"
    port 9200
    scheme http
    ssl_verify true
    user "elastic"
    password xxxxxx
    reload_connections true
    logstash_prefix "logstash"
    logstash_format true
    type_name "fluentd"
    <buffer>
      flush_thread_count 8
      flush_interval 5s
      chunk_limit_size 2M
      queue_limit_length 32
      retry_max_interval 30
      retry_forever true
    </buffer>
  </match>
</ROOT>
2019-01-02 03:38:22 +0000 [info]: starting fluentd-1.3.0 pid=5 ruby="2.3.3"
2019-01-02 03:38:22 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby2.3", "-Eascii-8bit:ascii-8bit", "/fluentd/vendor/bundle/ruby/2.3.0/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--gemfile", "/fluentd/Gemfile", "--under-supervisor"]
2019-01-02 03:38:23 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '2.11.11'
2019-01-02 03:38:23 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.1.4'
2019-01-02 03:38:23 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.1.0'
2019-01-02 03:38:23 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.1'
2019-01-02 03:38:23 +0000 [info]: gem 'fluentd' version '1.3.0'
2019-01-02 03:38:23 +0000 [info]: adding match pattern="fluent.**" type="null"
2019-01-02 03:38:23 +0000 [info]: adding match pattern="**" type="elasticsearch"
2019-01-02 03:38:23 +0000 [info]: #0 [out_es] Connection opened to Elasticsearch cluster => {:host=>"10.240.0.4", :port=>9200, :scheme=>"http", :user=>"elastic", :password=>"obfuscated"}
2019-01-02 03:38:23 +0000 [info]: #0 [out_es] Detected ES 6.x: ES 7.x will only accept `_doc` in type_name.
2019-01-02 03:38:23 +0000 [info]: adding source type="tail"
2019-01-02 03:38:23 +0000 [info]: #0 starting fluentd worker pid=9 ppid=5 worker=0
2019-01-02 03:38:23 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/nginx-deploy-54d77946c9-cmjj7_default_nginx-9b6f8b66d16d9967d7e2fc019918ab36300ddb42b7d1b741bec4436003ca3212.log
2019-01-02 03:38:23 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/nginx-deploy-54d77946c9-cmjj7_default_nginx-c4bb67ea7576473de695a9cb0add2c63a1d6ffac604ae5556d5a3c360836c710.log
2019-01-02 03:38:23 +0000 [info]: #0 fluentd worker is now running worker=0

These constitute yaml with reference to Issue et al.

Summary

It is possible to use fluentd flexibly even with kubernetes by setting configMap.
I targeted Elastic Search this time, but I think that it can be used with other DeamonSets.

Original Content (Japanese) : http://level69.net/archives/26196

Fluentd
Kubernetes

--

--

Jun Kudo

Written by Jun Kudo

146 Followers

I’m Japanese, Microsoft MVP for Azure. Alibaba Cloud MVP. . https://twitter.com/jkudo

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams