If we want to send container logs using Fluent Bit, from self-managed Kubernetes cluster pods to AWS S3 bucket. That is possible now because from Oct 2020 the Fluent Bit allows AWS S3 as a destination to route container logs. Lets see how to setup Fluent Bit to send logs to S3 bucket.
Follow the below steps to send records into Amazon S3-
Create a new Namespace for the FluentBit in your k8s cluster,
1. Make a namespace (fluentbit-namespace.yaml)
apiVersion: v1
kind: Namespace
metadata:
name: fluentbit
2. Make a ClusterRole (cluster-role.yaml).
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: fluent-bit-read
namespace: fluentbit
rules:
- apiGroups: ["", "extensions", "apps"]
resources: ["deployments", "pods", "namespaces"]
verbs: ["get", "list", "watch"]
3. Make ServiceAccount (service-account.yaml). Give your service account and namespace name.
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluent-bit
namespace: fluentbit
4. Make ClusterRoleBinding (cluster-role-binding.yaml). Give your serviceaSccount and namespace name.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: fluent-bit-read
namespace: fluentbit
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: fluent-bit-read
subjects:
- kind: ServiceAccount
name: fluent-bit
namespace: fluentbit
5. Make Configmap for Fluent Bit (config-map.yaml). Update namespace and S3 bucket name.
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit
namespace: fluentbit
labels:
app: fluent-bit
data:
fluent-bit.conf: |
[SERVICE]
Parsers_File parsers.conf
@INCLUDE input-*.conf
@INCLUDE output-*.conf
@INCLUDE filters.conf
parsers.conf: |
[PARSER]
Name s3_logs
Format json
Time_Key requestReceivedTimestamp
Time_Format %Y-%m-%dT%H:%M:%S.%LZ
Time_Keep On
[PARSER]
Name kubernetes-tag
Format regex
Regex (?.+)\.(?.+)\.(?.+)
input-s3logs.conf: |
[INPUT]
Name tail
Tag kube...
Tag_Regex (?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?[^_]+)_(?.+)-
Path /var/log/containers/*.log
Exclude_Path /var/log/containers/*_fluentbit_*.log
Parser s3_logs
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
output-s3.conf: |
[OUTPUT]
Name s3
Match *
bucket /fluentbit-s3-bucket/
region us-west-2
use_put_object On
s3_key_format /$TAG[1]/$TAG[3]/%Y/%m/%d/
s3_key_format_tag_delimiters .
total_file_size 5M
upload_timeout 1m
[OUTPUT]
Name stdout
Match *
filters.conf: |
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix kube.
Merge_Log On
Merge_Log_Key log_processed
K8S-Logging.Parser On
K8S-Logging.Exclude On
Regex_Parser kubernetes-tag
Labels Off
Annotations Off
6. Make Daemonset file (daemon-set.yaml). Update namespace name, secretKeyRef name.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluent-bit
namespace: fluentbit
labels:
k8s-app: fluent-bit
version: v1
kubernetes.io/cluster-service: "true"
spec:
selector:
matchLabels:
app: fluent-bit
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
template:
metadata:
labels:
app: fluent-bit
k8s-app: fluent-bit
version: v1
kubernetes.io/cluster-service: "true"
spec:
containers:
- env:
- name: AWS_REGION
value: ap-southeast-2
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: fluentbit-s3-key
key: AWS_ACCESS_KEY_ID
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: fluentbit-s3-key
key: AWS_SECRET_ACCESS_KEY
image: 906394416424.dkr.ecr.ap-southeast-2.amazonaws.com/aws-for-fluent-bit:2.8.0
imagePullPolicy: Always
name: fluent-bit
volumeMounts:
- name: fluentbitconfigvol
mountPath: /etc/fluent-bit/conf/
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
command: ["/fluent-bit/bin/fluent-bit"]
args: ["-c", "/etc/fluent-bit/conf/fluent-bit.conf"]
imagePullSecrets:
- name: aws-registry
volumes:
- name: fluentbitconfigvol
configMap:
name: fluent-bit
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
7. Make a secret into Pod as environment variable. Give the name of the secret in daemonset file.
kubectl create secret generic fluentbit-s3-key --from-literal=AWS_ACCESS_KEY_ID= --from-literal=AWS_SECRET_ACCESS_KEY=
The above Fluent Bit DaemonSet will start sending the container logs to S3 bucket while run on Kubernetes cluster..
Comments (0)