Syntax highlighting of e0d846b ~( devops/logs)

= Logs =

<<TableOfContents()>>

== Loki ==

{{{#!highlight yaml
# config.yaml
# ...
schema_config:
  configs:
    - from: 2022-10-01
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h
# ...
table_manager:
  retention_deletes_enabled: true
  retention_period: 168h

}}}

{{{#!highlight yaml
# docker-compose.yaml
  loki:
    image: grafana/loki:2.3.0
    command: -config.file=/etc/loki/loki-config.yaml
    volumes:
      - ./loki/config.yaml:/etc/loki/loki-config.yaml
      - loki_data:/loki
    restart: unless-stopped
    expose:
      - 3100
    networks:
      - monitor-net
    labels:
      - "traefik.http.routers.loki.entrypoints=loki"
      - "traefik.http.routers.loki.rule=PathPrefix(`/`)"
      - "traefik.http.routers.loki.middlewares=auth"

}}}

== Fluentd ==

=== Installation ===

Before installation - https://docs.fluentd.org/installation/before-install

[[ https://docs.fluentd.org/installation/install-by-deb ]]

{{{#!highlight bash
# Ubuntu 18
curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-bionic-td-agent3.sh | sh
sudo systemctl start td-agent.service
sudo systemctl status td-agent.service

}}}

Файлы конфигурации сервиса находятся в директории ''/etc/td-agent''

{{{#!highlight bash
# --- testing
# console1
tail -f /var/log/td-agent/td-agent.log
# console2
$ curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test
# console1
# 2020-06-02 05:39:46.149078100 +0000 debug.test: {"json":"message"}

}}}

== FluentBit ==

=== Installation ===

'''Daemon'''

[[https://docs.fluentbit.io/manual/]]
[[https://docs.fluentbit.io/manual/installation/linux/ubuntu]]

{{{#!highlight bash
# 18.04
wget -qO - https://packages.fluentbit.io/fluentbit.key | sudo apt-key add -

}}}

{{{#!highlight bash
# /etc/apt/sources.list
deb https://packages.fluentbit.io/ubuntu/bionic bionic main

}}}

{{{#!highlight bash
sudo apt-get update
sudo apt-get install td-agent-bit
sudo  service td-agent-bit start
sudo  service td-agent-bit status
tail -f /var/log/syslog

}}}

Файлы конфигурации демона находятся в директории `/etc/td-agent-bit`

'''Client'''

{{{#!highlight bash
sudo apt-get install cmake flex bison
# --- @see
#     https://fluentbit.io/download/
#     https://github.com/fluent/fluent-bit/releases
wget https://fluentbit.io/releases/1.4/fluent-bit-1.4.4.tar.gz
# --- testing
fluent-bit -i tail -p path=/var/log/syslog -o stdout

}}}

=== Configuration ===

Создаем файл конфигурации: забираем все логи из докера и отправляем в stdout

{{{
# fluent.conf 
[INPUT]
    Name tail
    Path /var/lib/docker/containers/*/*.log
    Parser docker

[FILTER]
    Name         nest
    Match        *
    Operation    lift
    Nested_under attrs

[FILTER]
    Name   grep
    Match  *
    Regex  location ^pro.itmonitoring.dev*

[OUTPUT]
    Name stdout
    Match *

}}}

{{{
# parser.conf
[PARSER]
    Name        docker
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S

}}}

{{{#!highlight bash
# используем sudo, т.к. по умолчанию доступа к директории с логами докера нет
sudo fluent-bit -c fluent.conf -R parser.conf

}}}

== Fluent-Bit Forward Fluentd ==

Конфигурация для встречной работы:

''fluentd''

{{{
<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match **>
  @type stdout
</match>

}}}

''fluent-bit''

{{{
[SERVICE]
...

[INPUT]
    Name tail
    storage.type  filesystem
    Path /var/lib/docker/containers/*/*.log
    Parser docker

[FILTER]
    Name         nest
    Match        *
    Operation    lift
    Nested_under attrs

[FILTER]
    Name   grep
    Match  *
    Regex  location ^pro.itmonitoring.dev*

[OUTPUT]
    Name          forward
    Match         *
    Host          127.0.0.1
    Port          24224

}}}

== Logrotate ==

[[https://debianworld.ru/articles/rotaciya-logov-s-pomoshyu-logrotate-v-debian-ubuntu/]]

Создаем файл конфигурации для ротации

{{{
# sudo vim /etc/logrotate.d/ethereum-ruby-http
/tmp/ethereum_ruby_http.log {
	daily
	missingok
	rotate 7
	compress
	nodelaycompress
	notifempty
	su
}
}}}

Пример конфигурации для Rails-приложения
{{{
# /etc/logrotate.d/gosplan-api
/var/www/gosplan-api/current/log/*.log {
  daily
  missingok
  rotate 7
  compress
  delaycompress
  notifempty
  copytruncate
}
}}}

Тестовый запуск

{{{#!highlight bash
sudo logrotate -d /etc/logrotate.d/gosplan-api
}}}

Запуск ротации

{{{#!highlight bash
sudo logrotate -v -f /etc/logrotate.d/gosplan-api
}}}

Далее ротация будет запускаться автоматически в соответствии с файлом конфигурации.