Syntax highlighting of e0d846b ~( devops/logs)

# Log Collector

## Loki

```yaml
# config.yaml
# ...
schema_config:
  configs:
    - from: 2022-10-01
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h
# ...
table_manager:
  retention_deletes_enabled: true
  retention_period: 168h
```

```yaml
# docker-compose.yaml
  loki:
    image: grafana/loki:2.3.0
    command: -config.file=/etc/loki/loki-config.yaml
    volumes:
      - ./loki/config.yaml:/etc/loki/loki-config.yaml
      - loki_data:/loki
    restart: unless-stopped
    expose:
      - 3100
    networks:
      - monitor-net
    labels:
      - "traefik.http.routers.loki.entrypoints=loki"
      - "traefik.http.routers.loki.rule=PathPrefix(`/`)"
      - "traefik.http.routers.loki.middlewares=auth"
```

## Fluentd

### Installation

 Before installation - https://docs.fluentd.org/installation/before-install

https://docs.fluentd.org/installation/install-by-deb

```bash
# Ubuntu 18
curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-bionic-td-agent3.sh | sh
sudo systemctl start td-agent.service
sudo systemctl status td-agent.service
```
Файлы конфигурации сервиса находятся в директории */etc/td-agent*

```bash
# --- testing
# console1
tail -f /var/log/td-agent/td-agent.log
# console2
$ curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test
# console1
# 2020-06-02 05:39:46.149078100 +0000 debug.test: {"json":"message"}
```

## FluentBit

### Installation

**Daemon**

https://docs.fluentbit.io/manual/
https://docs.fluentbit.io/manual/installation/linux/ubuntu

```bash
# 18.04
wget -qO - https://packages.fluentbit.io/fluentbit.key | sudo apt-key add -
```
```bash
# /etc/apt/sources.list
deb https://packages.fluentbit.io/ubuntu/bionic bionic main
```
```bash
sudo apt-get update
sudo apt-get install td-agent-bit
sudo  service td-agent-bit start
sudo  service td-agent-bit status
tail -f /var/log/syslog
```

Файлы конфигурации демона находятся в директории `/etc/td-agent-bit`

**Client**

```bash
sudo apt-get install cmake flex bison
# --- @see
#     https://fluentbit.io/download/
#     https://github.com/fluent/fluent-bit/releases
wget https://fluentbit.io/releases/1.4/fluent-bit-1.4.4.tar.gz
# --- testing
fluent-bit -i tail -p path=/var/log/syslog -o stdout
```

### Configuration

Создаем файл конфигурации: забираем все логи из докера и отправляем в stdout

```ini
# fluent.conf 
[INPUT]
    Name tail
    Path /var/lib/docker/containers/*/*.log
    Parser docker

[FILTER]
    Name         nest
    Match        *
    Operation    lift
    Nested_under attrs

[FILTER]
    Name   grep
    Match  *
    Regex  location ^pro.itmonitoring.dev*

[OUTPUT]
    Name stdout
    Match *
```
```ini
# parser.conf
[PARSER]
    Name        docker
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S
```
```bash
# используем sudo, т.к. по умолчанию доступа к директории с логами докера нет
sudo fluent-bit -c fluent.conf -R parser.conf
```

## Fluent-Bit Forward Fluentd

Конфигурация для встречной работы:

*fluentd*
```
<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match **>
  @type stdout
</match>
```

*fluent-bit*
```
[SERVICE]
...

[INPUT]
    Name tail
    storage.type  filesystem
    Path /var/lib/docker/containers/*/*.log
    Parser docker

[FILTER]
    Name         nest
    Match        *
    Operation    lift
    Nested_under attrs

[FILTER]
    Name   grep
    Match  *
    Regex  location ^pro.itmonitoring.dev*

[OUTPUT]
    Name          forward
    Match         *
    Host          127.0.0.1
    Port          24224
```


> Written with [StackEdit](https://stackedit.io/).