Loki没有通知警报管理员

Question

我是Loki的新手，在Loki发出了警告，但是我在Alertmanager中没有看到任何通知。Loki工作正常(收集日志)，alertmanager (从其他来源获得警报)，但是来自loki的日志不会被推送到警报管理器。

Loki配置：

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 1h       # Any chunk not receiving new logs in this time will be flushed
  max_chunk_age: 1h           # All chunks will be flushed when they hit this age, default is 1h
  chunk_target_size: 1048576  # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first
  chunk_retain_period: 30s    # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
  max_transfer_retries: 0     # Chunk transfers disabled

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/boltdb-shipper-active
    cache_location: /loki/boltdb-shipper-cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks

compactor:
  working_directory: /loki/boltdb-shipper-compactor
  shared_store: filesystem

limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

ruler:
  storage:
    type: local
    local:
      directory: etc/loki/rules
  rule_path: /etc/loki/
  alertmanager_url: http://171.11.3.160:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true

码头工人-撰写Loki：

 loki:
    image: grafana/loki:2.0.0
    container_name: loki
    ports:
      - "3100:3100"
    volumes:
      - ./loki/etc/local-config.yaml:/etc/loki/local-config.yaml
      - ./loki/etc/rules/rules.yaml:/etc/loki/rules/rules.yaml
    command:
      -  '--config.file=/etc/loki/local-config.yaml'

Loki规则：

groups:
  - name: rate-alerting
    rules:
    - alert: HighLogRate
      expr: |
           count_over_time(({job="grafana"})[1m]) >=0
      for: 1m

有人知道问题出在哪里吗？

Answer 1

我终于让它起作用了。

下面是我的标尺配置

ruler:
  storage:
    type: local
    local:
      directory: /etc/loki/rulestorage
  rule_path: /etc/loki/rules
  alertmanager_url: http://alertmanager:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true
  enable_alertmanager_v2: true

在目录下面创建

• /etc/loki/rulestorage/fake
• /etc/loki/rules/fake
• Copied alert_rules.yaml在/etc/loki/rulestorage/fake
• Gave下对/etc/loki/rulestorage/fake

下loki用户的完全权限

轰隆

Answer 2

配置看起来很好，类似于我的配置。我将通过以下步骤排除它的故障：

1. Exec到容器，并检查规则文件是否为空

1. 检查loki的日志。当规则正确加载时，会弹出这样的日志：

level=info ts=2021-05-06T11:18:33.355446729Z caller=module_service.go:58 msg=initialising module=ruler
level=info ts=2021-05-06T11:18:33.355538059Z caller=ruler.go:400 msg="ruler up and running"
level=info ts=2021-05-06T11:18:33.356584674Z caller=mapper.go:139 msg="updating rule file" file=/data/loki/loki-stack-alerting-rules.yaml

运行时bytes=...):期间的

1. 还记录关于您的规则的信息消息(我将向您展示我正在运行的规则，但稍微缩短)(注意status=200和非空的
2. )。

level=info 
ts=... 
caller=metrics.go:83 
org_id=... 
traceID=... 
latency=fast 
query="sum(rate({component=\"kube-apiserver\"} |~ \"stderr F E.*failed calling webhook \\\"webhook.openpolicyagent.org\\\". an error on the server.*has prevented the request from succeeding\"[1m])) > 1" 
query_type=metric 
range_type=instant 
length=0s 
step=0s 
duration=9.028961ms 
status=200 
throughput=40MB 
total_bytes=365kB

然后，

1. 确保您可以从loki容器中访问警报管理器http://171.11.3.160:9093，而不存在任何问题(可能存在网络问题，或者您已经设置了基本的身份验证等)。

1. 如果您设置的规则(您可以从grafana探索窗口进行测试)将超过设置的阈值1分钟，则警报应该会出现在警报管理器中。它很可能是未分组的，因为您没有向它添加任何标签。