Sensu补救-重启失败的受监控进程

Question

我们使用sensu来监控安装了sensu客户端的远程服务器上的某些进程。

当sensu获取信息失败时，有没有办法启动被监控进程的重启？我在网上找到了一些关于补救处理程序的信息

http://thesoftjaguar.com/posts/2015/06/14/sensu-remediation/

http://dev.nuclearrooster.com/2013/07/27/remediation-with-sensu/

但这似乎对我不起作用，补救措施永远不会触发：

另外，有两个ruby脚本应该会对此有所帮助，但不确定使用哪一个：

https://github.com/sensu-plugins/sensu-plugins-sensu/blob/master/bin/handler-sensu.rb

https://github.com/nstielau/sensu-community-plugins/blob/remediation/handlers/remediation/sensu.rb

4月20日更新：

我们使用的是sensu enterprise。同时，我设法让remediator.rb脚本被调用，但它不能正常工作，因为它不能从客户端读取JSON响应，并且有以下异常：

{"timestamp":"2017-04-20T03:06:41.733000-0700","level":"error","message":"handler output","handler":{"command":"/etc/sensu/plugins/remediator.rb","type":"pipe","timeout":10,"severities":["critical","warning","unknown"],"name":"remediator"},"event":{"id":"f38cd413-575a-46f6-8845-09d713a29815"},"output":["/opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-plugin/utils.rb:54:in `[]': no implicit conversion of String into Integer (TypeError)\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-plugin/utils.rb:54:in `block in deep_merge'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-plugin/utils.rb:52:in `each'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-plugin/utils.rb:52:in `deep_merge'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-plugin/utils.rb:22:in `block in settings'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-plugin/utils.rb:22:in `each'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-plugin/utils.rb:22:in `reduce'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-plugin/utils.rb:22:in `settings'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:123:in `api_settings'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:131:in `api_request'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:179:in `stash_exists?'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:191:in `block (2 levels) in filter_silenced'\n\tfrom /opt/sensu/embedded/lib/ruby/2.3.0/timeout.rb:91:in `block in timeout'\n\tfrom /opt/sensu/embedded/lib/ruby/2.3.0/timeout.rb:33:in `block in catch'\n\tfrom /opt/sensu/embedded/lib/ruby/2.3.0/timeout.rb:33:in `catch'\n\tfrom /opt/sensu/embedded/lib/ruby/2.3.0/timeout.rb:33:in `catch'\n\tfrom /opt/sensu/embedded/lib/ruby/2.3.0/timeout.rb:106:in `timeout'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:190:in `block in filter_silenced'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:188:in `each'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:188:in `filter_silenced'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:36:in `filter'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:80:in `block in <class:Handler>'\nwarning: event filtering in sensu-plugin is deprecated, see http:// bit.ly/sensu-plugin\n"]}

我们使用以下脚本作为remediator.rb https://github.com/sensu-plugins/sensu-plugins-sensu/blob/master/bin/handler-sensu.rb

检查执行正常，我们从sensu客户端服务器获得了响应，但看起来remediator.rb无法处理它。

配置如下：

remediator.json
{
  "handlers": {
    "remediator": {
      "command": "/etc/sensu/plugins/remediator.rb",
      "type": "pipe",
      "timeout": 10,
      "severities": ["critical", "warning", "unknown"]
    }
  }
}

使其尽可能简单，以便检查：

/etc/sensu/conf.d/checks
{
  "checks": {
    "seyren_check": {
      "command": "/opt/sensu/embedded/bin/ruby /etc/sensu/plugins/check-procs.rb -p unexisent_process",
      "interval": 30,
      "subscribers": ["trep"],
      "handlers": ["remediator","default","file"],
      "occurrences": 1,
      "refresh": 10,
      "remediation": {
        "first_remediation": {
          "occurrences": [1, 2],
          "severities": [1]
        },
        "medium_remediation": {
          "occurrences": ["3-10"],
          "severities": [1]
        },
        "heavy_remediation": {
          "occurrences": ["1+"],
          "severities": [2]
        }
      }
    },
    "first_remediation": {
      "command": "touch /etc/sensu/plugins/test_lr",
      "subscribers": ["my.machine.local"],
      "handlers": ["default"],
      "interval": 10,
      "publish": false
    },
    "medium_remediation": {
      "command": "touch /etc/sensu/plugins/test_mr",
      "subscribers": ["my.machine.local"],
      "handlers": ["default"],
      "interval": 10,
      "publish": false
    },
    "heavy_remediation": {
      "command": "touch /etc/sensu/plugins/test_hr",
      "subscribers": ["my.machine.local"],
      "handlers": ["default"],
      "interval": 10,
      "publish": false
    }
  }
}

Answer 1

• 我们更喜欢在这里使用一个名为Ansible的CM工具。下面是处理程序和基本思想：

 #! /usr/bin/env ruby

require 'sensu-handler'
    require 'json'

    class Ansible < Sensu::Handler
  def handle
    ansible = settings['ansible']['command'] || 'ansible-playbook'
    playbook = settings['ansible']['playbook'] || nil
    extra_vars = JSON.generate(@event)

    unless @event['check']['ansible'].nil?
      playbook = @event['check']['ansible']['playbook'] || playbook
    end

    command = ansible.to_s playbook.to_s
    output = `#{command}`

    if $?.exitstatus > 0
      puts output
      exit 1
    else
      puts "SUCCESS: #{command}"
    end
  end
end

然后创建

• ，处理程序配置：

{“handler”：{ "ansible"：{ "type"：“管道”，"command"：“/etc/sensu/handler/handler-ansible.rb”}

和补救行动手册的设置：

cat conf.d/handler/config_ansible.json 
{
    "ansible": {
        "command": "/etc/sensu/scripts/provision",
        "playbook": " --tags checksum"
    }
}

• ，并在您的检查配置中添加修正处理程序的名称：

{ ...

"handlers"："email"，"ansible"，"logstash“....}命令所在位置的

• ：

ssh root@my_ansible_server.comp.com -o BatchMode=yes -o "StrictHostKeyChecking no“-o ConnectTimeout=10 ansible-playbook -i /etc/ansible/generic.hosts /etc/ansible/remediation.yaml ${@}