팁 & 트릭

Output to Elasticsearch with Rsyslog in Logstash format

This post is just a reminder of how to ship logs directly from rsyslog to Elasticsearch so we have to go trough Logstash.

We got allot of help from the people on the rsyslog mailing list and for me the following is working pretty good so far.

Our config:

module(load="imfile")
module(load="mmjsonparse")
module(load="omelasticsearch")

template(name="logstash-index"
  type="list") {
    constant(value="logstash-")
    property(name="timereported" dateFormat="rfc3339" position.from="1" position.to="4")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="6" position.to="7")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="9" position.to="10")
}

input(type="imfile" File="/var/log/nginx/access.json"
      Tag="nginxulyaoth"
      PersistStateInterval="10000"
      StateFile="nginxulyaoth"
      Severity="info"
      MaxSubmitAtOnce="20000"
      Facility="user"
      Ruleset="nginxrule")

template(name="uly-nginx" type="list") {
  constant(value="{")
  constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
  constant(value="\",\"host\":\"")        property(name="hostname")
  constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
  constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
  constant(value="\",\"tag\":\"")         property(name="syslogtag" format="json")
  constant(value="\",\"message\":\"")     property(name="msg" format="json")
  constant(value="\"}")
}

ruleset(name="nginxrule") {
  action(type="mmjsonparse" name="jsonparse")
  action(type="omelasticsearch"
  server="loghost.ulyaoth.net"
  serverport="9200"
  template="uly-nginx"
  searchIndex="logstash-index"
  dynSearchIndex="on"
  errorFile="/var/log/rsyslog/ES-error.log")
}

We are still working on getting the json log split into variables so it will show more nice in Kibana.

Related posts

How to create a SELinux policy module file

Sjir Bagmeijer

Rack::OnePiece

Sjir Bagmeijer

MacBook Air (mid 2011) bootcamp Windows 8.1 enterprise wifi fix

Sjir Bagmeijer

이 웹사이트에서는 이용 환경 개선을 위해 쿠키를 사용합니다. 귀하께서 쿠키의 사용에 동의하시는 것으로 간주할 예정입니다만, 원하실 경우 철회를 하실 수 있습니다. 수락 더 읽어보기