Grok

As every application has one to multiple log files, which have their own format, it is important to make these format indexable for elastic search. This can be done by creating own filter and filter criteria. Grok provides such an functionality and uses the following syntax:

%{PATTERN:IDENTIFIER}

Once again digitalocean.com has a nice article how to do this.

Pattern Configuration

The pattern is referenced from Groks repository. The repository already contains default parameters for usual cases, such as words or numbers. A typical parameter has the following format:

Reference_Name Regular_Expression

You can create your own pattern for your application and attach them to Grok:

# sudo mkdir -p /opt/logstash/patterns
# sudo chown logstash: /opt/logstash/patterns
# sudo vim /opt/logstash/patterns/myLogPattern
DATETIME [0-9]{4}[\-][0-9]{2}[\-][0-9]{2}[ \t]+[0-9]{2}[\:][0-9]{2}[\:][0-9]{2}
# sudo chown logstash: /opt/logstash/patterns/myLogPattern
# sudo service logstash restart

For writing regular expressions some tools – such as RegExr.com or Grok Pattern Tester –  might be very useful.

Identifier Configuration

For using the created pattern, you can edit the logtash filter configuration and reference your own or existing Grok patterns:

# sudo vim /etc/logstash/conf.d/11-myLogPattern-filter.conf
filter {
    if [type] == "app-access" {
        grok {
            match => { "message" => "%{WORD:user}%{SPACE}%{DATETIME:Datetime}%{SPACE}%{BASE16FLOAT:Duration}%{SPACE}%{INT:Bytes}%{SPACE}%{IPV4:ip_source}%{SPACE}%{IPORHOST:ip_destination}:%{POSINT:ip_port}%{SPACE}%{HOSTNAME:host_source}%{SPACE}%{HOSTNAME:host_destination}%{SPACE}%{WORD:method}%{SPACE}%{URIPATH:request}%{SPACE}%{NUMBER:statuscode}" }
        }
        syslog_pri {}
        date {
            match => [ "Datetime", "YYYY-MM-dd HH:mm:ss", "MM dd HH:mm:ss" ]
        }
    }
}
# sudo service logstash configtest
Configuration OK
# sudo service logstash restart

In case you experience errors, please check /var/log/syslog if logstash is able to start up propably. If data is transmitted you should check if data is transmitted successfully:

monitoringRepository/2017-04-18%20Kibana%20succssfull%20JSON%20parsing.PNG

In case of Grok Pattern errors you can see a JSON parsing tag (_grokparsefailure); for resolution you can check for forums, such as discussion.elastic.co. In this case the IP address is invalid; please keep in mind that you might have a restriction for your own patterns, e.g. HOSTNAME could overwrite the general hostname pattern:

2017-04-18%20Kibana%20JSON%20grok%20failure.PNG

Filebeat adjustments

The configuration for an input of type “app-access” requires, that the agent – which is Filebeat in my case –  provides the appropriate data. The document_type referenced here must also be defined within /etc/logstash/conf.d/10-syslog-filter.conf:

# sudo vi /etc/filebeat/filebeat.yml
################### Filebeat Configuration Example #########################
############################# Filebeat ######################################
filebeat:
  # List of prospectors to fetch data.
  prospectors:
    # Each - is a prospector. Below are the prospector specific configurations
    -
      # Paths that should be crawled and fetched. Glob based paths.
      # To fetch all ".log" files from a specific level of subdirectories
      # /var/log//.log can be used.
      # For each file found under this path, a harvester is started.
      # Make sure not file is defined twice as this can lead to unexpected behaviour.
      paths:
        - /home/fabiansc/Dokumente/
      document_type: app-access
        - /var/log/auth.log
        - /var/log/syslog
       # - /var/log/.log
       # sudo service filebeat restart

Please note that only spaces and not tabulators are used; also the amount of spaces are important; document_type must have the same depth as paths, otherwise filebeat cannot start up due to errors in /var/log/syslog:

Apr  2 15:09:15 ubuntu systemd[1]: filebeat.service: Main process exited, code=exited, status=1/FAILURE
Apr  2 15:09:15 ubuntu systemd[1]: filebeat.service: Unit entered failed state.
Apr  2 15:09:15 ubuntu systemd[1]: filebeat.service: Failed with result 'exit-code'.
Apr  2 15:09:15 ubuntu systemd[1]: filebeat.service: Service hold-off time over, scheduling restart.
Apr  2 15:09:15 ubuntu systemd[1]: Stopped filebeat.
Apr  2 15:09:15 ubuntu systemd[1]: Started filebeat.
Apr  2 15:09:15 ubuntu filebeat[3494]: Loading config file error: YAML config parsing failed on /etc/filebeat/filebeat.yml: yaml: line 15: did not find exp
ected '-' indicator. Exiting.