All notes
Syslog

Commands

syslog-ng


#---------- Debug related
# https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/chapter-troubleshooting-syslog-ng.html

# Check the configuration files for any syntax errors on both the client and the server using
syslog-ng --syntax-only
syslog-ng --syntax-only -f /etc/syslog-ng/syslog-ng.conf

# use tcpdump or a similar packet sniffer tool on the client to verify that the messages are sent correctly, and on the server to verify that it receives the messages.

# Run in foreground, and display debug messages
syslog-ng -Fevd

# Turn on --verbose or --debug command-line options for more-detailed log messages. You can enable these messages without restarting syslog-ng using:
syslog-ng-ctl verbose --set=on

syslog-ng-ctl


# https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/syslog-ng-ctl.1.html

syslog-ng-ctl verbose/trace/debug --set=on
syslog-ng-ctl stats
syslog-ng-ctl reload

# NOTE: this command needs a socket open:
syslog-ng-ctl stats
# Error connecting control socket, socket='/var/syslog-ng.ctl', error='No such file or directory'

logger



logger System rebooted
logger -p local0.notice -t HOSTIDM -f /dev/idmc
logger -n loghost.example.com System rebooted

# "-t rails-projname" sets the program name.
# "-p user.debug" sets the facility and priority level.
# https://help.papertrailapp.com/discussions/questions/96-how-to-log-a-message-from-the-linux-command-line.html
logger -p user.debug -t rails-projname "Hello world"

##### Common options.

-P, --port port
  Use the specified port.
-f, --file file
  Log the contents of the specified file.  This option cannot be combined with a command-line message.
-p, --priority priority
-t, --tag tag
  Mark every line to be logged with the specified tag. "tag" is usually the "program name" field.

##### Used in debugging.

-s, --stderr
  Output the message to standard error as well as to the system log.

##### Others

-T, --tcp
  Use stream (TCP) only.  By default the connection is tried to the syslog-conn port defined in /etc/services, which is often 601.

-d, --udp
  Use datagram (UDP) only.

-u, --socket socket
  Write to the specified socket instead of to the builtin syslog routines.

-n, --server server
  Write  to the specified remote syslog server instead of to the builtin syslog routines.  Unless --udp or --tcp is specified, logger will first try to use UDP, but if thist fails a TCP connection is attempted.

Log buffer

 # https://stackoverflow.com/questions/208098/can-syslog-performance-be-improved
You can configure syslogd (and rsyslog at least) not to sync the log files after a log message by prepending a "-" to the log file path in the configuration file. This speeds up performance at the expense of the danger that log messages could be lost in a crash.

wcfNote: if you just want to debug for syslog.conf, use "logger -s" option instead.

Macro

balabit.com: syslog-ng macros.

FACILITY
  The name of the facility (for example, kern) that sent the message.
FACILITY_NUM
  The numerical code of the facility (for example, 0) that sent the message.
LEVEL
  The priority of the message, for example, error. For the textual representation of this value, use the ${LEVEL} macro.

Global options

balabit.com: reference options.

ts_format

Accepted values:	rfc3164 | bsd | rfc3339 | iso
Default:	rfc3164

RFC 3164
1987-06-18T15:20:30.337Z

The stackExchange states that:

RFC3164 does not include a year in the timestamp portion of a log entry.
RFC5424 is supposed to make RFC3164 obsolete.
When you eventually switch to systemd (resistance is futile), you can use journalctl's -o short-iso option to get real ISO 8601 timestamps.

Statements

destination

Files under /var/log/

askUbuntu.com.

According to /etc/syslog.conf, default /var/log/kern.log captures only the kernel's messages of any loglevel; i.e. the output of dmesg.

/var/log/messages instead aims at storing valuable, non-debug and non-critical messages. This log should be considered the "general system activity" log.

/var/log/syslog in turn logs everything, except auth related messages.

Other insteresting standard logs managed by syslog are /var/log/auth.log, /var/log/mail.log.

Regarding your question: if you need solely kernel messages log, use the kern.log or call dmesg.

dmesg vs /var/log/messages

unix.stackExchange.com: dmesg vs /var/log/messages.

dmesg prints the contents of the ring buffer. This information is also sent in real time to syslogd or klogd, when they are running, and ends up in /var/log/messages; when dmesg is most useful is in capturing boot-time messages from before syslogd and/or klogd started

filter

balabit.com: filter's functions.

Wildcards in filters

balabit.com: syslog-ng regexp.

The host(), match(), and program() filter functions accept regular expressions as parameters. The exact type of the regular expression to use can be specified with the type() option. By default, syslog-ng OSE uses PCRE regular expressions.

message("^(.+)\\1$" type("posix"))

# If you do not need regular expressions, only wildcards, use type(glob):
filter f_wildcard {host("myhost*" type(glob));};

Combine filters

balabit.com.


filter demo_filter { not host("example1") and not host("example2"); };
# Alternatively, you can use parentheses to avoid this confusion:
filter demo_filter { not (host("example1") or host("example2")); };

facility

It accepts both the name and the numerical code of the facility or the importance level. Facility codes 0-23 are predefined.

facility(user)
facility(1)
# range with facility names
facility(local0..local5)

local facility

lists.balabit.hu.



source s_udp  {
  udp( ip(0.0.0.0) port(514) );
};

destination d_application { file("/var/log/application.log"); };
destination d_syslog { file("/var/log/syslog"); };

filter f_my_servers { host(h-001) or host(h-002) or host(h-003); };
filter f_test1  { facility(local5) and filter(f_my_servers); };
filter f_test2 { filter(f_my_servers); };

log { source(s_udp); filter(f_test1); destination(d_application);
flags(final); };
log { source(s_udp); filter(f_test2); destination(d_syslog); flags(final);
};

# "logger -p local5.info test" outputs to /var/log/application.log.

program

Match messages by using a regular expression against the program name field of log messages.

For example, "rails-myproj-3586701d6057" here is the program field:

2017-10-11T05:13:00+00:00 10.116.29.171 [user:debug] rails-myproj-3586701d6057[java-sdk-http-connection-reaper] org.apache.http.impl.conn.PoolingHttpClientConnectionManager Closing connections idle longer than 60 SECONDS

priority or level

The level() filter accepts the following levels: emerg, alert, crit, err, warning, notice, info, debug.

source

balabit.com: sources.


# Receives messages on the 1999 TCP port and the 1999 UDP port of the interface having the 10.1.2.3 IP address.
source s_demo_two_drivers {
  network(ip(10.1.2.3) port(1999));
  network(ip(10.1.2.3) port(1999) transport("udp"));
};

source s_demo {
  internal(); # Messages generated by syslog-ng. wcfNote: receives messages from klogd(8)? See "logging in Linux" below.
  network(transport("udp")); # Messages arriving to the 514/UDP port of any interface of the host.
  unix-dgram("/dev/log"); # Messages arriving to the /dev/log socket. You may use "unix-stream()" for stream socket e.g. tcp.
};

# Setting default priority and facility
# If the message received does not have a proper syslog header, it will use the default settings.
source headerless_messages { network(default-facility(syslog) default-priority(emerg)); };

Concepts

Logging in Linux

unix.stackExchange.com: understanding logging in Linux.

The kernel logs messages (using the printk() function) to a ring buffer in kernel space. These messages are made available to user-space applications in two ways:

There are two main applications that read (and, to some extent, can control) the kernel's ring buffer:

In user space, there's syslogd(8). It has three sources:

It then writes these messages to some files in /log, or to named pipes, or sends them to some remote hosts (via the syslog protocol, on UDP port 514).

User-space applications normally use the libc function syslog(3) to log messages. libc sends these messages to the UNIX domain socket /dev/log (where they are read by syslogd(8)).

Other daemons (such as rsyslog and syslog-ng, as you mention) can replace the plain syslogd(8), and do all sorts of nifty things, like send messages to remote hosts via encrypted TCP connections, provide high resolution timestamps, and so on. And there's also systemd, that is slowly phagocytosing the UNIX part of Linux.

systemd has its own logging mechanisms (askUbuntu.com: /dev/log is missing):


# Managed by systemd-journald.service, "Symlinks=/dev/log", similar to:
sudo ln -s /run/systemd/journal/dev-log /dev/log