Actions 动作

The Action object describe what is to be done with a message. They are implemented via output modules.

The action object has different parameters:

• those that apply to all actions and are action specific. These are documented below.
• action队列的参数会应用到所有 action，参考 queue parameters.
• action-specific parameters. These are specific to a certain type of actions. They are documented by the output modules in question.

General Action Parameters

大小写不敏感

• name word

  This names the action. The name is used for statistics gathering and documentation. 如果没有名字，rsyslog会运达产生一个，编号从1ton

• type string

  强制参数， The name of the module that should be used.

• action.writeAllMarkMessages on/off

  默认on，每20分钟会发送mark消息给action，无论之前这个action是否被执行过，可以认为是一种心跳。一般设为on

• action.execOnlyEveryNthTime integer

  每隔一定的消息执行一次action，比如设为3，第1，2条消息会被丢弃，执行第3条，如是反复

• action.execOnlyEveryNthTimeout integer

  每隔一定的时间执行一次action。0表示不启用。在消息间隔很长时，会很有用，

• action.errorfile string

  New in version 8.32.0.

  一条消息由于某些原因执行失败，默认就被丢弃了。

  如果设置了本项，可以输出到指定文件。格式为JSON，包括失败的消息，action的名字，错误码，及错误原因。

  action.execOnlyOnceEveryInterval integer

  在最后一次执行后指定时间执行Execute action only if the last execute is at last seconds in the past (more info in ommail, but may be used with any action)

• action.execOnlyWhenPreviousIsSuspended on/off

  设置为on时，当前一个action被挂起时，才会执行。比如用来自动切换目的地址，当主服务器失败后，发送到备用服务器

• action.repeatedmsgcontainsoriginalmsg on/off

  如果消息重复了，会提示“last message repeated n times” messages, 比较消息的n个字符，n至少为80

• action.resumeRetryCount integer

  [default 0, -1 means eternal]

  Sets how often an action is retried before it is considered to have failed. Failed actions discard messages.

• action.resumeInterval integer

  重试间隔，单位s。为防止在多消耗，每10次失败后，这个间隔扩大， (numRetries / 10 + 1) * action.resumeInterval. 比如30的间隔，在10次失败后会变成60，100次失败后变成 330

• action.resumeIntervalMax integer

  Default: 1800 (30 minutes)

  重试间隔最大值

  action.reportSuspension on/off

  Configures rsyslog to report suspension and reactivation of the action. This is useful to note which actions have problems (e.g. connecting to a remote system) and when. The default for this setting is the equally-named global parameter.

• action.reportSuspensionContinuation on/off

  Configures rsyslog to report continuation of action suspension. This emits new messages whenever an action is to be retried, but continues to fail. If set to “on”, action.reportSuspension is also automatically set to “on”. The default for this setting is the equally-named global parameter.

• action.copyMsg on/off

  Configures action to copy the message if on. Defaults to off (which is how actions have worked traditionally), which causes queue to refer to the original message object, with reference-counting. (Introduced with 8.10.0).

Useful Links

• Rainer’s blog posting on the performance of main and action queue worker threads

Legacy Format

Legacy action很能写正确，有可能的话， A key problem with legacy format is that a single action is defined via multiple configurations lines, which may be spread all across rsyslog.conf. Even the definition of multiple actions may be intermixed (often not intentional!). If legacy actions format needs to be used (e.g. some modules may not yet implement the RainerScript format), it is strongly recommended to place all configuration statements pertaining to a single action closely together.

Please also note that legacy action parameters do not affect RainerScript action objects. So if you define for example:

1
2
3

$actionResumeRetryCount 10
action(type="omfwd" target="server1.example.net")
@@server2.example.net

server1’s “action.resumeRetryCount” parameter is not set, instead server2’s is!

A goal of the new RainerScript action format was to avoid confusion which parameters are actually used. As such, it would be counter-productive to honor legacy action parameters inside a RainerScript definition. As result, both types of action definitions are strictly (and nicely) separated from each other. The bottom line is that if RainerScript actions are used, one does not need to care about which legacy action parameters may (still…) be in effect.

Note that not all modules necessarily support legacy action format. Especially newer modules are recommended to NOT support it.

Legacy Description

Templates can be used with many actions. If used, the specified template is used to generate the message content (instead of the default template). To specify a template, write a semicolon after the action value immediately followed by the template name. Beware: templates MUST be defined BEFORE they are used. It is OK to define some templates, then use them in selector lines, define more templates and use use them in the following selector lines. But it is NOT permitted to use a template in a selector line that is above its definition. If you do this, the action will be ignored.

可以对一个selector使用多个actions (or more precisely a single filter of such a selector line). 每一个action必须单独一行，以&打头

1
2
3

*.=crit :omusrmsg:rger
& root
& /var/log/critmsgs

These three lines send critical messages to the user rger and root and also store them in /var/log/critmsgs. Using multiple actions per selector is convenient and also offers a performance benefit. As the filter needs to be evaluated only once, there is less computation required to process the directive compared to the otherwise-equal config directives below:

1
2
3

*.=crit :omusrmsg:rger
*.=crit root
*.=crit /var/log/critmsgs

Regular File

消息一般会记录到文件 ，文件名使用全路径，以/起始，新版本也支持相对路径，必须以./起始。如“./file-in-current-dir.log”。但相对路径不可靠，必须要经过严格测试。可以在前面加 “-‘ 来省略文件同步，但如果系统崩溃的话可能会丢失消息，但会提高性能，尤其是记录很长的消息。所以如果有可靠的USP，并且要记录很多消息，可以开启 -

文件名可以是静态，也可以动态，有助于将消息分类保存。

可以和模板一起工作。首先定义一个文件名的模板，我们取名模板为DynFile，然后使用”?”来调用文件动态文件名

1	*.* ?DynFile

1	*.* -?DynFile

也可以使用模板来输出

1	*.* ?DynFile;MyTemplate

支持自动创建目录

1	$template DynFile,"/var/log/%HOSTNAME%/%programname%.log"

Named Pipes

This version of rsyslogd(8) has support for logging output to named pipes (fifos). A fifo or named pipe can be used as a destination for log messages by prepending a pipe symbol (“|’‘) to the name of the file. This is handy for debugging. Note that the fifo must be created with the mkfifo(1) command before rsyslogd(8) is started.

Terminal and Console

If the file you specified is a tty, special tty-handling is done, same with /dev/console.

Remote Machine

要发行其它host，在hostname前加@，默认UDP，如果需要TCP，加@@。TCP不是syslog的官方标准，但另一些app支持，比如syslog-ng或WinSyslog。

可以加一些option，不要加空格，用()包围，多个option用逗号分割

z

Enable zlib-compression for the message. The is the compression level. It can be 1 (lowest gain, lowest CPU overhead) to 9 (maximum compression, highest CPU overhead). The level can also be 0, which means “no compression”. If given, the “z” option is ignored. So this does not make an awful lot of sense. There is hardly a difference between level 1 and 9 for typical syslog messages. You can expect a compression gain between 0% and 30% for typical messages. Very chatty messages may compress up to 50%, but this is seldom seen with typically traffic. Please note that rsyslogd checks the compression gain. Messages with 60 bytes or less will never be compressed. This is because compression gain is pretty unlikely and we prefer to save CPU cycles. Messages over that size are always compressed. However, it is checked if there is a gain in compression and only if there is, the compressed message is transmitted. Otherwise, the uncompressed messages is transmitted. This saves the receiver CPU cycles for decompression. It also prevents small message to actually become larger in compressed form.

Please note that when a TCP transport is used, compression will also turn on syslog-transport-tls framing. See the “o” option for important information on the implications.

Compressed messages 能被接受方自动识别，无需配置。

o

This option is experimental. Use at your own risk and only if you know why you need it! If in doubt, do NOT turn it on.

This option is only valid for plain TCP based transports. It selects a different framing based on IETF internet draft syslog-transport-tls-06. This framing offers some benefits over traditional LF-based framing. However, the standardization effort is not yet complete. There may be changes in upcoming versions of this standard. Rsyslog will be kept in line with the standard. There is some chance that upcoming changes will be incompatible to the current specification. In this case, all systems using -transport-tls framing must be upgraded. There will be no effort made to retain compatibility between different versions of rsyslog. The primary reason for that is that it seems technically impossible to provide compatibility between some of those changes. So you should take this note very serious. It is not something we do not like to do (and may change our mind if enough people beg…), it is something we most probably can not do for technical reasons (aka: you can beg as much as you like, it won’t change anything…).

The most important implication is that compressed syslog messages via TCP must be considered with care. Unfortunately, it is technically impossible to transfer compressed records over traditional syslog plain tcp transports, so you are left with two evil choices…

The hostname may be followed by a colon and the destination port.

The following is an example selector line with forwarding:

. @@(o,z9)192.168.0.1:1470

In this example, messages are forwarded via plain TCP with experimental framing and maximum compression to the host 192.168.0.1 at port 1470.

. @192.168.0.1

In the example above, messages are forwarded via UDP to the machine 192.168.0.1, the destination port defaults to 514. Messages will not be compressed.

Note that IPv6 addresses contain colons. So if an IPv6 address is specified in the hostname part, rsyslogd could not detect where the IP address ends and where the port starts. There is a syntax extension to support this: put square brackets around the address (e.g. “[2001::1]”). Square brackets also work with real host names and IPv4 addresses, too.

A valid sample to send messages to the IPv6 host 2001::1 at port 515 is as follows:

. @[2001::1]:515

This works with TCP, too.

Note to sysklogd users: sysklogd does not support RFC 3164 format, which is the default forwarding template in rsyslog. As such, you will experience duplicate hostnames if rsyslog is the sender and sysklogd is the receiver. The fix is simple: you need to use a different template. Use that one:

• $template sysklogd,”<%PRI%>%TIMESTAMP% %syslogtag%%msg%””

  . @192.168.0.1;sysklogd

List of Users

Usually critical messages are also directed to “root’’ on that machine. You can specify a list of users that shall get the message by simply writing “:omusrmsg: followed by the login name. For example, the send messages to root, use “:omusrmsg:root”. You may specify more than one user by separating them with commas (“,’‘). Do not repeat the “:omusrmsg:” prefix in this case. For example, to send data to users root and rger, use “:omusrmsg:root,rger” (do not use “:omusrmsg:root,:omusrmsg:rger”, this is invalid). If they’re logged in they get the message.

Everyone logged on

Emergency messages often go to all users currently online to notify them that something strange is happening with the system. To specify this wall(1)-feature use an asterisk as the user message destination(“:omusrmsg:*’‘).

Call Plugin

This is a generic way to call an output plugin. The plugin must support this functionality. Actual parameters depend on the module, so see the module’s doc on what to supply. The general syntax is as follows:

:modname:params;template

Currently, the ommysql database output module supports this syntax (in addition to the “>” syntax it traditionally supported). For ommysql, the module name is “ommysql” and the params are the traditional ones. The ;template part is not module specific, it is generic rsyslog functionality available to all modules.

As an example, the ommysql module may be called as follows:

:ommysql:dbhost,dbname,dbuser,dbpassword;dbtemplate

For details, please see the “Database Table” section of this documentation.

Note: as of this writing, the “:modname:” part is hardcoded into the module. So the name to use is not necessarily the name the module’s plugin file is called.

Database Table

This allows logging of the message to a database table. Currently, only MySQL databases are supported. However, other database drivers will most probably be developed as plugins. By default, a MonitorWare-compatible schema is required for this to work. You can create that schema with the createDB.SQL file that came with the rsyslog package. You can also use any other schema of your liking - you just need to define a proper template and assign this template to the action. The database writer is called by specifying a greater-then sign (“>”) in front of the database connect information. Immediately after that sign the database host name must be given, a comma, the database name, another comma, the database user, a comma and then the user’s password. If a specific template is to be used, a semicolon followed by the template name can follow the connect information. This is as follows: >dbhost,dbname,dbuser,dbpassword;dbtemplate

Important: to use the database functionality, the MySQL output module must be loaded in the config file BEFORE the first database table action is used. This is done by placing the

1

$ModLoad ommysql

directive some place above the first use of the database write (we recommend doing at the beginning of the config file).

Discard / Stop

If the discard action is carried out, the received message is immediately discarded. No further processing of it occurs. Discard has primarily been added to filter out messages before carrying on any further processing. For obvious reasons, the results of “discard” are depending on where in the configuration file it is being used. Please note that once a message has been discarded there is no way to retrieve it in later configuration file lines.

Discard can be highly effective if you want to filter out some annoying messages that otherwise would fill your log files. To do that, place the discard actions early in your log files. This often plays well with property-based filters, giving you great freedom in specifying what you do not want.

Discard is just the word “stop” with no further parameters:

stop

For example,

. stop

discards everything (ok, you can achieve the same by not running rsyslogd at all…).

Note that in legacy configuration the tilde character “~” can also be used instead of the word “stop”.

Output Channel

Binds an output channel definition (see there for details) to this action. Output channel actions must start with a $-sign, e.g. if you would like to bind your output channel definition “mychannel” to the action, use “$mychannel”. Output channels support template definitions like all all other actions.

Shell Execute

NOTE: This action is only supported for backwards compatibility. For new configs, use omprog instead. It provides a more solid and secure solution with higher performance.

This executes a program in a subshell. The program is passed the template-generated message as the only command line parameter. Rsyslog waits until the program terminates and only then continues to run.

^program-to-execute;template

The program-to-execute can be any valid executable. It receives the template string as a single parameter (argv[1]).

WARNING: The Shell Execute action was added to serve an urgent need. While it is considered reasonable save when used with some thinking, its implications must be considered. The current implementation uses a system() call to execute the command. This is not the best way to do it (and will hopefully changed in further releases). Also, proper escaping of special characters is done to prevent command injection. However, attackers always find smart ways to circumvent escaping, so we can not say if the escaping applied will really safe you from all hassles. Lastly, rsyslog will wait until the shell command terminates. Thus, a program error in it (e.g. an infinite loop) can actually disable rsyslog. Even without that, during the programs run-time no messages are processed by rsyslog. As the IP stacks buffers are quickly overflowed, this bears an increased risk of message loss. You must be aware of these implications. Even though they are severe, there are several cases where the “shell execute” action is very useful. This is the reason why we have included it in its current form. To mitigate its risks, always a) test your program thoroughly, b) make sure its runtime is as short as possible (if it requires a longer run-time, you might want to spawn your own sub-shell asynchronously), c) apply proper firewalling so that only known senders can send syslog messages to rsyslog. Point c) is especially important: if rsyslog is accepting message from any hosts, chances are much higher that an attacker might try to exploit the “shell execute” action.

Template Name

Every ACTION can be followed by a template name. If so, that template is used for message formatting. If no name is given, a hard-coded default template is used for the action. There can only be one template name for each given action. The default template is specific to each action. For a description of what a template is and what you can do with it, see the template documentation.

RainerScript

RainerScript是一种处理网络事件、配置、过程的脚本语言，是rsyslog的主要配置语言。不能简写成rscript，因为已经被注册了。

v6后支持。

Data Types 数据类型

RainerScript是一种typeless语言，不需要关心数据类型。不能使用 “A” + “B” ，可以使用 & 连接。有需要的话脚本会自动转换类型

Expressions 表达式

支持任意复杂表达式。下面按优先级列出各表达式

• expressions in parenthesis
• not, unary minus
• *, /, % (modulus, as in C)
• +, -, & (string concatenation)
• ==, !=, <>, <, >, <=, >=, contains (strings!), startswith (strings!)
• and
• or

比如，“not a == b” 的返回很有可能不是你想要的，因为脚本处理器会运算“not a”等到一个布尔值再与b比较，而你可能想要的是 “not (a == b)”。

如果要使用不等于，建议使用 “!=” or “<>”，两者是一样的。“not” 一般用在需要使用布尔值的场合。

Functions 函数

有两种函数，build-ins和modules。Built-in可以随时被调用 ，而modules需要事先被载入。

1

module(load="<name of module>")

如果有多于一个的函数使用了相同的名字，第一个载入的有效，同时，会产生一个错误消息，但不会中止。build-in函数总是预先被载入，所以会占据这些名字。

• Built-in Functions
  • ipv42num()
  • num2ipv4()
  • ltrim()
  • rtrim()
  • cnum()
  • cstr()
  • dyn_inc()
  • exec_template()
  • exists()
  • field()
  • format_time()
  • get_property()
  • getenv()
  • int2hex()
  • is_time()
  • lookup()
  • parse_json()
  • parse_time()
  • previous_action_suspended()
  • prifilt()
  • random()
  • re_extract()
  • re_extract_i()
  • re_match()
  • re_match_i()
  • replace()
  • script_error()
  • strlen()
  • substring()
  • tolower()
  • wrap()
• Module Functions
  • HashXX
  • HashXXmod
  • HTTP-Request
  • Unflatten

Control Structures 控制结构

if

1
2
3
4

if ($msg contains "important") then {
   if ( $.foo != "" ) then set $.foo = $.bar & $.baz;
   action(type="omfile" file="/var/log/important.log" template="outfmt")
}

if/else-if/else

1
2
3
4
5
6
7
8
9

if ($msg contains "important") then {
   set $.foo = $.bar & $.baz;
   action(type="omfile" file="/var/log/important.log" template="outfmt")
} else if ($msg startswith "slow-query:") then {
   action(type="omfile" file="/var/log/slow_log.log" template="outfmt")
} else {
   set $.foo = $.quux;
   action(type="omfile" file="/var/log/general.log" template="outfmt")
}

foreach

要说明的是，一般对于foreach会存在误解，这个只能工作于json结构，事实上，我们当时应该拒绝 foreach的，只是太迟了。

要记住，在这种脚本语言中，没有array的概念，因为我们不想把事情搞得太复杂。只能在一些配置对象和一组选择比较时使用。

如果你在解析json，foreach可以迭代json array和json objects。array是有序的，object是kay-value，无序。

For the foreach invocation below:

1
2
3

foreach ($.i in $.collection) do {
   ...
}

Say $.collection holds an array [1, "2", {"a": "b"}, 4], value of $.i across invocations would be 1, "2", {"a" : "b"} and 4.

$.collection 必须来源于 JSON (via mmjsonparse).

When $.collection holds an object {"a": "b", "c" : [1, 2, 3], "d" : {"foo": "bar"}}, value of $.i across invocations would be {"key" : "a", "value" : "b"}, {"key" : "c", "value" : [1, 2, 3]} and {"key" : "d", "value" : {"foo" : "bar"}} (not necessarily in the that order). In this case key and value will need to be accessed as $.i!key and $.i!value respectively.

Here is an example of a nested foreach statement:

1
2
3
4
5
6
7
8
9
10

foreach ($.quux in $!foo) do {
   action(type="omfile" file="./rsyslog.out.log" template="quux")
   foreach ($.corge in $.quux!bar) do {
      reset $.grault = $.corge;
      action(type="omfile" file="./rsyslog.out.log" template="grault")
      if ($.garply != "") then
          set $.garply = $.garply & ", ";
      reset $.garply = $.garply & $.grault!baz;
   }
}

Again, the itereted items must have been created by parsing JSON.

Please note that asynchronous-action calls in foreach-statement body should almost always set action.copyMsg to on. This is because action calls within foreach usually want to work with the variable loop populates (in the above example, $.quux and $.corge) which causes message-mutation and async-action must see message as it was in a certain invocation of loop-body, so they must make a copy to keep it safe from further modification as iteration continues. For instance, an async-action invocation with linked-list based queue would look like:

1
2
3
4

foreach ($.quux in $!foo) do {
    action(type="omfile" file="./rsyslog.out.log" template="quux
           queue.type="linkedlist" action.copyMsg="on")
}

1
2
3
4
5

# 这种代码是无效的，因为不是从json创建的对象
set $.noarr = ["192.168.1.1", "192.168.2."];
foreach ($.elt in $.noarr) do {
    ...
}

call

Details here: The rsyslog “call” statement

continue

A NOP, useful e.g. inside the then part of an if-structure.

configuration objects 配置对象

Common Parameters

config.enabled

New in version 8.33.0.

所有的配置对象都有 config.enabled 参数 ，用来禁用它们。如果设置为on或留空，配置会被启用，如果是其它值，值忽略。这个可以从环境变量或文件中，利用反引号`来调整。

比如，将环境变量LOAD_MPTCP设为off，然后构建

1
2

module(load="imptcp"
     config.enabled=`echo $LOAD_IMPTCP`)

该模块不会被载入

1
2

module(load="imptcp"
     config.enabled=`echo $LOAD_IMPTCP`)

1
2

module(load="imptcp"
     config.enabled=`echo $LOAD_IMPTCP`)

Objects

action()

The action object is the primary means of describing actions to be carried out.

global()

This is used to set global configuration parameters. For details, please see the rsyslog global configuration object.

input()

The input object is the primary means of describing inputs, which are used to gather messages for rsyslog processing.

module()

The module object is used to load plugins.

parser()

The parser object is used to define custom parser objects.

timezone()

The timezone object is used to define timezone settings.

include()

The include object is use to include configuration snippets stored elsewhere into the configuration.

String Constants 字串常量

字串常量是所有的脚本语言所必须，提供了程序开始时的一些值。

Uses

比较，配置参数，函数参数等地方都要用到。

在字串常量中，特殊字符需要用反斜杠去注明。想要知道如何去正确的escape，使用工具 RainerScript String Escape Online Tool.

Types

Rsyslog 提供了多种字串常量，与shell类似:

• 单引号

  可以被escape

• 双引号

  在单引号的基础上，对$进行escape，如果无法被escape，将会产生一个语法错误，导致启动失败

• 反引号

  This was added in 8.33.0. 与shell中类似，only the following is supported:

  • echo $VARNAME - It will evaluate the environment variable and use it as string constant. If the variable is not found, an empty string is generated (this is not an error).

    从8.37开始，echo被加强了，支持环境变量和字串常量混合

    An example:

    • env SOMEPATH is set to “/var/log/custompath”
    • config is: param=echo $SOMEPATH/myfile
    • param than is expanded to “/var/log/custompath/myfile”

    不支持${VAR}，一个环境变量只能被whitespace 或 / 终结。此功能的目的并不是会了模仿bash，而是为了使用自定义外界的参数

  • cat filename - It will evaluate to the content of the given file. 只支持读取一个文件名。如果无法读取，返回空值。

  echo和cat后只能跟一个空格

  Backticks are especially useful for configuration files that are auto-generated but need to contain a small set of special functionality.

  更多例子参考 https://github.com/rsyslog/rsyslog-docker/tree/master/appliance/alpine.

Variable (Property) types 变量类型

所有的属性都能被RainerScript所调用，用$表示

1

set $.x!host = $hostname;

也支持本地变量，只作用于当前消息，但并不是消息属性。（e.g. the “$!” all JSON property does not contain them）

只有 消息的json(CEE/Lumberjack) 属性可以被 set, unset and reset 修改, 本地变量也可以。

消息的 JSON 属性命名以 “$!” 打头，where the bang character represents the root.

本地变量以 “$.” 打头, whJSere the dot denotes the root.

JSON 属性和本地变量都支持任意长度的深度，! 常用来作为路径分割，无论是消息属性还是本地变量。比如 “$!path1!path2!varname” 和 “$.path1!path2!varname” 都是用!分割的，但$后的!和.区分是消息属性还是本地变量。

表达式最后要加上”;”，否则语法错误

参考以下例子

set

sets the value of a local-variable or json property, 如果要定义的变量已经存在内容，将会有以下几种行为方式:

merges 如果新值为object, 但新的值会附加到root而不是指定的key下，比如

1
2
3
4
5
6
7
8
9
10
11

set $.x!one = "val_1";
# results in $. = { "x": { "one": "val_1" } }
set $.y!two = "val_2";
# results in $. = { "x": { "one": "val_1" }, "y": { "two": "val_2" } }

set $.z!var = $.x;
# results in $. = { "x": { "one": "val_1" }, "y": { "two": "val_2" }, "z": { "var": { "one": "val_1" } } }

set $.z!var = $.y;
# results in $. = { "x": { "one": "val_1" }, "y": { "two": "val_2" }, "z": { "var": { "one": "val_1" } }, "two": "val_2" }
# note that the key *two* is at root level and not  under *$.z!var*.

ignores 如果原内容是object，而新值为string或num等非object. Eg:

1
2
3
4

set $.x!one = "val_1";
set $.x = "quux";
# results in $. = { "x": { "one": "val_1" } }
# note that "quux" was ignored

resets 如果旧值为非object.

1
2
3

set $.x!val = "val_1";
set $.x!val = "quux";
# results in $. = { "x": { "val": "quux" } }

unset

removes the key. Eg:

1
2
3

set $.x!val = "val_1";
unset $.x!val;
# results in $. = { "x": { } }

reset

force sets the new value regardless of what the variable originally contained or if it was even set. Eg.

1
2
3
4
5
6
7
8
9
10
11
12
13

# to contrast with the set example above, here is how results would look with reset
set $.x!one = "val_1";
set $.y!two = "val_2";
set $.z!var = $.x;
# results in $. = { "x": { "one": "val_1" }, "y": { "two": "val_2" }, "z": { "var": { "one": "val_1" } } }
# 'set' or 'reset' can be used interchangeably above(3 lines), they both have the same behaviour, as variable doesn't have an existing value

reset $.z!var = $.y;
# results in $. = { "x": { "one": "val_1" }, "y": { "two": "val_2" }, "z": { "var": { "two": "val_2" } } }
# note how the value of $.z!var was replaced

reset $.x = "quux";
# results in $. = { "x": "quux", "y": { "two": "val_2" }, "z": { "var": { "two": "val_2" } } }

Lookup Tables 查找表

Lookup tables are a powerful construct to obtain “class” information based on message content (e.g. to build log file names for different server types, departments or remote offices).

General Queue Parameters 通用队列参数

Usage

Queue parameters can be used together with the following statements:

• action()
• ruleset()
• main_queue()

Queues need to be configured in the action or ruleset it should affect. If nothing is configured, default values will be used. Thus, the default ruleset has only the default main queue. Specific Action queues are not set up by default.

To fully understand queue parameters and how they interact, be sure to read the queues documentation.

参考 https://www.rsyslog.com/doc/master/rainerscript/queue_parameters.html

Examples

Example 1

The following is a sample of a TCP forwarding action with its own queue.

1
2
3

action(type="omfwd" target="192.168.2.11" port="10514" protocol="tcp"
       queue.filename="forwarding" queue.size="1000000" queue.type="LinkedList"
      )

The rsyslog “call” statement

Call与ruleset联系，可以把rulesets看成是一个子程序，就容易理解call了。

Call可以调用任何rulsests，如果在ruleset中存在queue，由消息发送给queue异步处理，否则会同步执行，结束后返回控制

在注意同步和异步的区别

Call是用来取代已经废弃的 omruleset模块，使用了新引擎，更有效率，特别是对于那些同步操作，几乎是0开销。而omrulset需要复制消息，这至少要消耗250字节的内存和一些计算性能

syntax

1

call rulesetname

Where “rulesetname” is the name of a ruleset that is defined elsewhere inside the configuration. If the call is synchronous or asynchronous depends on the ruleset parameters. This cannot be overridden by the “call” statement.

related links

• Blog posting announcing “call” statement (with sample)

The rsyslog “call_indirect” statement

The rsyslog “call_indirect” 类亿于 “call”，区别在于被call的ruleset不是常量，还是一个实时计算的表达式

如果ruleset不存在，会产生一个错误消息，然后被跳过，继续执行下一语句

syntax

1

call_indirect expression;

Where “expression” is any valid expression. See expressions for more information.

结尾要有分号

examples

“call_indirect” 可以根据消息变量来调用ruleset，例如，rulesets以syslog tag命名，就可以这么写

1

call_indirect $syslogtag;

使用时要小心被注入，最好在ruleset前加上前缀，比如 “changeme-” :

1

call_indirect "changeme-" & $syslogtag;

call_indirect也可以调用常量名，比如

1

call_indirect "my_ruleset";

不过常量名的调用最好还是用call，效率会高很多。

additional information

We need to have two different statements, “call” and “call_indirect” because “call” already existed at the time “call_indirect” was added. We could not extend “call” to support expressions, as that would have broken existing configs. In that case call ruleset would have become invalid and call "ruleset" would have to be used instead. Thus we decided to add the additional “call_indirect” statement for this use case.

global() configuration object

允许设置全局变量，但每个变量只能设置一次，之后不能被reset。

参考 https://www.rsyslog.com/doc/master/rainerscript/global.html

The rsyslog include() object

include() 可以调用 配置片断，New in version 8.33.0.

How it Works

如果不熟悉include，可以认为是一种复制粘贴。rsyslog到到要include的对象，复制内容到指定位置，然后删除include文句

include时要注意位置，不同的位置可能会影响执行结果。

Parameters

参数名大小写不敏感，每个include中file和text只能选其一

file

Name of file to be included. May include wildcards, in which case all matching files are included (in order of file name sort order).

text

Text to be included. This is most useful when using backtick string constants.

mode

Affects how missing files are to be handled:

• abort-if-missing, with rsyslog aborting when the file is not present
• required (default), with rsyslog emitting an error message but otherwise continuing when the file is not present
• optional, which means non-present files will be skipped without notice

Examples

Include a required file

1

include(file="/path/to/include.conf")

Note

Unless otherwise specified, files referenced by an include() object must be present, otherwise an error will be generated.

Include an optional file

The referenced file will be used if found, otherwise no errors or warnings will be generated regarding its absence.

1
2
3
4

include(
   file="/path/to/include.conf"
   mode="optional"
)

Include multiple files

1

include(file="/etc/rsyslog.d/*.conf")

Note

Unless otherwise specified, files referenced by an include() object must be present, otherwise an error will be generated.

Include an environment variable as configuration

1

include(text=`echo $ENV_VAR`)

Include a file specified via an environment variable

1

include(file=`echo $ENV_VAR`)

Note

Unless otherwise specified, files referenced by an include() object must be present, otherwise an error will be generated.

Include an optional file specified via an environment variable

1
2
3
4

include(
   file=`echo $ENV_VAR`
   mode="optional"
)

Filter Conditions

rsyslog支持以下条件

• 传统的severity和facility
• Property-Based Filters 基于属性
• 基于表达式
• BSD-style blocks（不再向后兼容）

Selectors

Selectors 是传统的筛选方式. 简洁，高效，特别是在v7中，比advanced模式高效，目前的版本中两者已经没有差别。

Selector 由 facility和priority组成，由”.”分割，大小写敏感，可以写成十进制数字，但最好不这么做。个体可以参考syslog(3). 这些名字在文件/usr/include/syslog.h 定义，类似 LOG_-values

facility 关键字: auth, authpriv, cron, daemon, kern, lpr, mail, mark, news, security (same as auth), syslog, user, uucp and local0 through local7。 security 不应被再被app使用。

Priority : debug, info, notice, warning, warn (same as warning), err, error (same as err), crit, alert, emerg, panic (same as emerg). The keywords error, warn and panic are deprecated and should not be used anymore. The priority defines the severity of the message.

rsyslog和syslog行为类似，并有一些扩展，比如理解(“*’‘)和none，也可以使用”,”指定多个多个priority. 也可以指定多个facility，但要注意的时，如果是多个facility语句，priority将被忽略

多个selectors可以(“;’‘)连接，后者覆盖前者，还可以使用(“=’‘)和(“!’‘)

Property-Based Filters

rsyslogd特有。可以过滤任何属性。参考 rsyslog properties documentation

property-based filter以”:”起始，后跟属性名字，然后是”,”，比较操作符，”,”，比较值（用双引号包围）。逗号之间可以有空格和tab。属性名和比较符大小写敏感。

1

:property, [!]compare-operation, "value"

Compare-Operations

contains

​ 是否包含某值，精确匹配，不支持通配符

isequal

​ 两者必须精确匹配，一般用于 syslogtag or FROMHOST

startswith

​ Checks if the value is found exactly at the beginning of the property value. For example, if you search for “val” with

1

:msg, startswith, "val"

​ 可匹配 “values are in this message” 但不匹配 “There are values in this message” (“contains” 可以). startswith远比regex高效

regex

​ Compares the property against the provided POSIX BRE regular expression.

ereregex

​ Compares the property against the provided POSIX ERE regular expression.

可在比较操作之前使用(!)来反向操作。比如 “This is an informative message”

1
2
3
4
5

# 不匹配
:msg, contains, "error"

# 匹配
:msg, !contains, "error"

Value Part

vaule是一个引号间的字串，支持一些转义

“ - the quote character (e.g. “String with “Quotes””)

\ - the backslash character (e.g. “C:\tmp”)

一般来说，主要是针对真实的 msg 做一些过滤

1
2

# 注意，大小写敏感，不会匹配“id-4711”
:msg, contains, "ID-4711"

1
2

# 匹配 fatal和error，中间任何字符。
msg, regex, "fatal .* error"

使用属性过滤是一个挑战，可以使用debug来测试，rsyslogd -d

不支持布尔操作，如果要过滤facility或severity，建议使用 selectors

Expression-Based Filters

可以过滤任意复杂的表达式，包括布尔，算术和字串操作，类似于完整的脚本语言，不过在语法有些许区别。

表达式过滤以关键字 if 开始，类似于

1

if expr then action-part-of-selector-line

if 和 then是关键字，必有存在，expr是表达式，参考 expression documentation ，action-part-of-selector-line是action

BSD-style Blocks

v7+不再支持，所以不建议再使用

例子

1
2

*.* /var/log/file1 # 传统方式
if $msg contains 'error' then /var/log/errlog # 基于表达式的方式

Right now, you need to specify numerical values if you would like to check for facilities and severity. These can be found in RFC 5424. If you don’t like that, you can of course also use the textual property - just be sure to use the right one. As expression support is enhanced, this will change. For example, if you would like to filter on message that have facility local0, start with “DEVNAME” and have either “error1” or “error0” in their message content, you could use the following filter:

1
2

# 必须在一行
if $syslogfacility-text == 'local0' and $msg startswith 'DEVNAME' and ($msg contains 'error1' or $msg contains 'error0') then /var/log/somelog

1
2

# 如果要保存除了 error1 或 error0 的日志，只要加上 not
if $syslogfacility-text == 'local0' and $msg startswith 'DEVNAME' and not ($msg contains 'error1' or $msg contains 'error0') then /var/log/somelog

如果要忽略大小写的比较，使用 “contains_i” 代替 “contains” ，使用 “startswith_i” 代替 “startswith”. 注意，表达式过滤不支持正则表达式

The Property Replacer 属性替换

The Property Replacer 是 rsyslogd的 字符模板的核心组件，用来操纵属性值

Accessing Properties

通过两个 % 读取属性，并被修改，语法如下

1

%property:fromChar:toChar:options%

Available Properties

见 rsyslog properties

Character Positions

FromChar 和toChar 用来截字串，它们代表偏移量，从1开始计数。截取前两个字符语法为: “%msg:1:2%”. 如果只使用options，冒号还是需要的。比如 “%msg:::lowercase%”. 如果要截取从某位置到末尾，设置toChar为 (“$”) (e.g. %msg:10:$%, which will extract from position 10 to the end of the string).

支持正则表达式。设置FromChar为“R”，rsyslog就会知道使用正则面不是位置参数，表达式放在toChar上，必须以“–end” 结尾。rsyslog会返回匹配的部分。比如 “%msg:R:.*Sev:. (.*) [.*–end%”

在 “R” 后还可以支持一些参数

R,<regexp-type>,<submatch>,<nomatch>,<match-number>

regexp-type 可以使用 “BRE” for Posix basic regular expressions or “ERE” for extended ones. 必须要大写. 早期的版本只支持BRE。submatch指示器指示使用哪一个submatch结果，支持单数字。0表示整个匹配，1 to 9 为实际submatch. match-number 表示使用哪一个结果，从0开始。最多支持10个（数字是9）

nomatch 指示如果没有匹配将使用的值

这个例子使用ERE规则，从消息提取第一个子匹配，如果没有匹配，则使用整个字段

1

%msg:R,ERE,1,FIELD:for (vlan[0-9]\*):--end%

以下是从第二个匹配中，提取第一个子匹配

1

%msg:R,ERE,1,FIELD,1:for (vlan[0-9]\*):--end%

强烈建议使用 rsyslog regular expression checker/generator 工具。虽然不同版本有所不同，但基本覆盖了大部分情况。

关于nomatch mode

如果没有匹配，以早期会返回“**NO MATCH**“，此模式为 DFLT

现在增加了几个有用的模式。

提取也可以基于fields，将FromChar设为F。Fields以定位符分割，默认是TAB（ASCII 9），也可以自定义，例如需要用”,”作为定位符，则写成 “F,44”，44是”,”的ASCII值。用定位符比正则高效，如果消息是规则的话。Field是从1开始，如果设为0，或高于实际数量，会导致”field not found”错误。toChar上设置需要的field数，比如”%msg:F:3%”，或者用”;”分割: %msg:F,59:3%”

使用fields的不足是无法截取字串，从6.3.9开始，fromPos和toPos可以用来解决这个问题，但语法上有点丑。比如在上例中要截取第5到第9个字符，语法是“%msg:F,59,5:3,9%”

F和R必须大写，语法不能有空格。

每当有字段定位符出现，就会开启一个新的字段。可以在定位符后加上”+”，则多个定位符将视过一个。比如以下代码

1
2
3

int n, m;
...
syslog(LOG_ERR, "%d test %6d", n, m);

有可能输出“1 test 2”, “1 test 23”, “1 test 234567”，空格数量不可知。可以使用以下语法

1

"%msg:F,32:2%" to "%msg:F,32+:2%".

Property Options

大小写敏感

• uppercase

  convert property to uppercase only

• lowercase

  convert property text to lowercase only

• fixed-width

  当原始字串长度小于toChar时填充。This feature was introduced in rsyslog 8.13.0

• json

  将value转会json可以被解析的格式，比如将ASCII LF转为“\n”，不能和csv同存

• jsonf[:outname]

  该属性将会被转为json，与json不同的是，json是去转化value，让其可以被json，而jsonf是将整个结果是源于 “fieldname”=”value” 这种格式。其中fieldname是参数指定，否则使用默认属性名。value可以被option操控，但field name不能，所以需要指定恰当的名字。参考 this article from Rainer’s blog 提到

  1 2 3

  # json的模板，比较丑陋，为了生成json，相当于字段的拼接 $template tpl, “{“message”:”%msg:::json%”,”fromhost”:”%HOSTNAME:::json%”,”facility”:”%syslogfacility-text%”,”priority”:”%syslogpriority-text%”,”timereported”:”%timereported:::date-rfc3339%”,”timegenerated”:”%timegenerated:::date-rfc3339%”}”

  1 2 3

  # jsonf的模板，直接是从 “fieldname”=”value” 生成json $template tpl,”{%msg:::jsonf:message%,%HOSTNAME:::jsonf:fromhost%,%syslogfacility-text:::jsonf:facility%,%syslogpriority-text:::jsonf:priority%,%timereported:::date-rfc3339,jsonf%,%timegenerated:::date-rfc3339,jsonf%}”

  1 2 3

  # 如果不需要改变fieldname $template tpl,”{%msg:::json%,%HOSTNAME:::jsonf%,%syslogfacility-text:::jsonf%,%syslogpriority-text:::jsonf%,%timereported:::date-rfc3339,jsonf%,%timegenerated:::date-rfc3339,jsonf%}”

• csv

  根据 RFC 4180 生成csv格式。Rsyslog 总是使用双引号。 例如: $template csvline,”%syslogtag:::csv%,%msg:::csv%” ，你需要在模板中正常定义”,”的位置

• drop-last-lf

  The last LF in the message (if any), is dropped. Especially useful for PIX.

• date-utc

  convert data to UTC prior to outputting it (available since 8.18.0)

• date-mysql

  format as mysql date

• date-rfc3164

  format as RFC 3164 date，”Mmm dd hh:mm:ss”

• date-rfc3164-buggyday

  与 date-rfc3164 相同, 在RFC 3164 如果日期为单数字，需要写入一个空格，在buggyday中，用0来代替。如果要转发消息，不建议使用这个option，有可能会被远端服务器认识是错误

• date-rfc3339

  format as RFC 3339 date，”2013-09-12T22:50:20+08:00”

• date-unixtimestamp

  Format as a unix timestamp (seconds since epoch)

• date-year

  just the year part (4-digit) of a timestamp

• date-month

  just the month part (2-digit) of a timestamp

• date-day

  just the day part (2-digit) of a timestamp

• date-hour

  just the hour part (2-digit, 24-hour clock) of a timestamp

• date-minute

  just the minute part (2-digit) of a timestamp

• date-second

  just the second part (2-digit) of a timestamp

• date-subseconds

  just the subseconds of a timestamp (always 0 for a low precision timestamp)

• date-tzoffshour

  just the timezone offset hour part (2-digit) of a timestamp

• date-tzoffsmin

  just the timezone offset minute part (2-digit) of a timestamp. Note that this is usually 0, but there are some time zones that have offsets which are not hourly-granular. If so, this is the minute offset.

• date-tzoffsdirection

  just the timezone offset direction part of a timestamp. This specifies if the offsets needs to be added (“+”) or subtracted (“-“) to the timestamp in order to get UTC.

• date-ordinal

  returns the ordinal for the given day, e.g. it is 2 for January, 2nd

• date-week

  returns the week number

• date-wday

  just the weekday number of the timstamp. This is a single digit, with 0=Sunday, 1=Monday, …, 6=Saturday.

• date-wdayname

  just the abbreviated english name of the weekday (e.g. “Mon”, “Sat”) of the timestamp.

• escape-cc

  replace control characters (ASCII value 127 and values less then 32) with an escape sequence. The sequence is “#” where charval is the 3-digit decimal value of the control character. For example, a tabulator would be replaced by “#009”. Note: using this option requires that $EscapeControlCharactersOnReceive is set to off.

• space-cc

  replace control characters by spaces Note: using this option requires that $EscapeControlCharactersOnReceive is set to off.

• drop-cc

  drop control characters - the resulting string will neither contain control characters, escape sequences nor any other replacement character like space. Note: using this option requires that $EscapeControlCharactersOnReceive is set to off.

• compressspace

  compresses multiple spaces (US-ASCII SP character) inside the string to a single one. This compression happens at a very late stage in processing. Most importantly, it happens after substring extraction, so the FromChar and ToChar positions are NOT affected by this option. (available since v8.18.0)

• sp-if-no-1st-sp

  这个option有点吓人，一般不应该被使用。当字串第一个字符为非空格时，返回空格，否则返回空值。这主要是用来解决RFC3164的问题。在3164中，没有对tag和message指定分割符，而通常是以空格代替，所以当空格是消息的一部分时，会带来问题。tag后立即是另一个非空格字符，会给解释器带来误解。

  看不懂这个解释也很正常，反正忘掉它就好 ;)

• secpath-drop

  Drops slashes inside the field (e.g. “a/b” becomes “ab”). Useful for secure pathname generation (with dynafiles).

• secpath-replace

  Replace slashes inside the field by an underscore. (e.g. “a/b” becomes “a_b”). Useful for secure pathname generation (with dynafiles).

要使用多个options，只要简单的用”,”连接它们，比如“escape-cc,sp-if-no-1st-sp”，如果选项有冲突，后者会覆盖前者，比如 “escape-cc,drop-cc” will use drop-cc and “drop-cc,escape-cc” will use escape-cc mode.

Further Links

• Article on “Recording the Priority of Syslog Messages” (describes use of templates to record severity and facility of a message)

• Configuration file syntax, this is where you actually use the property replacer.

• Property Replacer nomatch mode

  • Summary of nomatch Modes

Properties

Properties 属性

rsyslog中的数据项称为Properties，有些来源于消息，有些来源于其它

Properties被用于

• 模板
• 条件声明

Properties是大小写敏感的

Properties也有被称为variables，两者意义相同，区别可参考 rsyslog lead author Rainer Gerhards explains the naming difference ，（最初properties是不可变的，但后来引入了可变的properties, 成了variable properties）

消息属性

rsyslog会从原始消息中解析出许多属性，所有的消息属性以字母开始。

msg

the MSG part of the message (aka “the message” ;))

rawmsg

原始消息，一般用来debug或需要原封不动的转发时。需要注意的是 EscapecontrolCharactersOnReceive 默认被启用，这可能会导致转发的消息有所改变

rawmsg-after-pri

移除了PRI的rawmsg。如果本来就没有PRI，那rawmsg-after-pri就等同于rawmsg。PRI是syslog的消息头部中，包含了facility和severity的信息。由 <> 包围，比如<191>。这个字段通常不写入日志，但常用来进行消息分类。

hostname

hostname from the message

source

alias for HOSTNAME

fromhost

消息从认证那接受，对于多个转发的链路，这代表了上一个发送者，而非原始发送者。这是一个DNS解析的名字，除非解析失败或者解析被禁用

fromhost-ip

The same as fromhost, but always as an IP address. Local inputs (like imklog) use 127.0.0.1 in this property.

syslogtag

TAG from the message

programname

BSD syslogd中定义的tag的static部分。比如 “named[12345]”, programname 就是 “named”。

确切的说，programname会被以下字符终结

• end of tag
• nonprintable character
• ‘:’
• ‘[‘
• ‘/’

如果你的程序包含了 ‘/‘，比如 “app/foo[1234]”，那programname就是app，如果是绝对路径比如“/app/foo[1234]”, programname 为空 (“”)。如果确实需要保存 ‘/‘，可以设置全局选项global(parser.permitSlashInProgramName=”on”) 来启用.

Note: this option is available starting at rsyslogd version 8.25.0.

pri

PRI part of the message - undecoded (single value)，数字

pri-text

the PRI part of the message in a textual form with the numerical PRI appended in brackets (e.g. “local0.err<133>”)

iut

the monitorware InfoUnitType - used when talking to a MonitorWare backend (also for Adiscon LogAnalyzer)

syslogfacility

the facility from the message - in numerical form

syslogfacility-text

the facility from the message - in text form

syslogseverity

severity from the message - in numerical form

syslogseverity-text

severity from the message - in text form

syslogpriority

an alias for syslogseverity - included for historical reasons (be careful: it still is the severity, not PRI!)

仍是sererity，不是PRI！！

syslogpriority-text

an alias for syslogseverity-text

timegenerated

timestamp when the message was RECEIVED. Always in high resolution

timereported

timestamp from the message. Resolution depends on what was provided in the message (in most cases, only seconds)

timestamp

alias for timereported

protocol-version

The contents of the PROTOCOL-VERSION field from IETF draft draft-ietf-syslog-protocol

structured-data

The contents of the STRUCTURED-DATA field from IETF draft draft-ietf-syslog-protocol

app-name

The contents of the APP-NAME field from IETF draft draft-ietf-syslog-protocol

procid

The contents of the PROCID field from IETF draft draft-ietf-syslog-protocol

msgid

The contents of the MSGID field from IETF draft draft-ietf-syslog-protocol

inputname

产生消息的imput module名 (e.g. “imuxsock”, “imudp”). 并不是所有的module会提供这个字段，可以留空，名字也可以随意定义，并不一定是真实的module名。

jsonmesg

Available since rsyslog 8.3.0

The whole message object as JSON representation. Note that the JSON string will not include an LF and it will contain all other message properties specified here as respective JSON containers. It also includes all message variables in the “$!” subtree (this may be null if none are present).

This property is primarily meant as an interface to other systems and tools that want access to the full property set (namely external plugins). Note that it contains the same data items potentially multiple times. For example, parts of the syslog tag will by contained in the rawmsg, syslogtag, and programname properties. As such, this property has some additional overhead. Thus, it is suggested to be used only when there is actual need for it.

System Properties

rsyslog引擎产生的properties，和消息本身无关，名字以$开始

注意一些时间相关的系统变量：

• timereported 消息头部中的时间戳，代表消息产生的时间，取决于转发链的长度，这个时间有可能比当前落后很多
• timegenerated 本地系统收到消息的时间。本地rsyslog接受缓存收到的时，在任何处理之前。如果buffer接收了多条消息，那这些消息的 timegenerated都是一样的
• $now is not from the message. 系统开始处理消息的时间。与 timegenerated会有一些差异。 如果消息在队列中长时间等待，那差异可能会达到秒级，甚至小时级。

一些系统属性：

• $bom

  The UTF-8 encoded Unicode byte-order mask (BOM). This may be useful in templates for RFC5424 support, when the character set is know to be Unicode.

• $myhostname

  The name of the current host as it knows itself (probably useful for filtering in a generic way)

Time-Related System Properties

以下这些系统属性存在于本地时间变量中（$now）和 UTC（￥now-utc）中。utc相关的变量都以“-utc” 结尾。在一个模板中，本地时间和UTC只能取其一。

不同的模板在使用 $now时，可能会产生不同的结果，为了避免这种情况，建议使用 timegenerated 。

• $now

  The current date stamp in the format YYYY-MM-DD

• $year

  The current year (4-digit)

• $month

  The current month (2-digit)

• $day

  The current day of the month (2-digit)

• $wday

  The current week day as defined by ‘gmtime()’. 0=Sunday, …, 6=Saturday

• $hour

  The current hour in military (24 hour) time (2-digit)

• $hhour

  The current half hour we are in. From minute 0 to 29, this is always 0 while from 30 to 59 it is always 1.

• $qhour

  The current quarter hour we are in. Much like $HHOUR, but values range from 0 to 3 (for the four quarter hours that are in each hour)

• $minute

  The current minute (2-digit)

• $now-unixtimestamp

  The current time as a unix timestamp (seconds since epoch). This actually is a monotonically increasing counter and as such can also be used for any other use cases that require such counters. This is an example of how to use it for rate-limiting:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

  # Get Unix timestamp of current message set $.tnow = $$now-unixtimestamp # Rate limit info to 5 every 60 seconds if ($!severity == 6 and $!facility == 17) then { if (($.tnow - $/trate) > 60) then { # 5 seconds window expired, allow more messages set $/trate = $.tnow; set $/ratecount = 0; } if ($/ratecount > 5) then { # discard message stop } else { set $/ratecount = $/ratecount + 1; } }

  NOTE: by definition, there is no “UTC equivalent” of the $now-unixtimestamp property.

说明

模板用来指定格式，动态文件名生成。所有的output都会用到模板。不光是文件输出，写入到数据库也需要合适的模块来组成语句。如果没有指定模板，会调用默认的模块，可以在rsconf.c中搜索”template_”查看。

模板使用 template() 声明，在legacy中，使用 $template

模板使用的属性可查看 rsyslog properties reference

模板执行

虽然没有标准化的日志格式，但一般来说，模板应该包含RFC5424定义的HEADER，rsyslog解析会用到。

如果一条消息，没有HOSTANME，没有TAG，内容为 “this:is a message”，刚会被解析成

1
2

TAG:this:
MSG:is a message

The template() statement

这是一个static声明，意味着所有的模板都在配置中被定义，所以模板也不会被if或其它声明所改变。

1

template(parameters) { list-descriptions }

每一个模板都要有一个 name ，必有唯一，否则会导致行为不可知。

每一个模板都有一个参数 type，不同的type 可以使能不同的方法解析不同的模板内容。

模板不会影响output插件。

type有以下几种

• list
• subtree
• string
• plugin

List

此模板一系列的constant和variable 组成，放在{ }中。这种类型常用在结构化的输出，如ommongodb，也能工作在文本型的输出。建议在需要复杂的属性替换中使用此模板，因为基本list的语法比基于string的方法清晰。

constant包括一些文本和 property ，可以对其作一些修改（如转换大小写）

1
2
3
4
5
6
7

template(name="tpl1" type="list") {
     constant(value="Syslog MSG is: '")
     property(name="msg")
     constant(value="', ")
     property(name="timereported" dateFormat="rfc3339" caseConversion="lower")
     constant(value="\n")
     }

constant statement

主要用于基于文本的输出，可以添加一下文字。比如需要添加换行符

1
2
3
4

template(name="outfmt" type="list") {
    property(name="$!usr!msgnum")
    constant(value="\n")
}

特殊字符

• \ - single backslash
• \n - LF
• \ooo - (three octal digits) - represents a character with this octal numerical value (e.g. \101 equals “A”). Note that three octal digits must be given (in contrast to some languages, where between one and three are valid). While we support octal notation, we recommend to use hex notation as this is better known.
• \xhh - (where h is a hex digit) - represents a character with this hexadecimal numerical value (e.g. \x41 equals “A”). Note that two hexadecimal digits must be given (in contrast to some languages where either one or two are valid).
• … some others … list needs to be extended

如果在不支持的字符前使用\，会导致问题。

为了帮助对基于文本的输出和结构化输出使用相同的模板，在为结构化输出创建名称/值树时，将忽略没有“outname”参数的常量文本。 因此，如果您想提供一些常量文本，例如 对于 mongodb，您必须包含一个 outname，如下所示：

1
2
3
4

template(name="outfmt" type="list") {
    property(name="$!usr!msgnum")
    constant(value="\n" outname="IWantThisInMyDB")
}

要生成一个常量json字段，可以使用format参数，如本例所示

1
2
3
4

template(name="outfmt" type="list" option.jsonf="on") {
          property(outname="message" name="msg" format="jsonf")
          constant(outname="@version" value="1" format="jsonf")
}

在本例中，constant语句将生成@version: 1。注意，要做到这一点，必须同时给出值和格式参数。

constant语句支持以下参数:

• value - 值
• outname - 输出字段名称(用于结构化输出)
• format - 可以为空或 jsonf

Property statement

可以使用option来截取或更改，支持下列参数

• name - 属性名

• outname - output字段名 (用于结构化输出)

• dateformat - 日期格式 (只用于时间相关属性)，参考 文档 。TODO: 目前，属性替换相关文档只包含了文本模板的一些option，这些option不同于非文本模板。比如，在文本模板中，日期格式option，会以”date-“开头，而在属性语句中不需要（比如”date-year”和”year”）。究其原因，在文本模板中，必须告诉这个这个声明所应用的地方，而在属性中名称中，这点已声明。

  可以自己定义格式，比如 YYYY-MM-DD:

1
2
3
4
5

property(name="timereported" dateformat="year")
constant(value="-")
property(name="timereported" dateformat="month")
constant(value="-")
property(name="timereported" dateformat="day")

• date.inUTC - date shall be shown in UTC (需要损失一点性能) Available since 8.18.0.

• caseconversion - 转换大小定， 取值为 “lower” and “upper”

• controlcharacters - 如何处理控制字符。 可使用的值 “escape”, which escapes them, “space”, which replaces them by a single space, and “drop”, which simply removes them from the string.

• securepath - 用于在动态文件模板中创建合适的文件路径

• format - specify format on a field basis. Supported values are:

  • “csv” for use when csv-data is generated
  • “json” which formats proper json content (but without a field header)
  • “jsonf” which formats as a complete json field
  • “jsonr” which avoids double escaping the value but makes it safe for a json field
  • “jsonfr” which is the combination of “jsonf” and “jsonr”.

• position.from - 获取从该位置起之后的字串 (从1开始)

• position.to - 获取位置之前的字串

• position.relativeToEnd - 获取到末性的相对偏移的字串. (available since rsyslog v7.3.10)

• fixedwidth - 更改position.to行为，当原始字串长度小于时，用空格填充，取值“on” 或 “off” (default) (available since rsyslog v8.13.0)

• compressspace - 压缩空格，将多个空格压缩为一个，这个特性会在很晚执行，所以不会影响position.from或position.to的执行。(available since v8.18.0).

• field.number - obtain this field match

• field.delimiter - decimal value of delimiter character for field extraction

• regex.expression - expression to use

• regex.type - either ERE or BRE

• regex.nomatchmode - what to do if we have no match

• regex.match - match to use

• regex.submatch - submatch to use

• droplastlf - drop a trailing LF, if it is present

• mandatory - 强制一个字段。如果设为 “on”，该字段始终会传递给结构输出，即使不存在。如果 “off” (the default)，空字段不会被传递. 这在输出支持动态 schemas 比较有用(like ommongodb).

• spifno1stsp - expert options for RFC3164 template processing

• datatype - 在jsonf 格式中生效; 允许为原始的字串指定数据类型。有时你在延时时需要用到数字或布尔类型，可以用这个option去更改。

  • number - value is treated as a JSON number and not enclosed in quotes.

    If the property is empty, the value 0 is generated.

  • string - 字串会被引号包裹

  • auto - value is treated as number if numeric and as string otherwise.

    只对整数有效

  • bool - the value is treated as boolean. If it is empty or 0, it will generate “false”, else “true”.

  默认使用string类型. This is a feature of rsyslog 8.1905.0 or later.

• onEmpty - 在jsonf 格式中生效; 指定如何处理空值. Possible values are:

  • keep - emit the empty element
  • skip - completely ignore the element, do not emit anything
  • null - emit a JSON ‘null’ value

  If not specified, ‘keep’ is assumed. This is a feature of rsyslog 8.1905.0 or later.

Subtree

7.1.4后支持

基于a complete (CEE) subtree 创建模板。用于输出层次给的结构，比如ommongodb。要使用该模板，必须在参数中声明。比如

1
2
3
4
5

# 包含所有 CEE 数据
template(name=”tpl1” type=”subtree” subtree=”$!”)  

# 只包含以 $!usr!tpl2 开始的数据
template(name=”tpl2” type=”subtree” subtree=”$!usr!tpl2”)

subtree类型与能用于文件输出，比如 omfile，但没有处理常量文本的能力。一般用来用于调试

一个例子

1
2
3

set $!usr!tpl2!msg = $msg;
set $!usr!tpl2!dataflow = field($msg, 58, 2);
template(name="tpl2" type="subtree" subtree="$!usr!tpl2")

String

非常类似于legacy模板声明。使用强制参数string。一个模板字串由常量文本和变量替换完成，对于没有复杂操作要求的消息，使用String-base的模板足够了。

1
2
3

template(name="tpl3" type="string"
         string="%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
        )

在“%”之间的内容会被property replacer解释。简而言之，包含了所要处理的property和option，这和list模板中的property对象类似。

在 “%”之外的就是常量文本，在上面的例子，我们使用了空格来分割，最后加上分行

可以使用\来打印控制字符。

Plugin

模板是于plugin产生（也叫做 strgen或 string generator）。格式是固定的，不可更改。它的性能是非常好（并不是说普通模板慢，只是在一些极端环境中，会有所差别）。使用前必须先载入plugin

1

template(name="tpl4" type="plugin" plugin="mystrgen")

options

option是可选的。

option.sql - 格式化为MySQL语句. 比如在”‘“前添加 \，请注意在MySQL的配置中,NO_BACKSLASH_ESCAPES 模式需要为off (this is the default).

option.stdsql - 格式化为 sql server 语句. 将 ‘ 改为两个 ‘’. You must use stdsql together with MySQL if in MySQL configuration the NO_BACKSLASH_ESCAPES is turned on.

option.json - format the string suitable for a json statement. This will replace single quotes (“’”) by two single quotes (“’‘”) inside each field.

option.jsonf - format the string as JSON object. 这意味着在所有非终结符属性和常量之间都将添加一个前导和后导的花括号{以及一个逗号。

option.casesensitive - treat property name references as case sensitive. 默认为 “off”, 所有模板在定时，属性名都会转为小写. With this option turned “on”, property names are looked up as defined in the template. Use this option if you have JSON ($!*), local (!.*), or global ($!\\*) properties which contain uppercase letters. The normal Rsyslog properties are case-insensitive, so this option is not needed for properly referencing those properties.

option.sql, option.stdsql, and option.json 是相互排斥的。

在使用sql or stdsql 向数据库写入时，要注意确保使用正确格式，防止被注入, If you choose the wrong one, you are still vulnerable to sql injection. The sql option 可以使用在文件中 - 比如想把它们导入到别的机器数据库中。如果没有必要，就不要使用这些选项同，会带来性能的损耗

默认的数据库模板使用 sql option。

1
2

template (name="TraditionalFormat" type="string"
string="%timegenerated% %HOSTNAME% %syslogtag%%msg%\\n")

例子

写入文件

1
2
3
4
5
6
7
8
9
10

template(name="FileFormat" type="list") {
    property(name="timestamp" dateFormat="rfc3339")
    constant(value=" ")
    property(name="hostname")
    constant(value=" ")
    property(name="syslogtag")
    property(name="msg" spifno1stsp="on" )
    property(name="msg" droplastlf="on" )
    constant(value="\n")
    }

也可以写成

1
2
3

template(name="FileFormat" type="string"
         string= "%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
        )

注意，string模板必须写成一行

转发到远端（RFC3164）

1
2
3
4
5
6
7
8
9
10
11
12

template(name="ForwardFormat" type="list") {
    constant(value="<")
    property(name="pri")
    constant(value=">")
    property(name="timestamp" dateFormat="rfc3339")
    constant(value=" ")
    property(name="hostname")
    constant(value=" ")
    property(name="syslogtag" position.from="1" position.to="32")
    property(name="msg" spifno1stsp="on" )
    property(name="msg")
    }

等同于

1
2
3

template(name="forwardFormat" type="string"
         string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"
        )

写入MySQL

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

template(name="StdSQLformat" type="list" option.sql="on") {
        constant(value="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)")
        constant(value=" values ('")
        property(name="msg")
        constant(value="', ")
        property(name="syslogfacility")
        constant(value=", '")
        property(name="hostname")
        constant(value="', ")
        property(name="syslogpriority")
        constant(value=", '")
        property(name="timereported" dateFormat="mysql")
        constant(value="', '")
        property(name="timegenerated" dateFormat="mysql")
        constant(value="', ")
        property(name="iut")
        constant(value=", '")
        property(name="syslogtag")
        constant(value="')")
        }

等同于

1
2
3

template(name="stdSQLformat" type="string" option.sql="on"
         string="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')"
        )

JSON

用于一些RESTful的API，比如ElasticSearch

1
2
3
4
5
6
7
8
9
10

template(name="outfmt" type="list" option.jsonf="on") {
         property(outname="@timestamp" name="timereported" dateFormat="rfc3339" format="jsonf")
         property(outname="host" name="hostname" format="jsonf")
         property(outname="severity" name="syslogseverity" caseConversion="upper" format="jsonf" datatype="number")
         property(outname="facility" name="syslogfacility" format="jsonf" datatype="number")
         property(outname="syslog-tag" name="syslogtag" format="jsonf")
         property(outname="source" name="app-name" format="jsonf" onEmpty="null")
         property(outname="message" name="msg" format="jsonf")

 }

产生的数据类似于

1

{"@timestamp":"2018-03-01T01:00:00+00:00", "host":"172.20.245.8", "severity":7, "facility":20, "syslog-tag":"tag", "source":"tag", "message":" msgnum:00000000:"}

整理后

1
2
3
4
5
6
7
8
9

{
  "@timestamp": "2018-03-01T01:00:00+00:00",
  "host": "172.20.245.8",
  "severity": 7,
  "facility": 20,
  "syslog-tag": "tag",
  "source": "tag",
  "message": " msgnum:00000000:"
}

如果 app-name 为空，当设置 onEmpty=”null” 时，输出如下

1

{"@timestamp":"2018-03-01T01:00:00+00:00", "host":"172.20.245.8", "severity":7, "facility":20, "syslog-tag":"tag", "source":null, "message":" msgnum:00000000:"}

为omfile创建动态文件名

可以为不同的hosts的消息创建不同的文件

1

template (name="DynFile" type="string" string="/var/log/system-%HOSTNAME%.log")

保留的模板名字

不要使用 “RSYSLOG_” 开始的模板名。一些预定义的模板如下

RSYSLOG_TraditionalFileFormat - The “old style” default log file format with low-precision timestamps.

1
2

template(name="RSYSLOG_TraditionalFileFormat" type="string"
     string="%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n")

RSYSLOG_FileFormat - A modern-style logfile format similar to TraditionalFileFormat, both with high-precision timestamps and timezone information.

1
2
3
4
5
6
7
8
9
10

template(name="RSYSLOG_FileFormat" type="list") {
     property(name="timereported" dateFormat="rfc3339")
     constant(value=" ")
     property(name="hostname")
     constant(value=" ")
     property(name="syslogtag")
     property(name="msg" spifno1stsp="on")
     property(name="msg" droplastlf="on")
     constant(value="\n")
}

RSYSLOG_TraditionalForwardFormat - The traditional forwarding format with low-precision timestamps. Most useful if you send messages to other syslogd’s or rsyslogd below version 3.12.5.

1
2

template(name="RSYSLOG_TraditionalForwardFormat" type="string"
     string="<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%")

RSYSLOG_SysklogdFileFormat - Sysklogd compatible log file format. If used with options: $SpaceLFOnReceive on, $EscapeControlCharactersOnReceive off, $DropTrailingLFOnReception off, the log format will conform to sysklogd log format.

1
2

template(name="RSYSLOG_SysklogdFileFormat" type="string"
     string="%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg%\n")

RSYSLOG_ForwardFormat - a new high-precision forwarding format very similar to the traditional one, but with high-precision timestamps and timezone information. Recommended to be used when sending messages to rsyslog 3.12.5 or above.

1
2

template(name="RSYSLOG_ForwardFormat" type="string"
     string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%")

RSYSLOG_SyslogProtocol23Format - the format specified in IETF’s internet-draft ietf-syslog-protocol-23, which is very close to the actual syslog standard RFC5424 (we couldn’t update this template as things were in production for quite some time when RFC5424 was finally approved). This format includes several improvements. You may use this format with all relatively recent versions of rsyslog or syslogd.

1
2

template(name="RSYSLOG_SyslogProtocol23Format" type="string"
     string="<%PRI%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n")

RSYSLOG_DebugFormat - a special format used for troubleshooting property problems. This format is meant to be written to a log file. Do not use for production or remote forwarding.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

template(name="RSYSLOG_DebugFormat" type="list") {
     constant(value="Debug line with all properties:\nFROMHOST: '")
     property(name="fromhost")
     constant(value="', fromhost-ip: '")
     property(name="fromhost-ip")
     constant(value="', HOSTNAME: '")
     property(name="hostname")
     constant(value="', PRI: '")
     property(name="pri")
     constant(value=",\nsyslogtag '")
     property(name="syslogtag")
     constant(value="', programname: '")
     property(name="programname")
     constant(value="', APP-NAME: '")
     property(name="app-name")
     constant(value="', PROCID: '")
     property(name="procid")
     constant(value="', MSGID: '")
     property(name="msgid")
     constant(value="',\nTIMESTAMP: '")
     property(name="timereported")
     constant(value="', STRUCTURED-DATA: '")
     property(name="structured-data")
     constant(value="',\nmsg: '")
     property(name="msg")
     constant(value="'\nescaped msg: '")
     property(name="msg" controlcharacters="drop")
     constant(value="'\ninputname: ")
     property(name="inputname")
     constant(value=" rawmsg: '")
     property(name="rawmsg")
     constant(value="'\n$!:")
     property(name="$!")
     constant(value="\n$.:")
     property(name="$.")
     constant(value="\n$/:")
     property(name="$/")
     constant(value="\n\n")
}

RSYSLOG_WallFmt - Contains information about the host and the time the message was generated and at the end the syslogtag and message itself.

1
2

template(name="RSYSLOG_WallFmt" type="string"
     string="\r\n\7Message from syslogd@%HOSTNAME% at %timegenerated% ...\r\n%syslogtag%%msg%\n\r")

RSYSLOG_StdUsrMsgFmt - The syslogtag followed by the message is returned.

1
2

template(name="RSYSLOG_StdUsrMsgFmt" type="string"
     string=" %syslogtag%%msg%\n\r")

RSYSLOG_StdDBFmt - Generates a insert command with the message properties, into table SystemEvents for a mysql database.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

template(name="RSYSLOG_StdDBFmt" type="list") {
     constant(value="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)")
     constant(value=" values ('")
     property(name="msg")
     constant(value="', ")
     property(name="syslogfacility")
     constant(value=", '")
     property(name="hostname")
     constant(value="', ")
     property(name="syslogpriority")
     constant(value=", '")
     property(name="timereported" dateFormat="date-mysql")
     constant(value="', '")
     property(name="timegenerated" dateFormat="date-mysql")
     constant(value="', ")
     property(name="iut")
     constant(value=", '")
     property(name="syslogtag")
     constant(value="')")
}

RSYSLOG_StdPgSQLFmt - Generates a insert command with the message properties, into table SystemEvents for a pgsql database.

1
2
3
4
5

template(name="RSYSLOG_StdPgSQLFmt" type="string"
string="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime,
     ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%',
     %syslogpriority%, '%timereported:::date-pgsql%', '%timegenerated:::date-pgsql%', %iut%,
     '%syslogtag%')")

RSYSLOG_spoofadr - Generates a message containing nothing more than the ip address of the sender.

1

template(name="RSYSLOG_spoofadr" type="string" string="%fromhost-ip%")

RSYSLOG_StdJSONFmt - Generates a JSON structure containing the message properties.

1
2
3
4
5

template(name="RSYSLOG_StdJSONFmt" type="string"
     string="{\"message\":\"%msg:::json%\",\"fromhost\":\"%HOSTNAME:::json%\",\"facility\":
             \"%syslogfacility-text%\",\"priority\":\"%syslogpriority-text%\",\"timereported\":
             \"%timereported:::date-rfc3339%\",\"timegenerated\":
             \"%timegenerated:::date-rfc3339%\"}")

基本结构

消息会通过 input module进入 rsyslog，然后传递给 ruleset，每一个ruleset 会包含 condition，但条件满足时，执行action。

原则

• inputs 提交消息给 rulesets
  • 如果没有ruleset匹配，将应用默认 ruleset
• 默认会有一条 ruleset (RSYSLOG_DefaultRuleset)
• 用户可定义额外的 rulesets
• 每一个ruleset可包含0或多个 rules
  • 当ruleset 中为0个rules，这个set就没有意义
• 一条rule 由一个filter和一个action list
• filters 用来判断和流程控制
• 如果filter被匹配，执行对应的action list
• 在一个ruleset内，rules被顺序执行
• 所有的rules都将被执行，无论之前是不是已经被命中。如果需要停止执行，则必须声明 discard action。所有后续rules将立即中止
• 一个 action list 可包含一个或多个动作
• action list 内不能再定义 filters
• 多个 action使用 “&” 连接
• actions consist of the action call itself (e.g. “:omusrmsg:”) as well as all action-defining configuration statements ($Action… directives)
• 如果要使用 legacy 语法, $Action… directives 必有在调用前声明
• 一些指令会和它之前的指令执行结果有关，但有些不是，需要查文档，但需要注意的是，文档也不一定准确。- -b
• v5版本最好被淘汰掉，用v7及以后版本
• legacy 配置不会影响 RainerScript objects (e.g. actions).

配置文件

rsyslog.conf 以及它 include的一些文件，可以使用 -f 参数改变

Statement 类型

Rsyslog 支持三种语法，见 《学习1》

• sysklogd
• legacy rsyslog - v6及之前版本支持，现在的版本也能兼容。每个语句前使用$ 声明。有些插件和特性级需要该方法，因为不是所有的插件都升级到新版本了。。。
• RainerScript - 之后主要讨论的内容

前两种格式和行相关，RainerScript 使用块，和行无关

注释

• # - 单行
• /* … */ - 多行

执行顺序

所有指令从上到下执行

流程控制

• Control Structures
• Filter Conditions

数据处理

使用 set, unset and reset 处理数据，参考 documented here in detail

Inputs

每一个input都需要一个input模块，参见 rsyslog modules

Outputs

即action, 一些常见的module已预先载入，比如 写文件，有些需要声明

Rulesets and Rules

ruleset会bound到一个input，参见rsyslog rulesets.

典型的ruleset如下

1
2
3
4
5

ruleset(name="rulesetname") {
    action(type="omfile" file="/path/to/file")
    action(type="..." ...)
    /* and so on... */
}

sysklogd 格式

​ 这个使用syslog来说明，rsyslog也使用同样的格式

说明

​ 在 *nix 系统上，syslogd(8)的配置文件为 syslog.conf

​ 每一条规则分为 selector 和 action 两部分，由一个以上的空格或tab分割

​ 用为注释，\ 用来合并多选

SELECTORS

​ selector 可分为 facility 和 priority 两部分，用 “.” 分割，区分大小写，也可以用 /usr/include/syslog.h 中定义的数字来表示。

​ facility 用来表示产生日志的子系统，可分为 auth, authpriv, cron, daemon, ftp, kern, lpr, mail, mark, news, security (same as auth，已废弃), syslog, user, uucp and local0 through local7 。任何人都可以发送消息到除了 kern 的任何 facility 。

​ priority 从低到高分为 debug, info, notice, warning, warn (same as warning，已废弃), err, error (same as err，已废弃), crit, alert, emerg, panic (same as emerg，已废弃)。

​ 默认当消息priority等于或高于配置的级别时，syslogd会动作。也可以使用 “=” 来只指定某一级别。

​ facility 和 priority 可以用 “*” 来代表所有。none 代表不匹配任何项。

​ 可以同时使用多个 facility，用”,”分割，但同时只能对应一个 priority 。

​ 可以同时使用多个 selectors 来对应一个action，使用”;”分割，执行顺序为从左到右，后者会覆盖前者，可用来作某些 exclude的操作。

​ 可以在 priority 前加 “!” 表示忽略等于或大于该级别的消息，可以和 “=” 一直使用。

ACTIONS

Regular File

​ 记录到文件，可以使用相对路径，但非常不建议这么做。

Named Pipes

​ 可以发送到 named pipes (fifos)，一般用来debug，需要使用 mkfifo 命令先创建 fifo 设备。

Terminal and Console

​ 可以发送到 /dev/console 或 tty。

1
2
3
4

# The tcp wrapper logs with mail.info, we display
# all the connections on tty12
#
mail.=info     /dev/tty12

Remote Machine

​ 可以从远程接受消息，或发送到远端机器。远程默认不会再转发消息。如果要发送消息到远端，格式为 @remote_hostname

​ 作为接受者，可以使用 named pipe 处理接受到消息。

1

*.*       @finlandia

List of Users

​ 通常，critical 消息会发送给本机root，也可以配置多个 user 接受消息，使用”,”分割，如果这些user在线的话，将会在terminal上收到消息。

1
2
3
4

# Messages of the priority alert will be directed
# to the operator
#
*.alert      root,joey

Everyone logged on

​ 一些紧急的消息会发送给所有在线用户，使用 “*” 来启用这个功能。

1
2
3

# Emergency messages will be displayed using wall
#
*.=emerg      *

例子

1
2
3

# Store critical stuff in critical
#
*.=crit;kern.none   /var/adm/critical

发送crit 到 /var/adm/critical，除了 kern的消息。

1
2
3
4
5
6
7
8

# Kernel messages are stored in the kernel file,
# critical messages and higher ones also go
# to another host and to the console
#
kern.*      /var/adm/kernel
kern.crit     @finlandia
kern.crit     /dev/console
kern.info;kern.!err   /var/adm/kernel-info

第四行，发磅kern的info以上的消息，除了err

1
2
3

# Write all mail related logs to a file
#
mail.*;mail.!=info   /var/adm/mail

1
2
3

# Log all mail.info and news.info messages to info
#
mail,news.=info    /var/adm/info

匹配来自mail和news的info消息

1
2
3
4

# Log info and notice messages to messages file
#
*.=info;*.=notice;\
mail.none /var/log/messages

1
2
3
4

# Log info messages to messages file
#
*.=info;\
mail,news.none  /var/log/messages

​ 年初搭建了一套ELK来收集现网中网络设备日志，通过设备发送syslog到各idc的rsyslog中转服务器，再转发到kafka，用logstash写入es。因为之前没有做过滤，跑了半年，发现日志量有点大，因为设备型号比较多，cisco/hw/zte/ruijie/h3c/dptech/f5，发的日志也千奇百怪，里面有很多无用的message，对于查找也存在干扰，所以决定好好的对日志进行一下过滤。

​ 过滤可以布置在两个点，一是rsyslog服务器上，二是logstash。所以先从服务器入手，好好的学习一下rsyslog的配置。主要参考文档

配置格式

basic

​ 又叫 sysklogd 格式。基本配置，一句一行，用了多年。

1
2

mail.info /var/log/mail.log
mail.err @@server.example.net

advanced

​ 又叫 RainerScript 格式。v6开始支持，v7有性能问题，目前版本(v8)正常使用。

• 更多的参数，更好的控制
• 使用块结构
• 容易编写
• 可以使用include

1

mail.err action(type="omfwd" protocol="tcp" queue.type="linkedList")

obsolete legacy

​ 又叫 legacy 格式。如其名，已被 obsoleted。

1

$ModLoad module-name

将配置转为 advanced

不要过度转换

1

mail.info   /var/log/maillog

​ 一些basic的配置不需要转换，保持就好。如果要转的话，可以转成

1
2
3

if prifilt("mail.info") then {
     action(type="omfile" file="/var/log/maillog")
}

或

1

if prifilt("mail.info") then action(type="omfile" file="/var/log/maillog")

或

1

mail.info action(type="omfile" file="/var/log/maillog")

转换module

​ legacy语法在多次引用时容易出现问题

1
2
3
4

$ModLoad module-name

$ModLoad imtcp
$InputTCPMaxSession 500

​ 可以转换为

1
2
3

module(load="module-name")

module(load="imtcp" maxSessions="500")

转换action

1

filter action

​ 转换action并不一定要转换filter，因为不同格式的filter都可以和action一起工作

​ 下面列了一些从basic到advance的action转换

​ 例如

1
2
3
4
5
6
7
8

OLD: :hostname, contains, "remote-sender" @@central
NEW: :hostname, contains, "remote-sender" action(type="omfwd" target="central" protocol="tcp")

OLD: if $msg contains "error" then @central
NEW: if $msg contains "error" then action(type="omfwd" target="central" protocol="udp")

OLD: *.emerg :omusrmsg:*
NEW: *.emerg action(type="omusrmsg" users="*")

带多个参数的action转换

​ 在basic中，多个action可以使用 & 来连接。

1
2

*.error /var/log/errorlog
&       @remote

​ 在advanced中，使用block来表示

1
2
3
4

*.error {
        action(type="omfile" file="/var/log/errorlog")
        action(type="omfwd" target="remote" protocol="udp")
}

​ 使用 stop 或 ~ 来停止动作

1
2
3
4
5

:msg, contains, "error" @remote
& ~

:msg, contains, "error" @remote
& stop

1
2
3
4

:msg, contains, "error" {
        action(type="omfwd" target="remote" protocol="udp")
        stop
}

或

1
2
3
4

if $msg contains "error" then {
        action(type="omfwd" target="remote" protocol="udp")
        stop
}

版本

1
2
3
4
5

~ cat /etc/centos-release
CentOS Linux release 7.6.1810 (Core)
~  yum list installed | grep filebeat
filebeat.x86_64                        7.10.2-1                     @elastic-7.x

需求

想让各出口的路由器，通过netflow、netstream将流量送到es保存起来进行分析。之前测试是直接用filebeat output到es的，但考虑到性能问题，决定先将数据送到kafka，通过logstash进行处理再送入es。

因为出口链路较多，设计不同的出口数据，送入不同的filebeat端口，以不同的topic存入kafka，方便管理。

当然，不同流量也可以采用相同的topic，后期在es上根据observer或tag来区分。

问题

按官网说明，在module中加上 fields.log_topic

1
2
3
4
5
6
7
8
9
10
11

~ grep -v "^\s*#" /etc/filebeat/modules.d/netflow_3055.yml | grep -v ^$
- module: netflow
  log:
    enabled: true
    var:
      netflow_host: 0.0.0.0
      netflow_port: 3055
      internal_networks:
        - 10.0.0.0/8
      fields:
        log_topic: ipnce

在filebeat.yml中指定topic

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

~ grep -v "^\s*#" /etc/filebeat/filebeat.yml | grep -v ^$
logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644
logging.to_syslog: false
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
output:
  kafka:
    output.kafka:
      hosts: ["kafka-1:9092", "kafka-2:9092", "kafka-3:9092"]
      topic: "netstream-%{[fields.log_topic]}"
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

按以上配置，一直报错

1

Dropping event: no topic could be selected

应该是fields中，没有log_topic值，虽然在module中已经设置了，但没有生效。问了google，有些说需要在配置中加上

1

fields_under_root: true

依然无效

解决

测试了几天，实在是找不头绪，只能使用全局topic传入kafka，在es上观察数据后，发现netflow的数据在存入es后，有一些特定的字段，比如

1

ecs.version: 1.5.0

这些字段应该是filebeat默认的模板通过processors加上去的，于是查看了netflow模块的默认模板，果然在processors中发现了ecs.version字段，该yml文件中，还有一些关于module中的变量替换。于是想通过在这里引用module中的vars，设定log_topic。

在末尾添加 log_topic

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

~ grep -v "^\s*#" /usr/share/filebeat/module/netflow/log/config/netflow.yml | grep -v ^$
type: netflow
protocols: [v1, v5, v6, v7, v8, v9, ipfix]
host: '{{.netflow_host}}:{{.netflow_port}}'
max_message_size: '{{.max_message_size}}'
expiration_timeout: '{{.expiration_timeout}}'
queue_size: {{.queue_size}}
{{if .timeout}}
timeout: '{{.timeout}}'
{{end}}
{{if .read_buffer}}
read_buffer: '{{.read_buffer}}'
{{end}}
{{ if .custom_definitions}}
custom_definitions:
{{range .custom_definitions}}
- '{{ . }}'
{{end}}
{{end}}
{{ if .detect_sequence_reset}}
detect_sequence_reset: {{.detect_sequence_reset}}
{{end}}
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}
processors:
  - add_fields:
      target: ''
      fields:
        ecs.version: 1.5.0
        log_topic: {{ .log_topic  }}

修改module，添加log_topic变量

1
2
3
4
5
6
7
8
9
10
11
12

~ grep -v "^\s*#" /etc/filebeat/modules.d/netflow_3055.yml | grep -v ^$
- module: netflow
  log:
    enabled: true
    var:
      netflow_host: 0.0.0.0
      netflow_port: 3055
      tags:
        - ipn_ce
      log_topic: ipn_ce
      internal_networks:
        - 10.0.0.0/8

重启filebeat后，可根据log_topic在kafka中正常生成topic

1
2
3

~ kafka-topics.sh --list --zookeeper localhost:2181
__consumer_offsets
netstream-ipn_ce

能用了，为什么官方配置不行的问题，也不想深究了 。。。