rsyslog学习4 -- Templates模板

发表于 |
字数统计: 4.2k | 阅读时长 ≈ 21

说明

模板用来指定格式，动态文件名生成。所有的output都会用到模板。不光是文件输出，写入到数据库也需要合适的模块来组成语句。如果没有指定模板，会调用默认的模块，可以在rsconf.c中搜索”template_”查看。

模板使用 template() 声明，在legacy中，使用 $template

模板使用的属性可查看 rsyslog properties reference

模板执行

虽然没有标准化的日志格式，但一般来说，模板应该包含RFC5424定义的HEADER，rsyslog解析会用到。

如果一条消息，没有HOSTANME，没有TAG，内容为 “this:is a message”，刚会被解析成

1
2
TAG:this:
MSG:is a message

The template() statement

这是一个static声明，意味着所有的模板都在配置中被定义，所以模板也不会被if或其它声明所改变。

1
template(parameters) { list-descriptions }

每一个模板都要有一个 name ，必有唯一，否则会导致行为不可知。

每一个模板都有一个参数 type，不同的type 可以使能不同的方法解析不同的模板内容。

模板不会影响output插件。

type有以下几种

  • list
  • subtree
  • string
  • plugin

List

此模板一系列的constant和variable 组成，放在{ }中。这种类型常用在结构化的输出，如ommongodb，也能工作在文本型的输出。建议在需要复杂的属性替换中使用此模板，因为基本list的语法比基于string的方法清晰。

constant包括一些文本和 property ，可以对其作一些修改（如转换大小写）

1
2
3
4
5
6
7
template(name="tpl1" type="list") {
     constant(value="Syslog MSG is: '")
     property(name="msg")
     constant(value="', ")
     property(name="timereported" dateFormat="rfc3339" caseConversion="lower")
     constant(value="\n")
     }

constant statement

主要用于基于文本的输出，可以添加一下文字。比如需要添加换行符

1
2
3
4
template(name="outfmt" type="list") {
    property(name="$!usr!msgnum")
    constant(value="\n")
}

特殊字符

  • \ - single backslash
  • \n - LF
  • \ooo - (three octal digits) - represents a character with this octal numerical value (e.g. \101 equals “A”). Note that three octal digits must be given (in contrast to some languages, where between one and three are valid). While we support octal notation, we recommend to use hex notation as this is better known.
  • \xhh - (where h is a hex digit) - represents a character with this hexadecimal numerical value (e.g. \x41 equals “A”). Note that two hexadecimal digits must be given (in contrast to some languages where either one or two are valid).
  • … some others … list needs to be extended

如果在不支持的字符前使用\，会导致问题。

为了帮助对基于文本的输出和结构化输出使用相同的模板，在为结构化输出创建名称/值树时，将忽略没有“outname”参数的常量文本。 因此，如果您想提供一些常量文本，例如 对于 mongodb，您必须包含一个 outname，如下所示：

1
2
3
4
template(name="outfmt" type="list") {
    property(name="$!usr!msgnum")
    constant(value="\n" outname="IWantThisInMyDB")
}

要生成一个常量json字段，可以使用format参数，如本例所示

1
2
3
4
template(name="outfmt" type="list" option.jsonf="on") {
          property(outname="message" name="msg" format="jsonf")
          constant(outname="@version" value="1" format="jsonf")
}

在本例中，constant语句将生成@version: 1。注意，要做到这一点，必须同时给出值和格式参数。

constant语句支持以下参数:

  • value - 值
  • outname - 输出字段名称(用于结构化输出)
  • format - 可以为空或 jsonf

Property statement

可以使用option来截取或更改，支持下列参数

  • name - 属性名

  • outname - output字段名 (用于结构化输出)

  • dateformat - 日期格式 (只用于时间相关属性)，参考 文档 。TODO: 目前，属性替换相关文档只包含了文本模板的一些option，这些option不同于非文本模板。比如，在文本模板中，日期格式option，会以”date-“开头，而在属性语句中不需要（比如”date-year”和”year”）。究其原因，在文本模板中，必须告诉这个这个声明所应用的地方，而在属性中名称中，这点已声明。

    可以自己定义格式，比如 YYYY-MM-DD:

1
2
3
4
5
property(name="timereported" dateformat="year")
constant(value="-")
property(name="timereported" dateformat="month")
constant(value="-")
property(name="timereported" dateformat="day")

  • date.inUTC - date shall be shown in UTC (需要损失一点性能) Available since 8.18.0.

  • caseconversion - 转换大小定， 取值为 “lower” and “upper”

  • controlcharacters - 如何处理控制字符。 可使用的值 “escape”, which escapes them, “space”, which replaces them by a single space, and “drop”, which simply removes them from the string.

  • securepath - 用于在动态文件模板中创建合适的文件路径

  • format - specify format on a field basis. Supported values are:

    • “csv” for use when csv-data is generated
    • “json” which formats proper json content (but without a field header)
    • “jsonf” which formats as a complete json field
    • “jsonr” which avoids double escaping the value but makes it safe for a json field
    • “jsonfr” which is the combination of “jsonf” and “jsonr”.

  • position.from - 获取从该位置起之后的字串 (从1开始)

  • position.to - 获取位置之前的字串

  • position.relativeToEnd - 获取到末性的相对偏移的字串. (available since rsyslog v7.3.10)

  • fixedwidth - 更改position.to行为，当原始字串长度小于时，用空格填充，取值“on” 或 “off” (default) (available since rsyslog v8.13.0)

  • compressspace - 压缩空格，将多个空格压缩为一个，这个特性会在很晚执行，所以不会影响position.from或position.to的执行。(available since v8.18.0).

  • field.number - obtain this field match

  • field.delimiter - decimal value of delimiter character for field extraction

  • regex.expression - expression to use

  • regex.type - either ERE or BRE

  • regex.nomatchmode - what to do if we have no match

  • regex.match - match to use

  • regex.submatch - submatch to use

  • droplastlf - drop a trailing LF, if it is present

  • mandatory - 强制一个字段。如果设为 “on”，该字段始终会传递给结构输出，即使不存在。如果 “off” (the default)，空字段不会被传递. 这在输出支持动态 schemas 比较有用(like ommongodb).

  • spifno1stsp - expert options for RFC3164 template processing

  • datatype - 在jsonf 格式中生效; 允许为原始的字串指定数据类型。有时你在延时时需要用到数字或布尔类型，可以用这个option去更改。

    • number - value is treated as a JSON number and not enclosed in quotes.

      If the property is empty, the value 0 is generated.

    • string - 字串会被引号包裹

    • auto - value is treated as number if numeric and as string otherwise.

      只对整数有效

    • bool - the value is treated as boolean. If it is empty or 0, it will generate “false”, else “true”.

    默认使用string类型. This is a feature of rsyslog 8.1905.0 or later.

  • onEmpty - 在jsonf 格式中生效; 指定如何处理空值. Possible values are:

    • keep - emit the empty element
    • skip - completely ignore the element, do not emit anything
    • null - emit a JSON ‘null’ value

    If not specified, ‘keep’ is assumed. This is a feature of rsyslog 8.1905.0 or later.

Subtree

7.1.4后支持

基于a complete (CEE) subtree 创建模板。用于输出层次给的结构，比如ommongodb。要使用该模板，必须在参数中声明。比如

1
2
3
4
5
# 包含所有 CEE 数据
template(name=”tpl1” type=”subtree” subtree=”$!”)  

# 只包含以 $!usr!tpl2 开始的数据
template(name=”tpl2” type=”subtree” subtree=”$!usr!tpl2”)

subtree类型与能用于文件输出，比如 omfile，但没有处理常量文本的能力。一般用来用于调试

一个例子

1
2
3
set $!usr!tpl2!msg = $msg;
set $!usr!tpl2!dataflow = field($msg, 58, 2);
template(name="tpl2" type="subtree" subtree="$!usr!tpl2")

String

非常类似于legacy模板声明。使用强制参数string。一个模板字串由常量文本和变量替换完成，对于没有复杂操作要求的消息，使用String-base的模板足够了。

1
2
3
template(name="tpl3" type="string"
         string="%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
        )

在“%”之间的内容会被property replacer解释。简而言之，包含了所要处理的property和option，这和list模板中的property对象类似。

在 “%”之外的就是常量文本，在上面的例子，我们使用了空格来分割，最后加上分行

可以使用\来打印控制字符。

Plugin

模板是于plugin产生（也叫做 strgen或 string generator）。格式是固定的，不可更改。它的性能是非常好（并不是说普通模板慢，只是在一些极端环境中，会有所差别）。使用前必须先载入plugin

1
template(name="tpl4" type="plugin" plugin="mystrgen")

options

option是可选的。

option.sql - 格式化为MySQL语句. 比如在”‘“前添加 \，请注意在MySQL的配置中,NO_BACKSLASH_ESCAPES 模式需要为off (this is the default).

option.stdsql - 格式化为 sql server 语句. 将 ‘ 改为两个 ‘’. You must use stdsql together with MySQL if in MySQL configuration the NO_BACKSLASH_ESCAPES is turned on.

option.json - format the string suitable for a json statement. This will replace single quotes (“’”) by two single quotes (“’‘”) inside each field.

option.jsonf - format the string as JSON object. 这意味着在所有非终结符属性和常量之间都将添加一个前导和后导的花括号{以及一个逗号。

option.casesensitive - treat property name references as case sensitive. 默认为 “off”, 所有模板在定时，属性名都会转为小写. With this option turned “on”, property names are looked up as defined in the template. Use this option if you have JSON ($!*), local (!.*), or global ($!\\*) properties which contain uppercase letters. The normal Rsyslog properties are case-insensitive, so this option is not needed for properly referencing those properties.

option.sql, option.stdsql, and option.json 是相互排斥的。

在使用sql or stdsql 向数据库写入时，要注意确保使用正确格式，防止被注入, If you choose the wrong one, you are still vulnerable to sql injection. The sql option 可以使用在文件中 - 比如想把它们导入到别的机器数据库中。如果没有必要，就不要使用这些选项同，会带来性能的损耗

默认的数据库模板使用 sql option。

1
2
template (name="TraditionalFormat" type="string"
string="%timegenerated% %HOSTNAME% %syslogtag%%msg%\\n")

例子

写入文件

1
2
3
4
5
6
7
8
9
10
template(name="FileFormat" type="list") {
    property(name="timestamp" dateFormat="rfc3339")
    constant(value=" ")
    property(name="hostname")
    constant(value=" ")
    property(name="syslogtag")
    property(name="msg" spifno1stsp="on" )
    property(name="msg" droplastlf="on" )
    constant(value="\n")
    }

也可以写成

1
2
3
template(name="FileFormat" type="string"
         string= "%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
        )

注意，string模板必须写成一行

转发到远端（RFC3164）

1
2
3
4
5
6
7
8
9
10
11
12
template(name="ForwardFormat" type="list") {
    constant(value="<")
    property(name="pri")
    constant(value=">")
    property(name="timestamp" dateFormat="rfc3339")
    constant(value=" ")
    property(name="hostname")
    constant(value=" ")
    property(name="syslogtag" position.from="1" position.to="32")
    property(name="msg" spifno1stsp="on" )
    property(name="msg")
    }

等同于

1
2
3
template(name="forwardFormat" type="string"
         string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"
        )

写入MySQL

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
template(name="StdSQLformat" type="list" option.sql="on") {
        constant(value="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)")
        constant(value=" values ('")
        property(name="msg")
        constant(value="', ")
        property(name="syslogfacility")
        constant(value=", '")
        property(name="hostname")
        constant(value="', ")
        property(name="syslogpriority")
        constant(value=", '")
        property(name="timereported" dateFormat="mysql")
        constant(value="', '")
        property(name="timegenerated" dateFormat="mysql")
        constant(value="', ")
        property(name="iut")
        constant(value=", '")
        property(name="syslogtag")
        constant(value="')")
        }

等同于

1
2
3
template(name="stdSQLformat" type="string" option.sql="on"
         string="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')"
        )

JSON

用于一些RESTful的API，比如ElasticSearch

1
2
3
4
5
6
7
8
9
10
template(name="outfmt" type="list" option.jsonf="on") {
         property(outname="@timestamp" name="timereported" dateFormat="rfc3339" format="jsonf")
         property(outname="host" name="hostname" format="jsonf")
         property(outname="severity" name="syslogseverity" caseConversion="upper" format="jsonf" datatype="number")
         property(outname="facility" name="syslogfacility" format="jsonf" datatype="number")
         property(outname="syslog-tag" name="syslogtag" format="jsonf")
         property(outname="source" name="app-name" format="jsonf" onEmpty="null")
         property(outname="message" name="msg" format="jsonf")

 }

产生的数据类似于

1
{"@timestamp":"2018-03-01T01:00:00+00:00", "host":"172.20.245.8", "severity":7, "facility":20, "syslog-tag":"tag", "source":"tag", "message":" msgnum:00000000:"}

整理后

1
2
3
4
5
6
7
8
9
{
  "@timestamp": "2018-03-01T01:00:00+00:00",
  "host": "172.20.245.8",
  "severity": 7,
  "facility": 20,
  "syslog-tag": "tag",
  "source": "tag",
  "message": " msgnum:00000000:"
}

如果 app-name 为空，当设置 onEmpty=”null” 时，输出如下

1
{"@timestamp":"2018-03-01T01:00:00+00:00", "host":"172.20.245.8", "severity":7, "facility":20, "syslog-tag":"tag", "source":null, "message":" msgnum:00000000:"}

为omfile创建动态文件名

可以为不同的hosts的消息创建不同的文件

1
template (name="DynFile" type="string" string="/var/log/system-%HOSTNAME%.log")

保留的模板名字

不要使用 “RSYSLOG_” 开始的模板名。一些预定义的模板如下

RSYSLOG_TraditionalFileFormat - The “old style” default log file format with low-precision timestamps.

1
2
template(name="RSYSLOG_TraditionalFileFormat" type="string"
     string="%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n")

RSYSLOG_FileFormat - A modern-style logfile format similar to TraditionalFileFormat, both with high-precision timestamps and timezone information.

1
2
3
4
5
6
7
8
9
10
template(name="RSYSLOG_FileFormat" type="list") {
     property(name="timereported" dateFormat="rfc3339")
     constant(value=" ")
     property(name="hostname")
     constant(value=" ")
     property(name="syslogtag")
     property(name="msg" spifno1stsp="on")
     property(name="msg" droplastlf="on")
     constant(value="\n")
}

RSYSLOG_TraditionalForwardFormat - The traditional forwarding format with low-precision timestamps. Most useful if you send messages to other syslogd’s or rsyslogd below version 3.12.5.

1
2
template(name="RSYSLOG_TraditionalForwardFormat" type="string"
     string="<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%")

RSYSLOG_SysklogdFileFormat - Sysklogd compatible log file format. If used with options: $SpaceLFOnReceive on, $EscapeControlCharactersOnReceive off, $DropTrailingLFOnReception off, the log format will conform to sysklogd log format.

1
2
template(name="RSYSLOG_SysklogdFileFormat" type="string"
     string="%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg%\n")

RSYSLOG_ForwardFormat - a new high-precision forwarding format very similar to the traditional one, but with high-precision timestamps and timezone information. Recommended to be used when sending messages to rsyslog 3.12.5 or above.

1
2
template(name="RSYSLOG_ForwardFormat" type="string"
     string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%")

RSYSLOG_SyslogProtocol23Format - the format specified in IETF’s internet-draft ietf-syslog-protocol-23, which is very close to the actual syslog standard RFC5424 (we couldn’t update this template as things were in production for quite some time when RFC5424 was finally approved). This format includes several improvements. You may use this format with all relatively recent versions of rsyslog or syslogd.

1
2
template(name="RSYSLOG_SyslogProtocol23Format" type="string"
     string="<%PRI%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n")

RSYSLOG_DebugFormat - a special format used for troubleshooting property problems. This format is meant to be written to a log file. Do not use for production or remote forwarding.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
template(name="RSYSLOG_DebugFormat" type="list") {
     constant(value="Debug line with all properties:\nFROMHOST: '")
     property(name="fromhost")
     constant(value="', fromhost-ip: '")
     property(name="fromhost-ip")
     constant(value="', HOSTNAME: '")
     property(name="hostname")
     constant(value="', PRI: '")
     property(name="pri")
     constant(value=",\nsyslogtag '")
     property(name="syslogtag")
     constant(value="', programname: '")
     property(name="programname")
     constant(value="', APP-NAME: '")
     property(name="app-name")
     constant(value="', PROCID: '")
     property(name="procid")
     constant(value="', MSGID: '")
     property(name="msgid")
     constant(value="',\nTIMESTAMP: '")
     property(name="timereported")
     constant(value="', STRUCTURED-DATA: '")
     property(name="structured-data")
     constant(value="',\nmsg: '")
     property(name="msg")
     constant(value="'\nescaped msg: '")
     property(name="msg" controlcharacters="drop")
     constant(value="'\ninputname: ")
     property(name="inputname")
     constant(value=" rawmsg: '")
     property(name="rawmsg")
     constant(value="'\n$!:")
     property(name="$!")
     constant(value="\n$.:")
     property(name="$.")
     constant(value="\n$/:")
     property(name="$/")
     constant(value="\n\n")
}

RSYSLOG_WallFmt - Contains information about the host and the time the message was generated and at the end the syslogtag and message itself.

1
2
template(name="RSYSLOG_WallFmt" type="string"
     string="\r\n\7Message from syslogd@%HOSTNAME% at %timegenerated% ...\r\n%syslogtag%%msg%\n\r")

RSYSLOG_StdUsrMsgFmt - The syslogtag followed by the message is returned.

1
2
template(name="RSYSLOG_StdUsrMsgFmt" type="string"
     string=" %syslogtag%%msg%\n\r")

RSYSLOG_StdDBFmt - Generates a insert command with the message properties, into table SystemEvents for a mysql database.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
template(name="RSYSLOG_StdDBFmt" type="list") {
     constant(value="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)")
     constant(value=" values ('")
     property(name="msg")
     constant(value="', ")
     property(name="syslogfacility")
     constant(value=", '")
     property(name="hostname")
     constant(value="', ")
     property(name="syslogpriority")
     constant(value=", '")
     property(name="timereported" dateFormat="date-mysql")
     constant(value="', '")
     property(name="timegenerated" dateFormat="date-mysql")
     constant(value="', ")
     property(name="iut")
     constant(value=", '")
     property(name="syslogtag")
     constant(value="')")
}

RSYSLOG_StdPgSQLFmt - Generates a insert command with the message properties, into table SystemEvents for a pgsql database.

1
2
3
4
5
template(name="RSYSLOG_StdPgSQLFmt" type="string"
string="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime,
     ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%',
     %syslogpriority%, '%timereported:::date-pgsql%', '%timegenerated:::date-pgsql%', %iut%,
     '%syslogtag%')")

RSYSLOG_spoofadr - Generates a message containing nothing more than the ip address of the sender.

1
template(name="RSYSLOG_spoofadr" type="string" string="%fromhost-ip%")

RSYSLOG_StdJSONFmt - Generates a JSON structure containing the message properties.

1
2
3
4
5
template(name="RSYSLOG_StdJSONFmt" type="string"
     string="{\"message\":\"%msg:::json%\",\"fromhost\":\"%HOSTNAME:::json%\",\"facility\":
             \"%syslogfacility-text%\",\"priority\":\"%syslogpriority-text%\",\"timereported\":
             \"%timereported:::date-rfc3339%\",\"timegenerated\":
             \"%timegenerated:::date-rfc3339%\"}")

0%