rsyslog学习6 -- The Property Replacer属性替换

The Property Replacer 属性替换

The Property Replacer 是 rsyslogd的 字符模板的核心组件，用来操纵属性值

Accessing Properties

通过两个 % 读取属性，并被修改，语法如下

%property:fromChar:toChar:options%

Available Properties

见 rsyslog properties

Character Positions

FromChar 和toChar 用来截字串，它们代表偏移量，从1开始计数。截取前两个字符语法为: “%msg:1:2%”. 如果只使用options，冒号还是需要的。比如 “%msg:::lowercase%”. 如果要截取从某位置到末尾，设置toChar为 (“$”) (e.g. %msg:10:$%, which will extract from position 10 to the end of the string).

支持正则表达式。设置FromChar为“R”，rsyslog就会知道使用正则面不是位置参数，表达式放在toChar上，必须以“–end” 结尾。rsyslog会返回匹配的部分。比如 “%msg:R:.*Sev:. (.*) [.*–end%”

在 “R” 后还可以支持一些参数

R,<regexp-type>,<submatch>,<nomatch>,<match-number>

regexp-type 可以使用 “BRE” for Posix basic regular expressions or “ERE” for extended ones. 必须要大写. 早期的版本只支持BRE。submatch指示器指示使用哪一个submatch结果，支持单数字。0表示整个匹配，1 to 9 为实际submatch. match-number 表示使用哪一个结果，从0开始。最多支持10个（数字是9）

nomatch 指示如果没有匹配将使用的值

这个例子使用ERE规则，从消息提取第一个子匹配，如果没有匹配，则使用整个字段

%msg:R,ERE,1,FIELD:for (vlan[0-9]\*):--end%

以下是从第二个匹配中，提取第一个子匹配

%msg:R,ERE,1,FIELD,1:for (vlan[0-9]\*):--end%

强烈建议使用 rsyslog regular expression checker/generator 工具。虽然不同版本有所不同，但基本覆盖了大部分情况。

关于nomatch mode

如果没有匹配，以早期会返回“**NO MATCH**“，此模式为 DFLT

现在增加了几个有用的模式。

Mode	Returned
DFLT	“NO MATCH”
BLANK	“” (empty string)
ZERO	“0”
FIELD	full content of original field

提取也可以基于fields，将FromChar设为F。Fields以定位符分割，默认是TAB（ASCII 9），也可以自定义，例如需要用”,”作为定位符，则写成 “F,44”，44是”,”的ASCII值。用定位符比正则高效，如果消息是规则的话。Field是从1开始，如果设为0，或高于实际数量，会导致”field not found”错误。toChar上设置需要的field数，比如”%msg:F:3%”，或者用”;”分割: %msg:F,59:3%”

使用fields的不足是无法截取字串，从6.3.9开始，fromPos和toPos可以用来解决这个问题，但语法上有点丑。比如在上例中要截取第5到第9个字符，语法是“%msg:F,59,5:3,9%”

F和R必须大写，语法不能有空格。

每当有字段定位符出现，就会开启一个新的字段。可以在定位符后加上”+”，则多个定位符将视过一个。比如以下代码

int n, m;
...
syslog(LOG_ERR, "%d test %6d", n, m);

有可能输出“1 test 2”, “1 test 23”, “1 test 234567”，空格数量不可知。可以使用以下语法

"%msg:F,32:2%" to "%msg:F,32+:2%".

Property Options

大小写敏感

• uppercase

  convert property to uppercase only

• lowercase

  convert property text to lowercase only

• fixed-width

  当原始字串长度小于toChar时填充。This feature was introduced in rsyslog 8.13.0

• json

  将value转会json可以被解析的格式，比如将ASCII LF转为“\n”，不能和csv同存

• jsonf[:outname]

  该属性将会被转为json，与json不同的是，json是去转化value，让其可以被json，而jsonf是将整个结果是源于 “fieldname”=”value” 这种格式。其中fieldname是参数指定，否则使用默认属性名。value可以被option操控，但field name不能，所以需要指定恰当的名字。参考 this article from Rainer’s blog 提到

  # json的模板，比较丑陋，为了生成json，相当于字段的拼接
  
  $template tpl, “{“message”:”%msg:::json%”,”fromhost”:”%HOSTNAME:::json%”,”facility”:”%syslogfacility-text%”,”priority”:”%syslogpriority-text%”,”timereported”:”%timereported:::date-rfc3339%”,”timegenerated”:”%timegenerated:::date-rfc3339%”}”

  # jsonf的模板，直接是从 “fieldname”=”value” 生成json
  
  $template tpl,”{%msg:::jsonf:message%,%HOSTNAME:::jsonf:fromhost%,%syslogfacility-text:::jsonf:facility%,%syslogpriority-text:::jsonf:priority%,%timereported:::date-rfc3339,jsonf%,%timegenerated:::date-rfc3339,jsonf%}”

  # 如果不需要改变fieldname
  
  $template tpl,”{%msg:::json%,%HOSTNAME:::jsonf%,%syslogfacility-text:::jsonf%,%syslogpriority-text:::jsonf%,%timereported:::date-rfc3339,jsonf%,%timegenerated:::date-rfc3339,jsonf%}”

• csv

  根据 RFC 4180 生成csv格式。Rsyslog 总是使用双引号。 例如: $template csvline,”%syslogtag:::csv%,%msg:::csv%” ，你需要在模板中正常定义”,”的位置

• drop-last-lf

  The last LF in the message (if any), is dropped. Especially useful for PIX.

• date-utc

  convert data to UTC prior to outputting it (available since 8.18.0)

• date-mysql

  format as mysql date

• date-rfc3164

  format as RFC 3164 date，”Mmm dd hh:mm:ss”

• date-rfc3164-buggyday

  与 date-rfc3164 相同, 在RFC 3164 如果日期为单数字，需要写入一个空格，在buggyday中，用0来代替。如果要转发消息，不建议使用这个option，有可能会被远端服务器认识是错误

• date-rfc3339

  format as RFC 3339 date，”2013-09-12T22:50:20+08:00”

• date-unixtimestamp

  Format as a unix timestamp (seconds since epoch)

• date-year

  just the year part (4-digit) of a timestamp

• date-month

  just the month part (2-digit) of a timestamp

• date-day

  just the day part (2-digit) of a timestamp

• date-hour

  just the hour part (2-digit, 24-hour clock) of a timestamp

• date-minute

  just the minute part (2-digit) of a timestamp

• date-second

  just the second part (2-digit) of a timestamp

• date-subseconds

  just the subseconds of a timestamp (always 0 for a low precision timestamp)

• date-tzoffshour

  just the timezone offset hour part (2-digit) of a timestamp

• date-tzoffsmin

  just the timezone offset minute part (2-digit) of a timestamp. Note that this is usually 0, but there are some time zones that have offsets which are not hourly-granular. If so, this is the minute offset.

• date-tzoffsdirection

  just the timezone offset direction part of a timestamp. This specifies if the offsets needs to be added (“+”) or subtracted (“-“) to the timestamp in order to get UTC.

• date-ordinal

  returns the ordinal for the given day, e.g. it is 2 for January, 2nd

• date-week

  returns the week number

• date-wday

  just the weekday number of the timstamp. This is a single digit, with 0=Sunday, 1=Monday, …, 6=Saturday.

• date-wdayname

  just the abbreviated english name of the weekday (e.g. “Mon”, “Sat”) of the timestamp.

• escape-cc

  replace control characters (ASCII value 127 and values less then 32) with an escape sequence. The sequence is “#” where charval is the 3-digit decimal value of the control character. For example, a tabulator would be replaced by “#009”. Note: using this option requires that $EscapeControlCharactersOnReceive is set to off.

• space-cc

  replace control characters by spaces Note: using this option requires that $EscapeControlCharactersOnReceive is set to off.

• drop-cc

  drop control characters - the resulting string will neither contain control characters, escape sequences nor any other replacement character like space. Note: using this option requires that $EscapeControlCharactersOnReceive is set to off.

• compressspace

  compresses multiple spaces (US-ASCII SP character) inside the string to a single one. This compression happens at a very late stage in processing. Most importantly, it happens after substring extraction, so the FromChar and ToChar positions are NOT affected by this option. (available since v8.18.0)

• sp-if-no-1st-sp

  这个option有点吓人，一般不应该被使用。当字串第一个字符为非空格时，返回空格，否则返回空值。这主要是用来解决RFC3164的问题。在3164中，没有对tag和message指定分割符，而通常是以空格代替，所以当空格是消息的一部分时，会带来问题。tag后立即是另一个非空格字符，会给解释器带来误解。

  看不懂这个解释也很正常，反正忘掉它就好 ;)

• secpath-drop

  Drops slashes inside the field (e.g. “a/b” becomes “ab”). Useful for secure pathname generation (with dynafiles).

• secpath-replace

  Replace slashes inside the field by an underscore. (e.g. “a/b” becomes “a_b”). Useful for secure pathname generation (with dynafiles).

要使用多个options，只要简单的用”,”连接它们，比如“escape-cc,sp-if-no-1st-sp”，如果选项有冲突，后者会覆盖前者，比如 “escape-cc,drop-cc” will use drop-cc and “drop-cc,escape-cc” will use escape-cc mode.

Further Links

• Article on “Recording the Priority of Syslog Messages” (describes use of templates to record severity and facility of a message)

• Configuration file syntax, this is where you actually use the property replacer.

• Property Replacer nomatch mode

  • Summary of nomatch Modes