loggie 编码以及换行符测试

一、测试非编码单行换行符

1、测试场景

\r\n测试

pipeline配置文件：

pipelines:
  - name: local
    sources:
      - type: file
        name: demo
        lineEnd:
          type: carriage_return_line_feed
          value: "\r\n"
          charset: utf-8
        paths:
          - /home/zhanglei/*.log
        fields:
          topic: "loggie"
    interceptors:
      - type: iconv
        charset: "utf-8"
    sink:
      type: dev
      printEvents: true
      codec:
        pretty: true

2、测试脚本

<?php
$a = "1ddsjaodjasiodjasoidjasoidjasid段晒苏杭怠速后端三十u的\r\n";
file_put_contents("a.log", $a, FILE_APPEND);
var_dump("1ddsjaodjasiodjasoidjasoidjasid段晒苏杭怠速后端三十u的");

3、测试结果

输出正常

二、测试编码单行换行符

1、测试场景

gbk 编码

\r\n测试

pipeline配置文件：

pipelines:
  - name: local
    sources:
      - type: file
        name: demo
        lineEnd:
          type: carriage_return_line_feed
          value: "\r\n"
          charset: gbk
        paths:
          - /home/zhanglei/*.log
        fields:
          topic: "loggie"
    interceptors:
      - type: iconv
        charset: "gbk"
    sink:
      type: dev
      printEvents: true
      codec:
        pretty: true

2.测试脚本

<?php
$a = iconv("utf-8", "gbk", "哈哈哈顿巴斯涉及到八级大把货都不花钱我闻不到几千万望
变电气文件变动祭敖包\r\n");

file_put_contents("a.log", $a, FILE_APPEND);
var_dump($a);

3.测试结果

三、测试非编码 auto 换行符

1、测试场景

\n测试

pipeline配置文件：

pipelines:
  - name: local
    sources:
      - type: file
        name: demo
        lineEnd:
          type: auto
        paths:
          - /home/zhanglei/*.log
        fields:
          topic: "loggie"
    interceptors:
      - type: iconv
    sink:
      type: dev
      printEvents: true
      codec:
        pretty: true

2.测试脚本

<?php
$a = "1ddsjaodjasiodjasoidjasoidjasid段晒苏杭怠速后端三十u的\r\n";
file_put_contents("b.log", $a, FILE_APPEND);
var_dump($a);

3.测试结果

 四、测试编码 auto 换行符

1、测试场景

gbk 编码

\n测试

pipeline配置文件：

pipelines:
  - name: local
    sources:
      - type: file
        name: demo
        lineEnd:
          type: auto
          value: "\r\n"
          charset: gbk
        paths:
          - /home/zhanglei/*.log
        fields:
          topic: "loggie"
    interceptors:
      - type: iconv
        charset: "gbk"
    sink:
      type: dev
      printEvents: true
      codec:
        pretty: true

2.测试脚本

<?php
$a = iconv("utf-8", "gbk", "哈哈哈顿巴斯涉及到八级大把货都不花钱我闻不到几千万望
变电气文件变动祭敖包\r\n");

file_put_contents("a.log", $a, FILE_APPEND);
var_dump($a);

3.测试结果

五、测试非编码自定义换行符(特殊字符)

1、测试场景

pipeline:

pipelines:
  - name: local
    sources:
      - type: file
        name: demo
        lineEnd:
          type: custome
          value: "\r\n"
        paths:
          - /home/zhanglei/*.log
        fields:
          topic: "loggie"
    interceptors:
      - type: iconv
    sink:
      type: dev
      printEvents: true
      codec:
        pretty: true

2、测试结果

3、结论

custom 配置不能解决 特殊ascii码问题 

特殊字符使用

	lineTerminators = map[string]LineTerminator{
		"auto":                      AutoLineTerminator,
		"line_feed":                 LineFeed,
		"vertical_tab":              VerticalTab,
		"form_feed":                 FormFeed,
		"carriage_return":           CarriageReturn,
		"carriage_return_line_feed": CarriageReturnLineFeed,
		"next_line":                 NextLine,
		"line_separator":            LineSeparator,
		"paragraph_separator":       ParagraphSeparator,
		"null_terminator":           NullTerminator,
	}

六、测试非编码自定义换行符(非特殊字符)

1、测试场景

pipeline:

pipelines:
  - name: local
    sources:
      - type: file
        name: demo
        lineEnd:
          type: custom
          value: "的"
        paths:
          - /home/zhanglei/*.log
        fields:
          topic: "loggie"
    interceptors:
      - type: iconv
    sink:
      type: dev
      printEvents: true
      codec:
        pretty: true

2、测试结果

3、结论

正常

 七、测试编码自定义换行符(特殊字符)

1、测试场景

pipeline:

pipelines:
  - name: local
    sources:
      - type: file
        name: demo
        lineEnd:
          type: custom
          value: "\r\n"
          charset: gbk
        paths:
          - /home/zhanglei/*.log
        fields:
          topic: "loggie"
    interceptors:
      - type: iconv
        charset: gbk
    sink:
      type: dev
      printEvents: true
      codec:
        pretty: true

2、测试结果

3、结论

custom 配置不能解决 特殊ascii码问题 

但是可见字符都是正常的，如果出现不可见字符请使用

	lineTerminators = map[string]LineTerminator{
		"auto":                      AutoLineTerminator,
		"line_feed":                 LineFeed,
		"vertical_tab":              VerticalTab,
		"form_feed":                 FormFeed,
		"carriage_return":           CarriageReturn,
		"carriage_return_line_feed": CarriageReturnLineFeed,
		"next_line":                 NextLine,
		"line_separator":            LineSeparator,
		"paragraph_separator":       ParagraphSeparator,
		"null_terminator":           NullTerminator,
	}

解码后出现bug

 八、测试编码自定义换行符(非特殊字符)

1、测试场景

pipeline:

pipelines:
  - name: local
    sources:
      - type: file
        name: demo
        lineEnd:
          type: custom
          value: "\r\n"
          charset: gbk
        paths:
          - /home/zhanglei/*.log
        fields:
          topic: "loggie"
    interceptors:
      - type: iconv
        charset: gbk
    sink:
      type: dev
      printEvents: true
      codec:
        pretty: true

2、测试结果

3、结论

非特殊字符解码正常