Go json. Decoder Considered Harmful

If you're using Go Language programming , And use json.Decoder Inverse solution JSON load , You may have unexpected effects . You should use json.Unmarshal Instead of json.Decoder.

1. json.Decoder It is designed to solve JSON flow , Not complete JSON object .
2. json.Decoder Will ignore some illegal JSON grammar .
3. json.Decoder The network connection is not released for reuse （ It will slow down HTTP Request to about 4 Multiple duration ）.

If you get through json Package documentation , You won't be surprised , Such is the case . I have made mistakes many times . Most developers find using json.Decoder.Decode(...) Than using json.Unmarshal(...) It's easier to parse io.Reader type .

1. json.Decoder by JSON Flow design

JSON Streams are generally in series （concatenated） Or split with a new line JSON value . Here is an example ：

{"Name":"Ed"}{"Name":"Sam"}{"Name":"Bob"}

Copy

Complete streaming content is not a legal JSON, Only the outermost layer is used [ ] It is legal to surround JSON type .

It's just in series JSON object , let me put it another way , It's legal JSON flow .

json.Decoder The type is specially designed for JSON flow . The most likely thing , Yours JSON The load is not suitable for this .

that JSON Why does flow exist ？ Can't we use JSON Array ？JSON Stream is mainly used in ：

• Store structured data in files , And quickly append without completely parsing the entire file
• from API And other real-time structured streaming data （ Such as docker logs/docker events API Wait is to use this method ）

If you are parsing a single complete JSON object , Do not use json.Decoder.

2. json.Decoder Will ignore illegal syntax

Not to ignore all illegal grammar , But mixed illegal and legal grammar JSON Flow will be json.Decoder Ignore mistakes . For example, suppose a API return ：

{"Name": "Bob"}

Copy

But the service introduces bug, All of a sudden, it's coming back

{}{"Name": "Bob"}

Copy

This is obviously illegal JSON load , But it is a legal JSON flow ,json.Decoder Acceptable .

But you don't know this situation , Your code will reverse this return to complete JSON object ：

type Person struct {Name string}

...
var v Person
if err := dec.Decode(&v); err != nil {
    panic(err)
}
fmt.Println(v.Name)

Copy

You will get v.Name Is an empty string , There is no mistake .json.Decoder It reverses the first JSON object , The rest is ignored .

Can this happen ？ Maybe not , But you can 100% Are you sure? ？ Because when it happens, you can't easily debug come out .

3. json.Decoder Not properly exhausted HTTP Connect

This problem was recently solved by Flippo Valsorda Bring up （link）, You may be affected by this , Unless you're using Go 1.7 (or above, translator's note ).

If you are creating a HTTP request , Transfer the return body to json.Decoder#Decode() （ Most people will do this ） Then it is very likely that your connection has not been properly exhausted , May make your HTTP The client is slow 4 times .

If HTTP The endpoint returns a single complete JSON object , And you only call json.Decoder#Decode() once , This may mean that you have not read io.EOF Return signal . So you have no basis io.EOF End json.Decoder, The returnee is still open ,TCP Connect （ Or others in use Transport） It cannot be returned to the connection pool even if you have finished reading . For more information, please click URL.

Now in golang The main branch master Has been fixed in , Most likely in Go1.7 In the release .（ This article is written in 2016 year 4 month 28 Japan , That fashion has not been released Go1.7）

meanwhile , If your returnee is small enough , Only need to use ioutil.ReadAll Read all into memory and use json.Unmarshal Inverse solution . If you want to keep using json.Decoder, You need to exhaust the unread part of the return body , for example :

io.Copy(ioutil.Discard, resp.Body)

Copy

therefore , If you are using json.Decoder, Please check your code , Will all defer resp.Body.Close() Replace with ：

defer func() {
    io.Copy(ioutil.Discard, resp.Body)
    resp.Body.Close()
}

Copy

Conclusion

If you don't handle JSON flow , Please do not use json.Decoder.

Use json.Unmarshal:

• If you don't know Go JSON What is flow
• If you are dealing with a single JSON object
• If remote API It is possible to return the problem JSON

Now you know the trade-off strategy , Then decide for yourself .

Reference material

https://ahmet.im/blog/golang-json-decoder-pitfalls/