Research on the principle of Tencent persistence framework mmkv

Preface ：

MMKV It's Tencent. 18 A persistence framework launched at the end of the year , There's Android ,IOS,PC Version, etc. , The persistence function of wechat uses MMKV, Project address ：https://github.com/Tencent/MMKV

The biggest feature is high efficiency , Claims to be more efficient than traditional persistence tools 100 times , The goal is to replace the original SharedPreferences（ follow-up SharedPreferences Collectively referred to as SP）. This article mainly explores MMKV And why it is better than SP Efficient .

This paper is mainly based on the Android project for analysis and experiment .

One .MMKV Actually measured

1.1 Import MMKV And simple practical ways

Since it says MMKV Efficient , Let's actually do an example to verify .

MMKV The use of is very simple , First build.gradle Species Import MMKV My bag , Then initialize it in the code .

 implementation 'com.tencent:mmkv:1.2.13'

val initialize = MMKV.initialize(context)

Usage and SP Almost the same as , as follows ：

val kv = MMKV.defaultMMKV()
// write in key=i1,value=1 Value 
kv.encode("i1", 1)
// Reading area key=i1 Value , The return is 1
val decodeInt = kv.decodeInt("i1")

1.2 and SP comparing

In order to make the data more obvious , So we store strings separately , Numbers ,Boolean,1000 Time , Then look at the time spent comparing .

The code is as follows ：

override fun clickItem(position: Int) {
        val random = Random(1000)
        if (position == 0 || position == 2 || position == 4) {
            val sp = when (position) {
                0 -> {
                    requireContext().getSharedPreferences("sp_int", MODE_PRIVATE);
                }
                2 -> {
                    requireContext().getSharedPreferences("sp_boolean", MODE_PRIVATE);
                }
                else -> {
                    requireContext().getSharedPreferences("sp_string", MODE_PRIVATE);
                }
            }
            val edit = sp.edit()
            val currentTimeMillis = System.currentTimeMillis()
            for (i in 0 until 1000) {
                when (position) {
                    0 -> {
                        edit.putInt("key$i", random.nextInt())
                    }
                    1 -> {
                        edit.putBoolean("key$i", true)
                    }
                    else -> {
                        edit.putString("key$i", "key$i")
                    }
                }
                edit.commit()
            }
            Log.i(TAG, "SP spendTime:${System.currentTimeMillis() - currentTimeMillis}")
            return
        }
        if (position == 1 || position == 3 || position == 5) {
            val kv = when (position) {
                1 -> {
                    MMKV.defaultMMKV(0, "sp_int")
                }
                3 -> {
                    MMKV.defaultMMKV(0, "sp_boolean")
                }
                else -> {
                    MMKV.defaultMMKV(0, "sp_string")
                }
            }
            val currentTimeMillis = System.currentTimeMillis()
            for (i in 0 until 1000) {
                if (position == 1) {
                    kv.putInt("key$i", random.nextInt())
                } else if (position == 3) {
                    kv.putBoolean("key$i", true)
                } else {
                    kv.putString("key$i", "key$i")
                }

            }
            Log.i(TAG, "MMKV spendTime:${System.currentTimeMillis() - currentTimeMillis}")
        }
    }

Verify it ,1000 operations , The final result is as follows ：

// Write random for the first time Int
2022-06-29 16:50:54.211 30092-30092/com.xt.client I/MMKVFragment: SP spendTime:14289
2022-06-29 16:50:56.399 30092-30092/com.xt.client I/MMKVFragment: MMKV spendTime:24

// Write random for the second time Int
2022-06-29 16:50:54.211 30092-30092/com.xt.client I/MMKVFragment: SP spendTime:14189
2022-06-29 16:50:56.399 30092-30092/com.xt.client I/MMKVFragment: MMKV spendTime:25

// First write Boolean
2022-06-29 16:51:10.612 30092-30092/com.xt.client I/MMKVFragment: SP spendTime:12485
2022-06-29 16:51:12.810 30092-30092/com.xt.client I/MMKVFragment: MMKV spendTime:30

// Second write Boolean
2022-06-29 16:51:14.567 30092-30092/com.xt.client I/MMKVFragment: SP spendTime:36
2022-06-29 16:51:16.192 30092-30092/com.xt.client I/MMKVFragment: MMKV spendTime:9

// First write String
2022-06-29 16:51:33.950 30092-30092/com.xt.client I/MMKVFragment: SP spendTime:12718
2022-06-29 16:51:38.381 30092-30092/com.xt.client I/MMKVFragment: MMKV spendTime:12

Pass the result , We can find these two phenomena ：

1. On first write ,MMKV Is extremely efficient , stay 20 Many milliseconds , and SP You need to 14000 millisecond .

2. On the second write , If the data does not change , be SP Is also quite efficient . stay 100 Within milliseconds , No matter what Int,Boolean still String. and MMKV Be as efficient as ever , Still 20 Many milliseconds .（ The reasons will be analyzed in Chapter 2 ）

To sum up , If the data changes ,MMKV The efficiency of is much better than SP Of （ Even reached the level of hundreds of times ）, If the data does not change , because SP There is a caching mechanism , So the impact is not big .

Two .SharedPreferences What are the problems

All say MMKV It is used to replace Android native SharedPreferences Of , So we naturally want to explore , Native SP What are the drawbacks ？

2.1 SP Realization principle - Write

First of all, a brief understanding of SP Principle .SP The implementation class is SharedPreferencesImpl,Editor The implementation class is SharedPreferencesImpl.EditorImpl.

We putString when , Finally call to EditorImpl.putString(), The logic is simple , Is to put key,value Store in Map in .

 @Override
        public Editor putString(String key, @Nullable String value) {
            synchronized (mEditorLock) {
                mModified.put(key, value);
                return this;
            }
        }

When we look at the final submission for editing ,commit Method （apply similar ）, Core are commitToMemory Method , It's just commit One more. CountDownLatch Lock of .

 private MemoryCommitResult commitToMemory() {
            long memoryStateGeneration;
            boolean keysCleared = false;
            List<String> keysModified = null;
            Set<OnSharedPreferenceChangeListener> listeners = null;
            Map<String, Object> mapToWriteToDisk;
            //1. Lock operation , Avoid multithreading 
            synchronized (SharedPreferencesImpl.this.mLock) {
                // We optimistically don't make a deep copy until
                // a memory commit comes in when we're already
                // writing to disk.
                if (mDiskWritesInFlight > 0) {
                    // We can't modify our mMap as a currently
                    // in-flight write owns it.  Clone it before
                    // modifying it.
                    // noinspection unchecked
                    //2. Copy a new Map, Store all the original Map data .
                    mMap = new HashMap<String, Object>(mMap);
                }
                mapToWriteToDisk = mMap;
                mDiskWritesInFlight++;

                boolean hasListeners = mListeners.size() > 0;
                if (hasListeners) {
                    keysModified = new ArrayList<String>();
                    listeners = new HashSet<OnSharedPreferenceChangeListener>(mListeners.keySet());
                }

                synchronized (mEditorLock) {
                    boolean changesMade = false;

                    if (mClear) {
                        if (!mapToWriteToDisk.isEmpty()) {
                            changesMade = true;
                            mapToWriteToDisk.clear();
                        }
                        keysCleared = true;
                        mClear = false;
                    }
                    //3. Traverse the modified content Map, Compared with the old one Map, To synthesize . If the value does not change, skip , If it changes, it will be stored in the full amount Map in .
                    for (Map.Entry<String, Object> e : mModified.entrySet()) {
                        String k = e.getKey();
                        Object v = e.getValue();
                        // "this" is the magic value for a removal mutation. In addition,
                        // setting a value to "null" for a given key is specified to be
                        // equivalent to calling remove on that key.
                        if (v == this || v == null) {
                            if (!mapToWriteToDisk.containsKey(k)) {
                                continue;
                            }
                            mapToWriteToDisk.remove(k);
                        } else {
                            if (mapToWriteToDisk.containsKey(k)) {
                                Object existingValue = mapToWriteToDisk.get(k);
                                if (existingValue != null && existingValue.equals(v)) {
                                    continue;
                                }
                            }
                            mapToWriteToDisk.put(k, v);
                        }

                        changesMade = true;
                        if (hasListeners) {
                            keysModified.add(k);
                        }
                    }

                    mModified.clear();
                    //4. If changesMade=false, It means that the data has not changed .
                    if (changesMade) {
                        mCurrentMemoryStateGeneration++;
                    }

                    memoryStateGeneration = mCurrentMemoryStateGeneration;
                }
            }
//5. The resulting MemoryCommitResult Object returns , What you end up writing is MemoryCommitResult object . It's right map The wrapper class 
            return new MemoryCommitResult(memoryStateGeneration, keysCleared, keysModified,
                    listeners, mapToWriteToDisk);
        }

It mainly includes the following steps ：

1. Lock operation , Avoid multithreading

2. Copy a new Map, Store all the original Map data .

3. Traverse the modified content mModified, Compared with the old one mapToWriteToDisk, To synthesize . If the value does not change, skip , If it changes, it will be stored in the full amount Map in .

4. If the value is modified , Record changesMade=true. here mCurrentMemoryStateGeneration+1;

5. The resulting MemoryCommitResult Object returns , What you end up writing is MemoryCommitResult object . It's right map The wrapper class

The final method of writing is writeToFile, The code is not posted , Simply speaking , That is to say, if there is no modification and the original file exists , No write operation is required for direct callback （ This also corresponds to the experimental results in Chapter 1 2, Why is the efficiency not low without modification ）. otherwise , Write to XML The file of .

The final file is saved in data/data/ Package name /shared_prefs/ Under the folder ：

The content format is as follows ：

<?xml version='1.0' encoding='utf-8' standalone='yes' ?>
<map>
    <int name="key1251" value="-832759317" />
    <int name="key1252" value="1723359929" />
    <int name="key1253" value="469068865" />
    <int name="key1254" value="-836324061" />
    <int name="key1250" value="129252392" />
</map>

2.2 SP Realization principle - read

initialization SharedPreferencesImpl When , I will read it in the specified file , adopt startLoadFromDisk Method .

 SharedPreferencesImpl(File file, int mode) {
        mFile = file;
        mBackupFile = makeBackupFile(file);
        mMode = mode;
        mLoaded = false;
        mMap = null;
        mThrowable = null;
        startLoadFromDisk();
    }

Start a new thread to read the contents of the file , Then put the read content into Map On .

private void startLoadFromDisk() {
        synchronized (mLock) {
            mLoaded = false;
        }
        new Thread("SharedPreferencesImpl-load") {
            public void run() {
                loadFromDisk();
            }
        }.start();
    }

Then we are looking at getString Method , The rest are similar .

 @Override
    @Nullable
    public String getString(String key, @Nullable String defValue) {
        synchronized (mLock) {
            awaitLoadedLocked();
            String v = (String)mMap.get(key);
            return v != null ? v : defValue;
        }
    }

Wait for the contents of the above file , The execution will not continue until the reading is completed , Otherwise it will be blocked .

2.3 SP The problem is

Through the understanding of the principle , We will find that there are many problems in doing so . All in all , The main problems are as follows ：

1. Final write XML The utility of the file is IO operation ,IO The operation requires two copies , Efficiency is relatively low .（ The reason lies in Baidu , I won't go into that here ）

2. practical XML Format for storage , And all of them are saved in the form of strings , Waste storage space . such as value="469068865". Take up 17 Bytes ,utf-8 One English character occupies 1 Bytes , Store the value 17 Bytes .

3. Every time you edit , You need to write to the file in full . Because every time it is the complete data Map Write operation , Even if only one value is modified . This is undoubtedly a great waste .

4.SP Although it supports multi process access , But multi process reading is quite unsafe , Because memory cannot be shared between processes , and SP The multiprocess is that each process operates on an object . So our safe use is still to use a process to read , And provide ContentProvider For other processes to access or increase file locks , This undoubtedly increases the complexity of our use .

5. Thread blocking problem . Above we see , Only when all loads are completed xml After the content in ,getString Can continue to execute . So the thread will be blocked .

3、 ... and .MMKV How to solve these problems

since SP There are so many problems , So Tencent will give up SP, newly build MMKV Projects to solve these problems , that MMKV How to solve these problems ？

3.1 Realize efficient file operation

IO The operation requires two memory copies , Copy from user memory space to kernel space for the first time , Copy from kernel space to disk for the second time . Especially the first copy , It's a waste CPU The performance of the .

be familiar with binder We all know the principle ,binder Achieved a copy , The underlying principle is mmap. therefore MMKV Also used. mmap Principle . Map part of the user memory and part of the kernel memory to the same physical memory , In this way, the user can operate on this part of memory , It will be directly reflected in the kernel space , Then the memory completes the final write operation , One less copy , Then the efficiency will be greatly increased . And because the copy of the kernel occurs in the system process , Does not block the operation of the user process . So actually mmap Write execution efficiency , Close to the efficiency of direct memory operation .

3.2 Achieve a more streamlined data format

As mentioned above ,SP If stored 469068865 This int value , Take up 17 Bytes .

But actually , If expressed in binary , It only needs 4 In bytes .

alike , If storage 255 This int If it's worth it ,SP need 10 individual （value+""+255=）, In fact, binary representation only needs 1 Bytes .

So we must wonder if there is a more efficient representation ？ This plan may have many , At present, the most recommended is google Of protobuf This serialization scheme .

The details of the protobuf The scheme will not be introduced in detail here , Just make a simple explanation of the principle .

Storage 255, Use protobuf Will 01 FF In the way of ,01 For occupation 1 Bytes ,FF Represents the actual value . therefore , Only two bytes are needed to represent 255 This value.

At this time, you must ask , Why? 255 It must be value Well ？ How to guarantee 255 What you read is not key The value of ？ The answer is simple .Map It must be key-value-key-value In the form of . So we just read the values in turn , The first representative key, The second representative value, The third representative key, The fourth represents value, Read in this order and no error will occur .

and MMKV This is the serialization principle .

3.3 Achieve better data update methods

It says SP When , It is updated every time , Need to put the whole map Write the data overwritten in to the file . Even if I only modified one of them . At this time we must think , Is there any way to update only the one I modified each time ？

The answer, of course, is yes , We can think about it map Principle . When we operate a map When , Add a lot key-value data . If you want to change one of them , Then you only need to enter the specified key-value You can override the previous value .

alike , When we persist , A similar action can be taken . We fill in the last part of the text every time we change the content . The original storage structure is ：

key1:value1,key2:value2,key3:value3,key4:value4

modify key2 The value of is value2222, The modified storage structure is as follows , And only the red part has been modified , So there are fewer changes .

key1:value1,key2:value2,key3:value3,key4:value4,key2:value2222

When reading data , Read first key="key2",value="value2", Then read to the second key2 when , modify value="value2222".

Of course , There is also a problem in doing so ,N After many modifications , Because it is the mode of each addition , So the storage content will become infinitely long . therefore MMKV I made a judgment on this piece , When the memory limit is reached , Will start a full update , Sift out duplicate data , So as to ensure that the data is restored to the most compact form .

3.4 How to solve multi process consistency

MMKV Solve multi process problems , It is solved by checking code .

Before reading the file , I'll read it first CRC Check code , If the check code is the same as expected , Then read .

Otherwise, update the check code and read the entire file again .

Speaking of this , Expand a little , Why do I use CRC check ？ as a result of CRC be relative to MD5 Faster , But the security will be lower .

3.5  How to block threads

MMVK in , In fact, it is a similar process , Also go through first getDataForKey Method to read all the values and save them to Map in （native in Map）, And then through key Go to map The value of .

difference SP Is an additional thread to read , and MMKV Is read directly from the current thread .

MMKV The implementation process in is as follows ：

MMKV The whole process runs in one thread , So there will be no loss of thread switching , So it will be more efficient .

The reason is actually MMKV adopt mmap Reading a value is an operation close to the memory level , So there won't be too much time , So there is no need to switch threads .