Player actual combat 22 to solve the problems of flower screen and Caton

1. Memory alignment

1.1 What is memory alignment
stay C In language , Structure is a composite data type , Its constituent elements can be basic data types （ Such as int、long、float etc. ） The variable of , It can also be some composite data types （ Such as arrays 、 structure 、 Union, etc ） Data unit of . In structure , The compiler follows its natural boundaries for each member of the structure （alignment） Allocate space . Members are stored in memory in the order in which they are declared , The address of the first member is the same as the address of the whole structure .

In order to make CPU The ability to access variables quickly , The starting address of a variable should have some properties , That is to say ” alignment ”. such as 4 Bytes of int type , Its starting address should be at 4 On the boundary of bytes , That is, the starting address can be 4 to be divisible by .

for instance , Theoretically ,32 A system. ,int Occupy 4byte,char Take one byte, So putting them in a structure should take up 4+1=5byte; But actually , The result of running the program is 8 byte, This is what memory alignment causes .

#include<stdio.h>
struct{
    
    int x;
    char y;
}s;

int main()
{
    
    printf("%d\n",sizeof(s);  //  Output 8
    return 0;
}

1.2 Why memory alignment
Although memory is in bytes , But most processors do not access memory by byte block . It usually takes double bytes , Four bytes ,8 byte ,16 Bytes even 32 Access memory in bytes , We call these access units memory access granularity .

Now consider 4 A processor with byte access granularity int Type variable （32 Bit system ）, The processor can only be accessed from the address 4 Multiple of memory starts reading data .

If there is no memory alignment mechanism , Data can be stored at will , Now one. int Variables are stored in the slave address 1 The initial contact is in the four byte address , When the processor fetches data , First of all 0 The address starts reading the first 4 Byte block , Remove the unwanted bytes （0 Address ）, And then from the address 4 Start reading next 4 Byte block , Also remove the data that you don't want （5,6,7 Address ）, The last two pieces of data are merged into registers . It takes a lot of work .

Now we have memory aligned ,int Type data can only be stored in memory according to the alignment rules , for instance 0 Address start memory （ An integer multiple of the access granularity ）. Now the processor can read the data at one time when fetching data , And there's no need to do anything extra , Improved efficiency , Is an example of saving time through space .

If the access granularity is reduced , It seems that there is no need to do memory alignment , But the reading speed is greatly slowed down, instead of trying , Therefore, the memory alignment technology makes the access granularity as large as possible to speed up the reading speed ; Secondly, for embedded devices , You can make full use of its valuable memory resources by using memory alignment .

Reference resources ：https://zhuanlan.zhihu.com/p/30007037

2.YUV And linesize

introduce stride
stride: Refers to the space occupied by each row of pixels in memory . In order to realize memory alignment, the space occupied by each row of pixels in memory is not necessarily the width of pixels , It may be necessary to add some bytes of data at the end of each line for memory alignment

see ffmpeg Structure AVFrame Three members of ：
01.uint8_t *data[AV_NUM_DATA_POINTERS];
data Store the original audio and video data （ Video for YUV, Audio for PCM）. There are two ways to store audio and video ,plannar The way and packet The way
plannar The way ： passageway n The data is stored in data[n] in ; take YUV In terms of video , Namely data[0],data[1],data[2] Store separately Y,U,V The data of . Take two channel audio , Namely data[0],data[1] Store the left channel separately , Right channel data ; For audio , The number of channels may be greater than AV_NUM_DATA_POINTERS, So much will be stored in extended_data Field
packet The way ： All data is stored in data[0] in （RGB）

02.int linesize[AV_NUM_DATA_POINTERS];
Indicates the size of each row of data ; about plannar（ Memory alignment 1） Format and packet（ After memory alignment The channel number ） Format , The values here are also different

03.uint8_t **extended_data;
Deposit data Data that cannot be stored

Like resolution 638480 Of RGB24（RGB by ） Images , When we process the memory, if we want to 16 Byte alignment , be 6383/16=119.625 Not divisible , So we can't 16 Byte alignment , We need to fill in the end of each line 6 Bytes , Namely 640*3/16=120. At this time, the stride by 1920 byte , As shown in the figure below ：

YUV420

When storing, follow plannar Format store , Pre deposit Y, Save again U, Last deposit V
On display , Bright purple frame (Y0,Y1,Y4,Y5) Purple frame with chromaticity （U0,V0） Combine ,Y:U:V=4:1:1

yuv420 Two formats of ：I420,NV12

When the contour does not change , The color display is misaligned , It could be UV The order is misplaced （ Adopted NV12）

source ：https://www.bilibili.com/video/BV1kL4y1G74m?from=search&seid=2376940568801521762&spm_id_from=333.337.0.0

3. Flower screen and solution

Play a .MP4 The following screen appears in the file ：

Breakpoints allow you to view AVFrame Medium linesize：

Because of the use of yuv420p plannar Format ,linesize[0]=1024 Deposit is Y, Should be and width equal , But you can see width by 1000

Because for byte alignment , Each line adds 24 Invalid data of bytes , When copying through the following methods ：

	memcpy(datas[0], frame->data[0], width * height);
	memcpy(datas[1], frame->data[1], width * height / 4);
	memcpy(datas[2], frame->data[2], width * height / 4);

frame According to linesize a line 1024 Deposit , use 1000 Copy its first line to data There is no problem displaying with the graphics card ,data The second row of data from linesize At the end of the first line 1001 Bytes to start copying , At this point, it will be disordered , This leads to a flower screen

The solutions are as follows ：

	for (int i = 0; i < height; i++)//Y
		memcpy(datas[0] + width * i, frame->data[0] + frame->linesize[0]* i, width);
	for (int i = 0; i < height / 2; i++)//U
		memcpy(datas[1] + width / 2 * i, frame->data[1] + frame->linesize[1] * i, width);
	for (int i = 0; i < height / 2; i++)//V
		memcpy(datas[2] + width / 2 * i, frame->data[2] + frame->linesize[2] * i, width);

With Y For example , because Y Data in a row , Each time the frame Chinese Reading linesize Size , Take only linesize In size width Put it in data in , loop height Time , The flower screen can be solved ：

Although the display is OK , But at run time there will still be bug：

Because the memory space of materials is allocated ：

	datas[0] = new unsigned char[width * height];		//Y
	datas[1] = new unsigned char[width * height / 4];	//U
	datas[2] = new unsigned char[width * height / 4];	//V

data[1] And data[2] The size of the space is wh/4
But from above frame When assigning a value in ,datas[1]/datas[2] Gongfu wh/4+（linsize-width）*height/2, For this media file linsize>width, As a result, a large amount of data is placed in an undefined space , That is to say Memory out of bounds , The correct code should be as follows ：

	for (int i = 0; i < height; i++)//Y
		memcpy(datas[0] + width * i, frame->data[0] + frame->linesize[0]* i, width);
	for (int i = 0; i < height / 2; i++)//U
		memcpy(datas[1] + width / 2 * i, frame->data[1] + frame->linesize[1] * i, width);
	for (int i = 0; i < height / 2; i++)//V
		memcpy(datas[2] + width / 2 * i, frame->data[2] + frame->linesize[2] * i, width);

episode ： I mistook it frame->data[] The subscript ：

	for (int i = 0; i < height; i++)//Y
		memcpy(datas[0] + width * i, frame->data[0] + frame->linesize[0]* i, width);
	for (int i = 0; i < height / 2; i++)//U
		memcpy(datas[1] + width / 2 * i, frame->data[0] + frame->linesize[1] * i, width);
	for (int i = 0; i < height / 2; i++)//V
		memcpy(datas[2] + width / 2 * i, frame->data[0] + frame->linesize[2] * i, width);

You can see , You can see some shapes , But there are still flower screens , And the color is green ：

To be analyzed in detail ing

4. Solve the stuck problem

stay Qt 5 Of GUI In the program , The main thread is also called GUI Threads , Because it is the only one allowed to execute GUI Threads for related operations . For some very time-consuming operations with a large amount of calculation , If placed in the main thread , The interface fails to respond . One way to solve this problem is , Put these time-consuming operations into the secondary thread , There is also a relatively simple method ： Add a delay in processing time-consuming operations , And call QCoreApplication::processEvents(). This function tells Qt To deal with all kinds of core events that have not been dealt with , Then return control to the caller .QElapsedTimer Member function of elapsed() The function returns the value of the QElapsedTimer Milliseconds since . The following code is to add a 25 Millisecond delay to process core events .

QT Middle thread qthread Running causes a stuck solution ：

To put the function of an infinite loop into run In the function , meanwhile msleep A few milliseconds .

In unpacked run The infinite loop in is unlocked after each loop msleep A few （3） millisecond （ In the video thread run The same can be done in the infinite loop of , But the effect is not well done in unpacking ）

void xdemuxthread::run()
{
    
	while (!isexit)
	{
    
		mux.lock();
		if (!demux)
		{
    
			mux.unlock();// if demux If it is not opened, unlock it to let other threads in , wait for 5 Millisecond expectation demux Can open the , Then make the next judgment , Judge demux Whether to open 
			msleep(5);
			continue;
		}
		if (vt && at)
		{
    
			vt->synpts = at->pts;
		}
		AVPacket* pkt = demux->readfz();
	
        if (!pkt)
		{
    
			// It's the end 
			mux.unlock();
			msleep(5);
			continue;
		}
		if (demux->isvideo(pkt))
		{
    
			if (vt)vt->push(pkt);
		}
		else
		{
    
			if (at)at->push(pkt);
		}
		mux.unlock();
		msleep(3);
	}
}

Problem solvable
Guess why ： The interval between each cycle is too short , Video threads consume a lot of resources , Very few thread resources are preempted, resulting in a jam .