H265 enables mobile phone screen projection

H265 code

Why is there H265

• Video resolution from 720p To 1080P Let's go to the back 4k 8k Television is booming
  • Video frame rate from 30 frame To 60 frame , Until then 120 frame
  • The number of macroblocks has exploded
  • Macroblock complexity is reduced
  • The complexity of motion vectors is greatly increased

H265 The advantages of

• 1、 Reduce bitstream , Improve coding efficiency , H.265 It provides more diversified means to reduce the bit stream . Besides the improvement of encoding and decoding efficiency , Adaptability to the network H.265 There is also a significant improvement , It can run well under the condition of multiple complex networks . So video conferencing applications H.265, It can ensure low network bandwidth , High resolution video playback is still possible
• 2、 The high quality 1080P60 image quality , Tradition H.264 Video conference system , stay 10Mb Of network bandwidth , Want to achieve 1080P30 Real time communication effect , It is already quite difficult . Now use H.265 Codec technology , This situation has been greatly improved , Support in the same bandwidth , Achieve higher than 1080P30 achieve 1080P60 Even 4k Video playback of , Greatly improve the sense of interaction and realism . It also means that ：H.265 Able to work with limited bandwidth , Deliver higher quality video content , It not only makes video conference users experience better effects , It also reduces the pressure of high-definition video transmission in the network bandwidth , Reduce the bandwidth cost of video conferencing .
• 3、 Reduce delay , More efficient and faster . H.265 Codec in H.264 A large number of technological innovations are carried out on the basis of , In particular, it has achieved remarkable results in reducing real-time delay , It reduces information acquisition time 、 Reduce random access delay 、 Reduce the algorithm complexity and other multi-dimensional technical advantages to achieve .

H265 characteristic

• H265 Change the size of the macroblock from H264 Of 16x16 Extended to 64x64, To facilitate the compression of high-resolution video
• H265 A more flexible coding structure is adopted to improve the coding efficiency undefined Including coding units （ similar H264 Macroblock , Used to code ）、 Prediction unit and transformation unit .
• H265 Intra prediction
  • H265: be-all CU block , The brightness has 35 Two prediction directions , chroma 5 Kind of
  • H264： brightness 4x4 and 8x8 Blocks are all 9 A direction ,16x16 yes 4 Two directions , chroma 4 Two directions

H265 Code stream analysis

• About SPS/PPS/IDR/P/B The concept will not be explained in detail here .H264 and H265 Every one of NALU The prefix code is the same , namely “0x00 00 00 01” perhaps “0x00 00 01”. You can see my previous article Android Audio and video development ——H264 Basic concepts of
• H264 The frame type of , because H264 It's the back 5 Bit saves frame type data , So with 1F that will do

image.png

• H265 The frame type of ： take value&7E>>1 You can get the frame type

image.png

• The type we often need

The frame type	value
vps	32
sps	33
pps	34
IDR	19
P	1
B	0

The example analysis

image.png

• We use 40 01 For example

    0100  0000  40

&   0111  1110  7E

=   0100  0000  40

>>1 0010  0000  =32

What we found out was that 32 That is to say vps

• 42 01 For example, we find that the result is 33, That is to say sps

    0100  0010  42

&   0111  1110  7E

=   0100  0010  42

>>1 0010  0001  =33

H265 Realize mobile screen projection

Realization effect .gif

principle

image.png

Core code

First we need to get the data of the recording screen , In fact, it is the coding layer

    public void startLive() {

        try {

            // Server side encoding H264 adopt socket Send to client 

            MediaFormat format = MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_HEVC, mWidth, mHeight);

            format.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface);

            format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1);

            format.setInteger(KEY_BIT_RATE, mWidth * mHeight);

            format.setInteger(KEY_FRAME_RATE, 20);

            mMediaCodec = MediaCodec.createEncoderByType(MediaFormat.MIMETYPE_VIDEO_HEVC);

            mMediaCodec.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);

            // Create site 

            Surface surface = mMediaCodec.createInputSurface();

            mVirtualDisplay = mMediaProjection.createVirtualDisplay("CodecLiveH265",

                    mWidth, mHeight, 1, DisplayManager.VIRTUAL_DISPLAY_FLAG_PUBLIC, surface, null, null);

            mHandler.post(this);

        } catch (IOException e) {

            e.printStackTrace();

        }

    }

    @Override

    public void run() {

        mMediaCodec.start();

        MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();

        while (true) {

            // Take out the data and send it to the client 

            int outIndex = mMediaCodec.dequeueOutputBuffer(bufferInfo, 1000);

            if (outIndex >= 0) {

                ByteBuffer buffer = mMediaCodec.getOutputBuffer(outIndex);

                dealFrame(buffer, bufferInfo);

                mMediaCodec.releaseOutputBuffer(outIndex, false);

            }

        }

    }

If you don't understand anything, you can read my previous articles ：Android Audio and video development ——MedCodec Realize screen recording coding into H264

We need to pay attention to the method of processing frames dealFrame. stay h265 Data in , It only happens once VPS,SPS and PPS, But in the projection process , We must pass every time I At the frame , We need to VPS_PPS_SPS Pass it along

    public static final int NAL_I = 19;

    public static final int NAL_VPS = 32;

    //vps+sps+pps It's a frame , So just get vps

    private byte[] vps_sps_pps_buffer;

    private void dealFrame(ByteBuffer buffer, MediaCodec.BufferInfo bufferInfo) {

        // Filter out the first one 0x00 00 00 01  perhaps 0x 00 00 01

        int offset = 4;

        if (buffer.get(2) == 0x01) {

            offset = 3;

        }

        // Get frame type 

        int type = (buffer.get(offset) & 0x7E) >> 1;

        if (type == NAL_VPS) {

            vps_sps_pps_buffer = new byte[bufferInfo.size];

            buffer.get(vps_sps_pps_buffer);

        } else if (type == NAL_I) {

            //I frame 

            final byte[] bytes = new byte[bufferInfo.size];

            buffer.get(bytes);

            //vps_pps_sps+I Frame data 

            byte[] newBuffer = new byte[vps_sps_pps_buffer.length + bytes.length];

            System.arraycopy(vps_sps_pps_buffer, 0, newBuffer, 0, vps_sps_pps_buffer.length);

            System.arraycopy(bytes, 0, newBuffer, vps_sps_pps_buffer.length, bytes.length);

            mWebSocketSendLive.sendData(newBuffer);

        }else{

            //P frame  B Just send the frame directly 

            final byte[] bytes = new byte[bufferInfo.size];

            buffer.get(bytes);

            mWebSocketSendLive.sendData(bytes);

        }

    }

The next step is for the receiver to parse and obtain buffer

The first step is to initialize the decoder

    // Initialize decoder 

    private fun initDecoder(surface: Surface?) {

        mMediaCodec = MediaCodec.createDecoderByType(MediaFormat.MIMETYPE_VIDEO_HEVC)

        val format =

            MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_HEVC, mWidth, mHeight)

        format.setInteger(MediaFormat.KEY_BIT_RATE, mWidth * mHeight)

        format.setInteger(MediaFormat.KEY_FRAME_RATE, 20)

        format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1)

        mMediaCodec.configure(

            format,

            surface,

            null, 0

        )

        mMediaCodec.start()

    }

The second step , Decode the obtained data

    override fun callBack(data: ByteArray?) {

        // Callback 

        LogUtils.e(" Length of received data :${data?.size}")

        // The client mainly decodes the acquired data , First we need to pass dsp decode 

        val index = mMediaCodec.dequeueInputBuffer(10000)

        if (index >= 0) {

            val inputBuffer = mMediaCodec.getInputBuffer(index)

            inputBuffer.clear()

            inputBuffer.put(data, 0, data!!.size)

            // notice dsp Chips help decode 

            mMediaCodec.queueInputBuffer(index, 0, data.size, System.currentTimeMillis(), 0)

        }

        // Take out the data 

        val bufferInfo = MediaCodec.BufferInfo()

        var outIndex: Int = mMediaCodec.dequeueOutputBuffer(bufferInfo, 10000)

        while (outIndex > 0) {

            mMediaCodec.releaseOutputBuffer(outIndex, true)

            outIndex = mMediaCodec.dequeueOutputBuffer(bufferInfo, 10000)

        }

    }