Capture video by buffering

Don't mention the application for camera permission , Go straight to the key code ：

class MainActivity : AppCompatActivity() {
    

    private val camera = Camera.open()
    private val surfaceTexture = SurfaceTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES)

    override fun onCreate(savedInstanceState: Bundle?) {
    
        super.onCreate(savedInstanceState)
        camera.setPreviewTexture(surfaceTexture)
        camera.setPreviewCallback {
     data, camera ->
            println(" The camera is capturing images ")
        }
        camera.startPreview()
    }

    override fun onDestroy() {
    
        super.onDestroy()
        camera.setPreviewCallback(null)
        camera.stopPreview()
        camera.release()
    }

}

This is a video capture without preview camera , Just a very simple code ,camera.setPreviewCallback Medium data Image data , We didn't do preservation , Every time I get data Will be a new array object , The experiment is as follows ：

class MainActivity : AppCompatActivity() {
    

    private val camera = Camera.open()
    private val surfaceTexture = SurfaceTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES)
    private var oldData = ByteArray(0)

    override fun onCreate(savedInstanceState: Bundle?) {
    
        super.onCreate(savedInstanceState)
        camera.setPreviewTexture(surfaceTexture)
        camera.setPreviewCallback {
     data, camera ->
            if (oldData !== data) {
    
                oldData = data
                println(" new data")
            } else {
    
                println(" used data")
            }
        }
        camera.startPreview()
    }

    override fun onDestroy() {
    
        super.onDestroy()
        camera.setPreviewCallback(null)
        camera.stopPreview()
        camera.release()
    }

}

Run code , It's all printed " new data". If it is 25 frame / second , Every second 25 A new one Byte Array , The performance of constantly creating new objects is relatively low , So you can use buffer objects , Use the same array every time , The code is as follows ：

camera.setPreviewCallbackWithBuffer {
     data, camera ->
    if (oldData !== data) {
    
        oldData = data
        println(" new data")
    } else {
    
        println(" used data")
    }
}

Here is the setPreviewCallback The function is replaced by setPreviewCallbackWithBuffer, Run code , You will find no output , This is because we also need to provide a buffer array , as follows ：

class MainActivity : AppCompatActivity() {
    

    private val camera = Camera.open()
    private val surfaceTexture = SurfaceTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES)
    private val previewSize = camera.parameters.previewSize
    private val width = previewSize.width
    private val height = previewSize.height
    private val callbackBuffer = ByteArray((width * height * 3) shr 1)

    override fun onCreate(savedInstanceState: Bundle?) {
    
        super.onCreate(savedInstanceState)
        camera.setPreviewTexture(surfaceTexture)
        camera.addCallbackBuffer(callbackBuffer)
        camera.setPreviewCallbackWithBuffer {
     data, camera ->
            if (callbackBuffer !== data) {
    
                println(" new data")
            } else {
    
                println(" used data")
            }
        }
        camera.startPreview()
    }

    override fun onDestroy() {
    
        super.onDestroy()
        camera.setPreviewCallback(null)
        camera.stopPreview()
        camera.release()
    }

}

Run code , You'll find that it's only printed once " used data", Just because the system will only call when we camera.addCallbackBuffer(callbackBuffer) after , Just use what we gave buffer Object load a frame of image to us , therefore , When we need data, we need to call camera.addCallbackBuffer(callbackBuffer), If you always need , You need to call all the time camera.addCallbackBuffer(callbackBuffer), Examples are as follows ：

class MainActivity : AppCompatActivity() {
    

    private val camera = Camera.open()
    private val surfaceTexture = SurfaceTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES)
    private val previewSize = camera.parameters.previewSize
    private val width = previewSize.width
    private val height = previewSize.height
    private val callbackBuffer = ByteArray((width * height * 3) shr 1)

    override fun onCreate(savedInstanceState: Bundle?) {
    
        super.onCreate(savedInstanceState)
        camera.setPreviewTexture(surfaceTexture)
        camera.addCallbackBuffer(callbackBuffer)
        camera.setPreviewCallbackWithBuffer {
     data, camera ->
            if (callbackBuffer !== data) {
    
                println(" new data")
            } else {
    
                println(" used data")
            }
            // TODO  Handle data
            // data Finished processing , Then give the buffer object to the framework , Let it fill the image data inside 
            //camera.addCallbackBuffer(data) //  This method can also , It's the same object anyway 
            camera.addCallbackBuffer(callbackBuffer)
        }
        camera.startPreview()
    }

    override fun onDestroy() {
    
        super.onDestroy()
        camera.setPreviewCallback(null)
        camera.stopPreview()
        camera.release()
    }

}

Run the code again , You will see that it has been outputting " used data", This avoids creating new ones for every frame data object , But we need to pay attention , We're dealing with data Data must be fast , Suppose you need 25 frame / second , Then the processing time of each frame is 40 millisecond , We have to 40 Finished processing in milliseconds data, Then take it. data Set to addCallbackBuffer in , If the processing time is slow , It will cause frame loss . such as , We add the statistical code of frame rate , as follows ：

class MainActivity : AppCompatActivity() {
    

    private val camera = Camera.open()
    private val surfaceTexture = SurfaceTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES)
    private val previewSize = camera.parameters.previewSize
    private val width = previewSize.width
    private val height = previewSize.height
    private val callbackBuffer = ByteArray((width * height * 3) shr 1)
    private var fps = 0
    private var start = 0L

    override fun onCreate(savedInstanceState: Bundle?) {
    
        super.onCreate(savedInstanceState)
        println("width = $width height = $height")
        camera.setPreviewTexture(surfaceTexture)
        camera.addCallbackBuffer(callbackBuffer)
        camera.setPreviewCallbackWithBuffer {
     data, camera ->
            fps++
            val current = System.currentTimeMillis()
            if (start == 0L) {
    
                start = current
            }
            if (current - start >= 1000) {
    
                println(" Frame rate ：${
      fps} frame / second ")
                fps = 0
                start = current
            }
            camera.addCallbackBuffer(data)
        }
        camera.startPreview()
    }

    override fun onDestroy() {
    
        super.onDestroy()
        camera.setPreviewCallback(null)
        camera.stopPreview()
        camera.release()
    }

}

The millet 11 pro The operation results are as follows ：

width = 1920 height = 1080
 Frame rate ：28 frame / second 
 Frame rate ：24 frame / second 
 Frame rate ：30 frame / second 
 Frame rate ：30 frame / second 
 Frame rate ：31 frame / second 
 Frame rate ：30 frame / second 
 Frame rate ：31 frame / second 
 Frame rate ：30 frame / second 
 Frame rate ：31 frame / second 
 Frame rate ：30 frame / second 

You can see , The frame rate is not stable , There will also be deviations .

Next , Simulate it, right data Data processing , Suppose this department needs 80 millisecond ：

class MainActivity : AppCompatActivity() {
    

    private val camera = Camera.open()
    private val surfaceTexture = SurfaceTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES)
    private val previewSize = camera.parameters.previewSize
    private val width = previewSize.width
    private val height = previewSize.height
    private val callbackBuffer = ByteArray((width * height * 3) shr 1)
    private var fps = 0
    private var start = 0L
    private val handler = object: Handler(Looper.getMainLooper()) {
    
        override fun handleMessage(msg: Message) {
    
            camera.addCallbackBuffer(callbackBuffer)
        }
    }

    override fun onCreate(savedInstanceState: Bundle?) {
    
        super.onCreate(savedInstanceState)
        println("width = $width height = $height")
        camera.setPreviewTexture(surfaceTexture)
        camera.addCallbackBuffer(callbackBuffer)
        camera.setPreviewCallbackWithBuffer {
     data, camera ->
            fps++
            val current = System.currentTimeMillis()
            if (start == 0L) {
    
                start = current
            }
            if (current - start >= 1000) {
    
                println(" Frame rate ：${
      fps} frame / second ")
                fps = 0
                start = current
            }

            //  Simulate one 80 Millisecond time-consuming operation 
            handler.sendEmptyMessageDelayed(0, 80L)
        }
        camera.startPreview()
    }

    override fun onDestroy() {
    
        super.onDestroy()
        camera.setPreviewCallback(null)
        camera.stopPreview()
        camera.release()
    }

}

The operation effect is as follows ：

width = 1920 height = 1080
 Frame rate ：7 frame / second 
 Frame rate ：8 frame / second 
 Frame rate ：6 frame / second 
 Frame rate ：6 frame / second 
 Frame rate ：8 frame / second 
 Frame rate ：7 frame / second 
 Frame rate ：8 frame / second 
 Frame rate ：8 frame / second 
 Frame rate ：9 frame / second 
 Frame rate ：8 frame / second 

You can see , Because it takes too much time to process data , The frame rate drops rapidly , So the video you see will be too laggy , Not stuck , But the video is not smooth , The picture is incoherent .

Next , Let's change the time-consuming into 30 millisecond , Run again , give the result as follows ：

width = 1920 height = 1080
 Frame rate ：13 frame / second 
 Frame rate ：13 frame / second 
 Frame rate ：14 frame / second 
 Frame rate ：14 frame / second 
 Frame rate ：14 frame / second 
 Frame rate ：14 frame / second 
 Frame rate ：14 frame / second 
 Frame rate ：13 frame / second 
 Frame rate ：14 frame / second 

Theoretically 1 Frame needs 30 millisecond ,1 second （1000 millisecond ） Can handle 33 Frame , But the results showed that only 14 Around the frame , Let's reduce the delay to 10 millisecond , Run again , give the result as follows ：

width = 1920 height = 1080
 Frame rate ：20 frame / second 
 Frame rate ：22 frame / second 
 Frame rate ：31 frame / second 
 Frame rate ：30 frame / second 
 Frame rate ：31 frame / second 
 Frame rate ：31 frame / second 
 Frame rate ：30 frame / second 
 Frame rate ：30 frame / second 
 Frame rate ：30 frame / second 
 Frame rate ：31 frame / second 
 Frame rate ：29 frame / second 
 Frame rate ：31 frame / second 
 Frame rate ：30 frame / second 
 Frame rate ：31 frame / second 

You can see , The frame rate is restored 30 frame / second , So the video won't get stuck , however 10 Processing time in milliseconds , It's not enough in real development , Each frame of image needs to be rotated 、 Add watermark 、H264 code 、 Network sending and a series of operations ,10 Milliseconds can't be completed at all , So the solution is multithreading , Double buffering mechanism can be used , Like video capture 、 format conversion 、 rotate 、 Add watermark 、 code 、 send out , Double buffering mechanism can be added between every two connected steps , In this way, each step can be run in parallel .