PS： If you want to reprint it, please indicate the source , All rights reserved .
PS: This is just based on 《 myself 》 The understanding of the ,
If it conflicts with your principles and ideas , Please understand , Don't spray .
Preposition
As the author of this paper csdn blog Backup of the master station of .（BlogID=116）
Environmental statement
- android mobile phone
- linux python Environmental Science
Preface
In recent months , For me, , A lot of things have happened , Lead to many idea, But no good articles have been formed . lately , Take advantage of this opportunity , Write an essay .
Due to the arrangement of business , We need to be in c/c++ Layer and java and python Layer for data exchange , The amount of data varies , However, our business has certain requirements for the delay of this data exchange , Therefore, some problems need to be solved . In the course of our experiment , We found that under normal circumstances , stay jvm Chinese new creation ByteArray/FloatArray Do big data （6Mb byte/2Mb floats） The transmission of , Time in 5ms/7ms, stay pvm Chinese new creation bytearray Large amount of data （8Mb byte） The transmission of , Time in 1ms about . From the experimental situation , We need to optimize jvm The method of mass data transmission in .
I've written about it before java,python and c/cpp Some interactive articles , If you are interested, please refer to .
- 《C++ call Python summary ( One )》 https://blog.csdn.net/u011728480/article/details/103903612
- 《java Generate... Manually jni The header file (JNI Static registration )》 https://blog.csdn.net/u011728480/article/details/87260113
- 《Android JNI Static and dynamic registration 、Java Reflect（C or C++ Layer reflection and JAVA Layer reflection ）、Java Variable parameters （JNI Realization ）》 https://blog.csdn.net/u011728480/article/details/78963494
jvm jni piece
jni There are many conventional data exchange methods on the Internet , Basically, they are as follows :
stay java Go to c/cpp return , Generally, it is the bottom address to obtain data , Then operate on the address .
jbyteArray array;//or jfloatArray array; passed by jni-func
void * _you_wanted_ptr = env->GetPrimitiveArrayCritical(array, nullptr);
// TODO
env->ReleasePrimitiveArrayCritical(array, _you_wanted_ptr, JNI_ABORT);
stay c/cpp Go to java When transmitting large amounts of data , There are two ways , One is direct new An array , Then return to the way , One is to get java Array address of layer , Then directly modify the relevant data . Its basic is as follows ：
// slow way
int len = xxx;
void * data_ptr = xxx;
jXXXArray array = env->NewXXXArray(len);
env->SetXXXArrayRegion(array, 0, len, (const jXXX *) data_ptr);
return array;
// fast way
jbyteArray array;//or jfloatArray array; passed by jni-func
int len = xxx;
void * data_ptr = xxx;
env->SetXXXArrayRegion(array, 0, len, (const jXXX *) data_ptr);
It is used here fast way After the model , stay jvm For large data volume （6Mb byte/2Mb floats） The transmission of , Time in 0.88ms/1ms, Be careful , There are restrictions on use . Here we must pay attention to the problem of multithreading safety .
pvm pybind11 piece
stay pybind11 in , There are generally two data structures for large-scale data transmission , One is py::bytes, One is the common numpy Array , Especially in image processing ,numpy Array is the most common format . below , According to these two ways , Respectively introduced .
py::bytes Type transmission
python Pass the layer to c/cpp.
const py::bytes &value;//passed by pybind11-func
Py_ssize_t size = PyBytes_GET_SIZE(value.ptr());
char * ptr = PyBytes_AsString(value.ptr());
//TODO
c/cpp Pass the layer to python.
char * buf = xxx;
int len = xxx;
return py::bytes(buf, len);//In pybind11, return to pvm
Be careful , stay py::bytes in , There are also ways to modify the address directly , It's not available here （python buffer protocol）, Those who want to study it by themselves .
numpy The data transfer
This is also like py::bytes Create an array like that , Then return to the way , It's not available here . Here is mainly a demonstration of how to quickly c/cpp In order to get numpy data . In fact, the data transmission here is direct acquisition numpy Array address , Basically not bad .
c/cpp To python
// python buffer protocol
py::array_t<float, py::array::c_style | py::array::forcecast> &buffer;//passed by pybind11-func
auto buf_info = buffer.unchecked<1>();
char * ptr = (char *)buf_info.data(0)
// set value to ptr(numpy)
// get value from ptr(numpy)
Be careful , Here we use a called python buffer protocol Things that are , If you are interested, you can have a look , I didn't delve into this .
pybind11 Memory management problems in
stay pybind11 in , Be careful with memory management , In particular, note the difference between the following two calls .
according to https://pybind11.readthedocs.io/en/stable/advanced/classes.html#non-public-destructors Explanation , We usually have two situations to choose .
// Single case
class MyClass{
private:
~MyClass(){}
};
// prohibit unique_ptr call Destructor , All resources need to be released in cpp Side to finish .
py::class_<MyClass, std::unique_ptr<MyClass, py::nodelete>>(m, "MyClass")
.def(py::init<>())
// commonly class
class MyClass{
public:
~MyClass(){}
};
// unique_ptr The destructor is called automatically when destructing , All resources are released by unique_ptr complete .
py::class_<MyClass, std::unique_ptr<MyClass>>(m, "MyClass")
.def(py::init<>())
Postscript
in general , stay jvm and pvm in , Fix the underlying pointer of the array by operation , We can get and transmit data quickly . But there are some phenomena , For example, we need to pay attention to some atomic operations and pvm/jvm The life cycle of arrays in , I suggest that , If it is large-scale data transmission , It is recommended to directly global array , This ensures that life cycle issues .
reference
[1]https://pybind11.readthedocs.io/en/stable/advanced/classes.html#non-public-destructors
PS: Please respect the original , No joy, no spray. .
PS: If you want to reprint it, please indicate the source , All rights reserved .
PS: Please leave a message if you have any questions , I will reply as soon as I see it .
