Pyspark operator processing spatial data full parsing (5): how to use spatial operation interface in pyspark

park Distributed Computing ,PySpark Actually Python Called Spark The bottom frame of , So how are these frameworks invoked ？ The last one said Python Inside use GDAL Space operators implemented by packages , What about these whole call processes ？ Today, let's explore .

 

The first article in this series said , Want to run PySpark, Need to use Py4J This package , The function of this package is to use Python To call Java Objects in the virtual machine , The principle is ：

 

 

Python Algorithm passed Socket Send the task to Java The virtual machine ,JAVA The virtual machine passes through Py4J This package is parsed , And then call Spark Of Worker Computing node , Then restore the task to Python Realization , After the execution is complete , Walk backwards again .

 

Please check this article for specific instructions , I won't repeat it ：

​ ​http://sharkdtu.com/posts/pyspark-internal.html​​

 

You can see it , use Python What you write , Finally, Worker End , Also use Python Algorithm or package to implement , Now let's do an experiment ：

 

utilize Python Of sys Package to view the running Python Version of , use socket Package to view the machine name of the node , These two bags are Python Peculiar , if PySpark Just run Java Words , On different nodes , It should be impossible to implement .

 

I have two machines here , named sparkvm.com and sparkvmslave.com, among Sparkvm.com yes master + worker, and sparkvmslave.com just worker.

 

The last execution shows , Different results are returned on different nodes .

 

It can be seen from the above experiment that , On different computing nodes , The end use is Python Algorithm package for , So how to use spatial analysis algorithm on different nodes ？

 

stay Spark On , Utilized Algorithm plug-in In this way ：

 

 

As long as the same is installed on different nodes Python Algorithm package , You can do it , The key point is to need Configure the system Python, because PySpark The default call is system Python.

 

Let's do another experiment ：

 

And then in PySpark An example is running above ：

 

Two nodes , Why is it all executed on one node ？ have a look debug Log out ：

 

 

Found in 153 Node , An exception has been thrown , Said he didn't find pygeohash package .

 

Next I'm in 153 above , hold pygeohash Package installation ：

 

 

Then execute the above content again ：

 

Finally, let's take advantage of gdal Spatial algorithm interface , Let's run an example ：

 

To be continued

The source code can be passed through my Gitee perhaps github download ：

github: https://github.com/allenlu2008/PySparkDemo

gitee: https://gitee.com/godxia/PySparkDemo