pyspark on hpc

Local internal cluster resources are limited , Simple data processing has gone 3 God .HPC There are many computing resources on , Out of the idea of eating the pot first and then the bowl , Consider making full use of shared resources first . Simple survey , It's not very complicated .

1 programme

spark use local Pattern

spark standalone It involves multi node communication , High complexity ; Multi task parallelism can be used to plan data fragmentation , One for each individual spark local Handle ; This avoids complex cluster construction . Through the requisition Mo node 、 many cpu、 Multi memory to achieve .

Give Way python The environment can find pyspark

This is essentially through env Environment variable implementation , The specific implementation is python Set up , One .bashrc or shell Set up .

2 step

1) install spark( It's decompression )

decompression spark-3.1.2-bin-hadoop3.2.tgz Go to the user directory , such as /users/username/tools/spark/spark

I used a soft connection , Consider switching between different versions later

cd /users/[username]/tools/
tar -zxvf spark-3.1.2-bin-hadoop3.2.tgz
ln -s spark-3.1.2-bin-hadoop3.2 spark

2） stay python Configure... In the code , To use the pyspark

The following build environment and test code can be found in py Document and jupyter Medium test passed .

import os
import sys
os.environ["PYSPARK_PYTHON"] = "/users/[username]/miniconda3/bin/python"
os.environ["SPARK_HOME"] = "/users/[username]/tools/spark"
os.environ["PYLIB"] = os.environ["SPARK_HOME"] + "/python/lib"
sys.path.insert(0, os.environ["PYLIB"] +"/py4j-10.9-src.zip")
sys.path.insert(0, os.environ["PYLIB"] +"/pyspark.zip")

# test code
import random
from pyspark import SparkContext
sc = pyspark.SparkContext(appName="myAppName")
def inside(p):
    x, y = random.random(), random.random()
    return x*x + y*y < 1
NUM_SAMPLES = 1000000
count = sc.parallelize(range(0, NUM_SAMPLES)) \
             .filter(inside).count()
print("Pi is roughly %f" % (4.0 * count / NUM_SAMPLES))
sc.stop()

3） adopt bashrc Or script configuration pyspark

To configure myspark.sh

#!/bin/sh
export SPARK_HOME='/users/[username]/tools/spark'
export PYTHONPATH=$SPARK_HOME/python:$PYTHONPATH
export PYSPARK_DRIVER_PYTHON="jupyter"
export PYSPARK_DRIVER_PYTHON_OPTS="notebook"
export PYSPARK_PYTHON="/users//[username]/miniconda3/bin/python"

Put this in .bashrc, You don't need the above python To configure , Senseless use pyspark.