Dear reader,

I installed PyroSAR, the needed extensions (SQLite + SpatiaLite) and SNAP as described here on Github on a Ubuntu server. Afterwards I was following the instruction of the example and everything works fine. BUT when I try to pre-process one Sentinel-1 image via pyroSAR, the geocode function results in an error.

The script I am using follows the instructions also given by the Readme on the front-page:

Script

from pyroSAR import identify
from pyroSAR.snap import geocode

name = 'path/S1A_IW_SLC__1SDV_20180605T111503_20180605T111531_022220_026773_82A0.zip'
scene = identify(name)
resolution = 20
outdir = 'path'

geocode(infile=scene, outdir=outdir, tr=resolution, scaling='db')

Result

In the outdir a .xml-file was created, but nothing else.

Error

Traceback (most recent call last):
  File "pre_processing_test.py", line 17, in <module>
    geocode(infile=scene, outdir=outdir, tr=resolution, scaling='db')
  File "/usr/local/lib/python2.7/dist-packages/pyroSAR/snap/util.py", line 266, in geocode
    gpt(outname + '_proc.xml')
  File "/usr/local/lib/python2.7/dist-packages/pyroSAR/snap/auxil.py", line 214, in gpt
    proc = sp.Popen(cmd, stdout=sp.PIPE, stderr=sp.PIPE)
  File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Question

The Question I have is, whether I am missing something I should have done during the installation or what the origin of that error is in general. Thank you very much in advance and if you need any additional information, please let me know.

Greetings, Johannes

• which operating system are you using?
  Linux
• which environment is pyroSAR running in?
  Anaconda Docker env
• which version of pyroSAR are you using?
  '0.15.1' from conda
• which function of pyroSAR did you call with which parameters?
  geocode(infile=check, outdir='test2')
• if applicable, which version of SNAP or GAMMA are you using in pyroSAR?

  snap version
/opt/snap/bin/../platform/lib/nbexec: WARNING: environment variable DISPLAY is not set
There are parameters but nobody wants to proces them: [version]

• the full error message none it just never ends. I think it's a SNAP issue actually but have not the knowledge to figure out what exactly.

Some initial work towards replacing the qc.sentine1.eo.esa.int endpoint with new scihub.copernicus.eu/gnss endpoint for OSV using the OpenSearchAPI functionality.

Poked around and looked like this is the place that had the qc.sentinel1 dependency, still need to add a few bits to get this fully functional I believe. @johntruckenbrodt let me know if I'm going in the wrong direction here.

Done / TODO:

• [x] support current querying/filtering and getting links in OSV.catch
• [x] Add page support in OSV.catch (currently only first 10 items)
• [x] Download in same format / directory as current method (OSV.retrieve)

Additionally this endpoint seems to be very incomplete, I did not see orbit files before ~ Dec 2020 - Jan 2021 and after Feb 25th 2021, despite it being tapped as the official source onward

Hi John,

i've recently set up a new testing environment for our SLC-processor on Ubuntu 22.04 with pyroSAR 0.19, SNAP 9 and python 3.10. I installed pyroSAR via pip. However, when I start the process and it needs to call init of pyroSAR/drivers.py I get the following error:

File "/home/s1porcessor/.local/lib/python3.10/site-packages/pyroSAR/drivers.py", line 2295, in __init__
    conn = self.engine.connect()
  File "/home/s1porcessor/.local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3315, in connect
    return self._connection_cls(self, close_with_result=close_with_result)
  File "/home/s1porcessor/.local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
    else engine.raw_connection()
  File "/home/s1porcessor/.local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3394, in raw_connection
    return self._wrap_pool_connect(self.pool.connect, _connection)
  File "/home/s1porcessor/.local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3364, in _wrap_pool_connect
    Connection._handle_dbapi_exception_noconnection(
  File "/home/s1porcessor/.local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2198, in _handle_dbapi_exception_noconnection
    util.raise_(
  File "/home/s1porcessor/.local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
  File "/home/s1porcessor/.local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3361, in _wrap_pool_connect
    return fn()
  File "/home/s1porcessor/.local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 320, in connect
    return _ConnectionFairy._checkout(self)
  File "/home/s1porcessor/.local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 884, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/home/s1porcessor/.local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 486, in checkout
    rec = pool._do_get()
  File "/home/s1porcessor/.local/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 256, in _do_get
    return self._create_connection()
  File "/home/s1porcessor/.local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 266, in _create_connection
    return _ConnectionRecord(self)
  File "/home/s1porcessor/.local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 381, in __init__
    self.__connect()
  File "/home/s1porcessor/.local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 677, in __connect
    with util.safe_reraise():
  File "/home/s1porcessor/.local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/home/s1porcessor/.local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
  File "/home/s1porcessor/.local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 673, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
  File "/home/s1porcessor/.local/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 578, in connect
    return dialect.connect(*cargs, **cparams)
  File "/home/s1porcessor/.local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 598, in connect
    return self.dbapi.connect(*cargs, **cparams)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
(Background on this error at: https://sqlalche.me/e/14/e3q8)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/s1porcessor/Documents/s1_processor/s1_processor/slc_processor/slc_processor.py", line 22, in <module>
    S1_SLC_proc(**proc)
  File "/home/s1porcessor/Documents/s1_processor/s1_processor/slc_processor/functions.py", line 2088, in S1_SLC_proc
    with Archive(dbfile= database_path) as archive:
  File "/home/s1porcessor/.local/lib/python3.10/site-packages/pyroSAR/drivers.py", line 2299, in __init__
    raise RuntimeError('could not load spatialite extension')
RuntimeError: could not load spatialite extension

For sqlite3 and spatialite I have found the following packages:

sqlite3-doc/jammy,jammy 3.37.2-2 all
sqlite3-pcre/jammy 0~git20070120091816+4229ecc-2 amd64
sqlite3-tools/jammy 3.37.2-2 amd64
sqlite3-tools/jammy 3.37.2-2 i386
sqlite3/jammy,now 3.37.2-2 amd64 [installed]
sqlite3/jammy 3.37.2-2 i386
sqlite/jammy 2.8.17-15fakesync1build1 amd64
sqlitebrowser/jammy 3.12.1-2 amd64
spatialite-bin/jammy,now 5.0.1-1build1 amd64 [installed]
spatialite-gui/jammy 2.1.0~beta1-1build1 amd64

Am I missing some packages or extension or is this a error specific to a python or sqlite3 version?

Best regards Johannes

Hi there,

Firstly, thanks so much for developing this software! In general it works brilliantly, but when running geocode to process s1 scenes, for some scenes I am getting an error relating to a missing file from snap:

Error: [NodeId: Read] Specified 'file' [/data/Sentinel-1/processed_to_dB/processed_no_filter/T30VUJ/2020/07/30/pyrosar/S1B__IW___D_20200730T065341_NR_Orb_TF_TC_dB/temp/S1B_IW_GRDH_1SDV_20200730T065341_20200730T065406_022697_02B13D_11A5_tmp0.dim] does not exist.

The full error message is provided at the end of this post. I'm currently running the geocode function within a loop to batch process a bunch of sentinel 1 scenes downloaded from the ASF. Most iterations of the loop it works like a dream, but every now and then this error keeps cropping up. Based on the lines printed out to the console, it appears that when the error is thrown, it occurs consistently during the 'ThermalNoiseRemoval' step. Is this something that you have come across before? Any ideas how to fix it?

• which operating system are you using?
  CentOS Linux release 8.3.2011
• which environment is pyroSAR running in?
  system-wide Python installation
• which version of pyroSAR are you using?
  clone of the GitHub repository
• which function of pyroSAR did you call with which parameters?
  geocode
• if applicable, which version of SNAP or GAMMA are you using in pyroSAR? SNAP version 8
• the full error message

`searching for new POE files https://qc.sentinel1.eo.esa.int/api/v1/?product_type=AUX_POEORB&sentinel1__mission=S1B&validity_start__gte=2020-07-29T06:53:41&validity_stop__lte=2020-07-31T06:53:41 executing node sequences.. Subset ThermalNoiseRemoval Executing processing graph done. INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL 3.0.0 found on system. JNI driver will be used. INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP. INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience. INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP. org.esa.snap.core.gpf.graph.GraphException: [NodeId: Read] Specified 'file' [/data/Sentinel-1/processed_to_dB/processed_no_filter/T30VUJ/2020/07/30/pyrosar/S1B__IW___D_20200730T065341_NR_Orb_TF_TC_dB/temp/S1B_IW_GRDH_1SDV_20200730T065341_20200730T065406_022697_02B13D_11A5_tmp0.dim] does not exist. at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:79) at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:199) at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182) at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182) at org.esa.snap.core.gpf.graph.GraphContext.initOutput(GraphContext.java:166) at org.esa.snap.core.gpf.graph.GraphContext.(GraphContext.java:85) at org.esa.snap.core.gpf.graph.GraphContext.(GraphContext.java:58) at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcessor.java:118) at org.esa.snap.core.gpf.main.DefaultCommandLineContext.executeGraph(DefaultCommandLineContext.java:86) at org.esa.snap.core.gpf.main.CommandLineTool.executeGraph(CommandLineTool.java:547) at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:391) at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:287) at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188) at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121) at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54) at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55) at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189) at org.esa.snap.runtime.Launcher.run(Launcher.java:51) at org.esa.snap.runtime.Launcher.main(Launcher.java:31) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:84) at com.install4j.runtime.launcher.UnixLauncher.start(UnixLauncher.java:66) at install4j.org.esa.snap.runtime.Launcher_gpt.main(Unknown Source) Caused by: org.esa.snap.core.gpf.OperatorException: Specified 'file' [/data/Sentinel-1/processed_to_dB/processed_no_filter/T30VUJ/2020/07/30/pyrosar/S1B__IW___D_20200730T065341_NR_Orb_TF_TC_dB/temp/S1B_IW_GRDH_1SDV_20200730T065341_20200730T065406_022697_02B13D_11A5_tmp0.dim] does not exist. at org.esa.snap.core.gpf.common.ReadOp.initialize(ReadOp.java:113) at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:528) at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:298) at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:385) at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:77) ... 30 more

Error: [NodeId: Read] Specified 'file' [/data/Sentinel-1/processed_to_dB/processed_no_filter/T30VUJ/2020/07/30/pyrosar/S1B__IW___D_20200730T065341_NR_Orb_TF_TC_dB/temp/S1B_IW_GRDH_1SDV_20200730T065341_20200730T065406_022697_02B13D_11A5_tmp0.dim] does not exist.`

Thanks again!

Hello,

I am having trouble geocoding scenes when specifying a shapefile in order to reduce processing time. There seems to be a problem with the RemoveThermalNoise step (which failed with an exception). It is to note the function is running smoothly if I do not specify the site shapefile.

• which operating system are you using?
  Ubuntu 18.4

• which environment is pyroSAR running in?
  A dockerised environement

• which version of pyroSAR are you using?
  Installed via pip: pyroSAR==0.11

• which function of pyroSAR did you call with which parameters?
  geocode(infile=scene, outdir='./experiment_v1/subset', tr=10, scaling='dB', shapefile='shape.shp')

• if applicable, which version of SNAP or GAMMA are you using in pyroSAR? SNAP version 7.0.0

• the full error message

Executing processing graph
java.lang.NullPointerException
 done.
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.hsqldb.persist.Logger: dataFileCache open start
org.esa.snap.core.gpf.graph.GraphException: [NodeId: ThermalNoiseRemoval] java.lang.NullPointerException
	at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:79)
	at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:195)
	at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:178)
	at org.esa.snap.core.gpf.graph.GraphContext.initOutput(GraphContext.java:162)
	at org.esa.snap.core.gpf.graph.GraphContext.<init>(GraphContext.java:91)
	at org.esa.snap.core.gpf.graph.GraphContext.<init>(GraphContext.java:64)
	at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcessor.java:128)
	at org.esa.snap.core.gpf.main.DefaultCommandLineContext.executeGraph(DefaultCommandLineContext.java:86)
	at org.esa.snap.core.gpf.main.CommandLineTool.executeGraph(CommandLineTool.java:534)
	at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:388)
	at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:287)
	at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
	at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
	at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54)
	at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
	at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
	at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
	at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:65)
	at com.install4j.runtime.launcher.UnixLauncher.main(UnixLauncher.java:57)
Caused by: org.esa.snap.core.gpf.OperatorException: java.lang.NullPointerException
	at org.esa.snap.engine_utilities.gpf.OperatorUtils.catchOperatorException(OperatorUtils.java:415)
	at org.esa.s1tbx.calibration.gpf.Sentinel1RemoveThermalNoiseOp.initialize(Sentinel1RemoveThermalNoiseOp.java:165)
	at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:486)
	at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:273)
	at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:387)
	at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:77)
	... 28 more
Caused by: java.lang.NullPointerException
	at org.esa.s1tbx.calibration.gpf.Sentinel1RemoveThermalNoiseOp.getProductType(Sentinel1RemoveThermalNoiseOp.java:179)
	at org.esa.s1tbx.calibration.gpf.Sentinel1RemoveThermalNoiseOp.initialize(Sentinel1RemoveThermalNoiseOp.java:140)
	... 32 more

Error: [NodeId: ThermalNoiseRemoval] java.lang.NullPointerException

[S1B_IW_GRDH_1SDV_20190501T180408_20190501T180433_016054_01E318_F627_tmp1.xml] failed with return code 1
failed: experiment_v1/subset/S1B__IW___A_20190501T180408_NR_Orb_TF_TC_dB/S1B__IW___A_20190501T180408_NR_Orb_TF_TC_dB_proc.xml

This adds a new Handler class to handle .dim files which are used by SNAP software.

Fixes #176.

Things to note:

• getCorners() may not be needed under BEAM_DIMAP because I extracted the meta['corners'] data from gdalinfo from the raster data. Or perhaps I should still include this under BEAM_DIMAP.getCorners()?
• Only tested with Sentinel-1 data (I don't have access to other sensors)
• Added flexibility for datasets with no CRS data.
• Tries to get all available metadata from the .dim first before loading the raster in GDAL.

Here is the sample output when using a terrain corrected .dim file with from pyroSAR import identify:

pyroSAR ID object of type BEAM_DIMAP
acquisition_mode: IW
cycleNumber: 113
frameNumber:
lines: 3176
orbit: ASCENDING
orbitNumber_abs: 17393
orbitNumber_rel: 171
polarizations: ['VV']
product: Sentinel-1 IW Level-1 SLC Product
projection: +proj=longlat +ellps=WGS84 +no_defs
samples: 2771
sensor: SENTINEL-1A
spacing: (0.0003772924193301991, -0.0003772924193301991)
start: 09-JUL-2017 11:24:23.927343
stop: 09-JUL-2017 11:24:40.803460

Let me know what you think @johntruckenbrodt. Hopefully I got the structure of your software right.

I am aware of the current discussion about the orbit files. SNAP Forum If I correctly identify the platform used to receive the Orbite files, you are already using the new address, correct?

However, is there a solution to get all orbite files starting at the beginning of the S1 mission? Even the files from 2020 seem not to be available with your functions...

Is there a possibility to search for orbit files on both adresses https://scihub.copernicus.eu and https://qc.sentinel1.eo.esa.int/?

• which operating system are you using?
  Windows 10

• which environment is pyroSAR running in?
  Anaconda environment (python3.8)

• which version of pyroSAR are you using?
  0.12.1

• which function of pyroSAR did you call with which parameters?

with OSV(osvdir) as osv:
    osv_files = osv.catch(sensor='S1A', osvtype='POE',
                     start='20150101T000000', stop='20211231T000000')
    osv.retrieve(osv_files)

• if applicable, which version of SNAP or GAMMA are you using in pyroSAR?

• the full error message

searching for new POE files https://scihub.copernicus.eu/gnss/search/?q=producttype:AUX_POEORB platformname:Sentinel-1 filename:S1A* beginPosition:[2015-01-01T00:00:00Z TO 2021-12-31T00:00:00Z] endPosition:[2015-01-01T00:00:00Z TO 2021-12-31T00:00:00Z]&format=json found 49 OSV results downloading 1 file deleting 0 RES files

I have a set of Level 1 GRD data taken from Copernicus which I wanted to geocode and calibrate using pyroSAR.

OS: Ubuntu 18.04

Environment Used: Anaconda environment

Library versions: pyroSAR: version 0.11.1 from conda-forge SNAP: 8.0

Files:

S1A_EW_GRDH_1SSH_20200317T225506_20200317T225606_031721_03A873_CB43.zip
S1A_EW_GRDM_1SSH_20191215T231834_20191215T231938_030365_037956_C2B6.zip
S1B_EW_GRDH_1SDH_20190820T075249_20190820T075349_017666_0213C3_F0A9.zip

Code:

from pathlib import Path
from pyroSAR.snap import geocode 

sar_zips = [str(Path.cwd() / i) for i in Path("./data").iterdir() if i.suffix == ".zip"]
outdir = "./outdir"

out_proj = 4326

geocode(
    infile=sar_zips[0],
    outdir=outdir,
    t_srs=out_proj,
    scaling="linear",
    speckleFilter="Refined Lee",
)

However the process produced a (seemingly) blank TIF.

Output of gdalinfo S1A__EW___A_20200317T225506_HH_NR_Orb_TF_Spk_TC.tif

Driver: GTiff/GeoTIFF
Files: S1A__EW___A_20200317T225506_HH_NR_Orb_TF_Spk_TC.tif
       S1A__EW___A_20200317T225506_HH_NR_Orb_TF_Spk_TC.tif.aux.xml
Size is 58554, 27919
Coordinate System is:
GEOGCRS["WGS 84",
    DATUM["World Geodetic System 1984",
        ELLIPSOID["WGS 84",6378137,298.257223563,
            LENGTHUNIT["metre",1]]],
    PRIMEM["Greenwich",0,
        ANGLEUNIT["degree",0.0174532925199433]],
    CS[ellipsoidal,2],
        AXIS["geodetic latitude (Lat)",north,
            ORDER[1],
            ANGLEUNIT["degree",0.0174532925199433]],
        AXIS["geodetic longitude (Lon)",east,
            ORDER[2],
            ANGLEUNIT["degree",0.0174532925199433]],
    ID["EPSG",4326]]
Data axis to CRS axis mapping: 2,1
Origin = (-55.043315251243307,-58.879481151356849)
Pixel Size = (0.000179663056824,-0.000179663056824)
Metadata:
  AREA_OR_POINT=Area
  TIFFTAG_RESOLUTIONUNIT=1 (unitless)
  TIFFTAG_XRESOLUTION=1
  TIFFTAG_YRESOLUTION=1
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  ( -55.0433153, -58.8794812) ( 55d 2'35.93"W, 58d52'46.13"S)
Lower Left  ( -55.0433153, -63.8954940) ( 55d 2'35.93"W, 63d53'43.78"S)
Upper Right ( -44.5233246, -58.8794812) ( 44d31'23.97"W, 58d52'46.13"S)
Lower Right ( -44.5233246, -63.8954940) ( 44d31'23.97"W, 63d53'43.78"S)
Center      ( -49.7833199, -61.3874876) ( 49d46'59.95"W, 61d23'14.96"S)
Band 1 Block=496x384 Type=Float32, ColorInterp=Gray
  NoData Value=0
  Metadata:
    STATISTICS_VALID_PERCENT=0

Could someone explain why this occurs? I'm new to SAR image processing so kindly let me know if my workflow is incorrect.

searching for new POE files Traceback (most recent call last): File "/home/c3urma/repositories/pyroSAR/pyroSAR/S1/auxil.py", line 197, in catch response = urlopen(subaddress, context=self.sslcontext).read().decode('utf-8') File "/usr/lib64/python3.6/urllib/request.py", line 223, in urlopen return opener.open(url, data, timeout) File "/usr/lib64/python3.6/urllib/request.py", line 532, in open response = meth(req, response) File "/usr/lib64/python3.6/urllib/request.py", line 642, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib64/python3.6/urllib/request.py", line 570, in error return self._call_chain(*args) File "/usr/lib64/python3.6/urllib/request.py", line 504, in _call_chain result = func(*args) File "/usr/lib64/python3.6/urllib/request.py", line 650, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 500: Internal Server Error

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "example_scoop_gamma.py", line 136, in osv.update() File "/home/c3urma/repositories/pyroSAR/pyroSAR/S1/auxil.py", line 423, in update raise e File "/home/c3urma/repositories/pyroSAR/pyroSAR/S1/auxil.py", line 421, in update files_poe = self.catch(sensor=['S1A', 'S1B'], osvtype='POE', start=self.maxdate('POE', 'start')) File "/home/c3urma/repositories/pyroSAR/pyroSAR/S1/auxil.py", line 200, in catch raise RuntimeError(e) RuntimeError: HTTP Error 500: Internal Server Error

OS: Ubuntu 18.04

Environment Used: Anaconda environment

Library versions: pyroSAR: version 0.11.1 from conda-forge SNAP: 8.0

Objective: I wanted to reproject (to South LAEA) and apply speckle filtering to a couple of Sentinel SAR images that I have acquired. However I am unable to do so.

Code:

from pathlib import Path
from pyroSAR.snap import geocode

outdir = './outdir'
# South Lambert Equal Area Projection (ESRI:102020)
out_proj = "+proj=laea +lat_0=-90 +lon_0=0 +x_0=0 +y_0=0 +ellps=WGS84 +datum=WGS84 +units=m +no_defs"

sar_zips = [ str(Path.cwd() / i) for i in Path('./data').iterdir() if i.suffix == ".zip"]
geocode(infile=sar_zips[0], outdir=outdir, t_srs=out_proj, speckleFilter="Lee")

This snippet outputs

searching for new POE files
https://qc.sentinel1.eo.esa.int/api/v1/?product_type=AUX_POEORB&sentinel1__mission=S1B&validity_start__gte=2019-08-19T07:52:49&validity_stop__lte=2019-08-21T07:52:49
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-15-10d2a2932d21> in <module>
----> 1 geocode(infile=sar_zips, outdir=outdir, t_srs=out_proj, speckleFilter="Lee")

~/anaconda3/envs/sar/lib/python3.6/site-packages/pyroSAR/snap/util.py in geocode(infile, outdir, t_srs, tr, polarizations, shapefile, scaling, geocoding_type, removeS1BorderNoise, removeS1BorderNoiseMethod, removeS1ThermalNoise, offset, allow_RES_OSV, demName, externalDEMFile, externalDEMNoDataValue, externalDEMApplyEGM, terrainFlattening, basename_extensions, test, export_extra, groupsize, cleanup, gpt_exceptions, gpt_args, returnWF, nodataValueAtSea, demResamplingMethod, imgResamplingMethod, speckleFilter, refarea)
    384 
    385     try:
--> 386         t_srs = crsConvert(t_srs, 'epsg')
    387     except TypeError:
    388         raise RuntimeError("format of parameter 't_srs' not recognized")

~/anaconda3/envs/sar/lib/python3.6/site-packages/spatialist/auxil.py in crsConvert(crsIn, crsOut)
     77         return srs.ExportToProj4()
     78     elif crsOut == 'epsg':
---> 79         srs.AutoIdentifyEPSG()
     80         return int(srs.GetAuthorityCode(None))
     81     elif crsOut == 'opengis':

~/anaconda3/envs/sar/lib/python3.6/site-packages/osgeo/osr.py in AutoIdentifyEPSG(self, *args)
    564     def AutoIdentifyEPSG(self, *args):
    565         """AutoIdentifyEPSG(SpatialReference self) -> OGRErr"""
--> 566         return _osr.SpatialReference_AutoIdentifyEPSG(self, *args)
    567 
    568 

RuntimeError: OGR Error: Unsupported SRS

Edit: I tried using osr and importing the wkt representation of South LAEA

wkt = """PROJCS["South_Pole_Lambert_Azimuthal_Equal_Area",GEOGCS["GCS_WGS_1984",DATUM["WGS_1984",SPHEROID["WGS_1984",6378137,298.257223563]],PRIMEM["Greenwich",0],UNIT["Degree",0.017453292519943295]],PROJECTION["Lambert_Azimuthal_Equal_Area"],PARAMETER["False_Easting",0],PARAMETER["False_Northing",0],PARAMETER["Central_Meridian",0],PARAMETER["Latitude_Of_Origin",-90],UNIT["Meter",1],AUTHORITY["EPSG","102020"]]"""
out_proj2 = osr.SpatialReference(wkt)

geocode(
    infile=sar_zips[0],
    outdir=outdir,
    t_srs=out_proj2,
    scaling="linear",
    speckleFilter="Refined Lee",
)

This worked till the Terrain Correction part, but failed for a similar reason.

searching for new POE files
https://qc.sentinel1.eo.esa.int/api/v1/?product_type=AUX_POEORB&sentinel1__mission=S1A&validity_start__gte=2020-03-16T22:55:06&validity_stop__lte=2020-03-18T22:55:06
unpacking scene
removing border noise..
border noise removal not necessary for IPF version 3.2
executing node sequences..
ThermalNoiseRemoval
Apply-Orbit-File
Calibration
Terrain-Flattening
Speckle-Filter
Terrain-Correction
Executing processing graph
No code "EPSG:102020" from authority "European Petroleum Survey Group" found for object of type "CoordinateReferenceSystem".
 done.
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Incompatible GDAL 3.1.4 found on system. Internal GDAL 3.0.0 from distribution will be used.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
INFO: org.hsqldb.persist.Logger: dataFileCache open start
org.esa.snap.core.gpf.graph.GraphException: [NodeId: Terrain-Correction] No code "EPSG:102020" from authority "European Petroleum Survey Group" found for object of type "CoordinateReferenceSystem".
	at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:79)
	at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:199)
	at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
	at org.esa.snap.core.gpf.graph.GraphContext.initOutput(GraphContext.java:166)
	at org.esa.snap.core.gpf.graph.GraphContext.<init>(GraphContext.java:85)
	at org.esa.snap.core.gpf.graph.GraphContext.<init>(GraphContext.java:58)
	at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcessor.java:118)
	at org.esa.snap.core.gpf.main.DefaultCommandLineContext.executeGraph(DefaultCommandLineContext.java:86)
	at org.esa.snap.core.gpf.main.CommandLineTool.executeGraph(CommandLineTool.java:547)
	at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:391)
	at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:287)
	at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
	at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
	at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54)
	at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
	at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
	at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
	at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:84)
	at com.install4j.runtime.launcher.UnixLauncher.start(UnixLauncher.java:66)
	at install4j.org.esa.snap.runtime.Launcher1159904018.main(Unknown Source)
Caused by: org.esa.snap.core.gpf.OperatorException: No code "EPSG:102020" from authority "European Petroleum Survey Group" found for object of type "CoordinateReferenceSystem".
	at org.esa.snap.engine_utilities.gpf.OperatorUtils.catchOperatorException(OperatorUtils.java:440)
	at org.esa.s1tbx.sar.gpf.geometric.RangeDopplerGeocodingOp.initialize(RangeDopplerGeocodingOp.java:349)
	at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:528)
	at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:298)
	at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:385)
	at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:77)
	... 29 more
Caused by: org.esa.snap.core.gpf.OperatorException: No code "EPSG:102020" from authority "European Petroleum Survey Group" found for object of type "CoordinateReferenceSystem".
	at org.esa.s1tbx.sar.gpf.geometric.RangeDopplerGeocodingOp.createTargetProduct(RangeDopplerGeocodingOp.java:564)
	at org.esa.s1tbx.sar.gpf.geometric.RangeDopplerGeocodingOp.initialize(RangeDopplerGeocodingOp.java:306)
	... 33 more
Caused by: org.opengis.referencing.NoSuchAuthorityCodeException: No code "EPSG:102020" from authority "European Petroleum Survey Group" found for object of type "CoordinateReferenceSystem".
	at org.geotools.referencing.factory.AbstractAuthorityFactory.noSuchAuthorityCode(AbstractAuthorityFactory.java:875)
	at org.geotools.referencing.factory.epsg.DirectEpsgFactory.createCoordinateReferenceSystem(DirectEpsgFactory.java:2239)
	at org.geotools.referencing.factory.BufferedAuthorityFactory.createCoordinateReferenceSystem(BufferedAuthorityFactory.java:732)
	at org.geotools.referencing.factory.AuthorityFactoryAdapter.createCoordinateReferenceSystem(AuthorityFactoryAdapter.java:780)
	at org.geotools.referencing.factory.BufferedAuthorityFactory.createCoordinateReferenceSystem(BufferedAuthorityFactory.java:732)
	at org.geotools.referencing.factory.AuthorityFactoryAdapter.createCoordinateReferenceSystem(AuthorityFactoryAdapter.java:780)
	at org.geotools.referencing.factory.FallbackAuthorityFactory.createCoordinateReferenceSystem(FallbackAuthorityFactory.java:625)
	at org.geotools.referencing.factory.AuthorityFactoryAdapter.createCoordinateReferenceSystem(AuthorityFactoryAdapter.java:780)
	at org.geotools.referencing.factory.ThreadedAuthorityFactory.createCoordinateReferenceSystem(ThreadedAuthorityFactory.java:636)
	at org.geotools.referencing.DefaultAuthorityFactory.createCoordinateReferenceSystem(DefaultAuthorityFactory.java:177)
	at org.geotools.referencing.CRS.decode(CRS.java:517)
	at org.esa.s1tbx.commons.CRSGeoCodingHandler.getCRS(CRSGeoCodingHandler.java:173)
	at org.esa.s1tbx.commons.CRSGeoCodingHandler.<init>(CRSGeoCodingHandler.java:55)
	at org.esa.s1tbx.sar.gpf.geometric.RangeDopplerGeocodingOp.createTargetProduct(RangeDopplerGeocodingOp.java:530)
	... 34 more

Error: [NodeId: Terrain-Correction] No code "EPSG:102020" from authority "European Petroleum Survey Group" found for object of type "CoordinateReferenceSystem".

[S1A_EW_GRDH_1SSH_20200317T225506_20200317T225606_031721_03A873_CB43_tmp5.xml] failed with return code 1
failed: ./outdir/S1A__EW___A_20200317T225506_HH_NR_Orb_TF_Spk_TC/S1A__EW___A_20200317T225506_HH_NR_Orb_TF_Spk_TC_proc.xml

How can I correctly reproject my SAR files to South LAEA using pyroSAR?

Hello. I am using pyroSAR to process images. However, I keep getting 2 errors in this runcell.

1. About the error message <UnicodeDecodeError: 'utf-8' code can't code byte 0xb4 in position 620: invalid start byte> When I run this, the file is generated and then the error <UnicodeDecodeError: 'utf-8' code can't code byte 0xb4 in position 620: invalid start byte> appears. Can you tell me what I should do in this case?

• In

• Out

• folder & file

2. Range-doppler

I've understood that <geocoding_type='Range-Doppler'> corrects the orientation of the image. However, with the error message 1), the resulting image's direction does not change in SNAP. Can you tell me how I change direction? (The image below should be rotated 90 degrees to the left.)

• showing in SNAP

sarsen (https://github.com/bopen/sarsen) is a fairly recent initiative to re-implement Sentinel-1 preprocessing and calibration using modern Python libraries such as Dask and XArray. This enables better scalability over distributed clusters of workers and removes the reliance on SNAP for preprocessing (which we can all agree is sometimes a pain to work with).

It would be nice if PyroSAR wrapped sarsen as an alternative SAR preprocessing backend, so that existing pipelines could be moved to use sarsen with little to no change in code.

For the DEM options 'Copernicus 30m Global DEM' and 'Copernicus 90m Global DEM', which are hosted on AWS, pyroSAR.auxdata.dem_autoload currently only enables download of the actual DEM and no ancillary layers. These layers have apparently been made available and this could hence be implemented. See here for example: https://copernicus-dem-30m.s3.eu-central-1.amazonaws.com/?&prefix=Copernicus_DSM_COG_10_N58_00_E044_00_DEM

Hello dear @johntruckenbrodt

First, I’ll like to thank and congratulate you for this excellent work, it has been really useful in my current project.

I’m facing a problem and would like to ask for your advice, one of the areas of interest I’m working on is a costal zone, when I use geocode on the selected scene with Copernicus 30m Global DEM all the ocean and marsh pixels are classified as noData, I’m guessing that the reason is because the DEM use 0 as ocean’s value which causes that ocean’s pixels to be classified as noData value. To try to solve the problem, I’ll like to change the noData value written to my rasters to “nodata” or something like that to avoid such issue. Is it possible to accomplish it with pyroSAR?

I know this problem could be potentially solved using SRTM 1sec HGT, but I’m trying to follow the Digital Earth Africa recommendations (basically Sentinel-Hub product) and they used Copernicus DEM. I would really appreciate if you could help me.

Saludos. Alexis A

The function has two arguments, shapefile and offset, which control the spatial extent that is processed.
It would be convenient to just be able to control this via the spatial extent of the externally provided DEM (as is the case for gamma.geocode).

Passing a VRT to dem_create that contains non-existing source files does not raise an error or at least warn the user of missing files. The resulting DEM is either just partly created or filled with nodata values if none of the source files are available. This could easily be overlooked by users and could result in incorrect results down the line if the output file is passed on to other software automatically.