Scheme proposed by @alexamici:

• [x] remove the (slow) parsing of GCP on open_dataset(..., group=None) and only report available groups by burst_id, e.g. ["IW1/1", "IW1/2", ...]
• [ ] move the naming logic to an external function in xarray_sentinel that takes either the filename or the gcp dataset (whatever is more reasonable) and returns the mapping between the label and the burst_id

I have a bunch of S1 data from the Alaska Satellite Facility in .zip format. I tried seeing what happens if I load in the .zip but it of course didn't work. Are there plans to support ASF data? Or am I doing something wrong? Does it need to be SLC?

Objectives for the accessors are:

• [ ] exposes a hierarchical dataset tree where accessing group datasets without the need to open_dataset again #7
• [ ] exposes original product exploration functions, especially the one that may require slow access when on a network #15
• [ ] exposes a way to get as much as possible of the original metadata #7
• [ ] exposes API to write STAC catalogs to the product folder
• [x] raises a user-friendly error message when not used on a sentinel-1 Dataset

For testing burst opening, we need real data to pass to rioxarray. Of course using the original TIFF files is not feasible, as they can easily be very big, order of GiBs.

I played a bit to solve this issue and I have a few options:

1. Put in the repository an all-zeros TIFF file with the same geometric properties of the original ones. By using the compression facility of TIFFs (method ZSTD gave the best result), I cut down 1.1GiB to 384KiB.
2. Same as before, but changing the block size to the whole image (the rational is that compression is performed at block level). In this way the same TIFF as before reduce to 344KiB.
3. Add setup code to tests that generates the needed TIFF on-the-fly. The code would be very simple, we could put the destination file name under .gitignore to avoid mistakes and we could also "cache" the produced file to avoid generating it repeatedly.

Personally I think the second option is not worth losing the original blocksize, while the third option is more code to maintain and it slightly complicates the tests. Nonetheless, the third option could be useful should we need more meainingful data to put in, for example for calibration.

Group names do not uniquely identify a burst, it possible (even if unlikely) that two relative orbits will have identical group names for different bursts, e.g. group="IW2/N433_W0120_VV".

@aurghs: "The relative orbit is useful mainly to identify uniquely the bursts, I would rather add the orbit in the burst name, for example: group="IW2/N433_W0120_VV_T22""

@alexamici: "I agree with the idea, but prefer: group="IW2/R022_N433_W0120_VV" (R for "relative orbit" and keep the name to a fixed length).

I also noticed that the polarisations should really be bands of the same dataset, not different datasets, but that is another issue (they have identical dimensions and coordinates)"

Originally posted by @alexamici in https://github.com/bopen/xarray-sentinel/pull/8#issuecomment-819350889

The lowest level in exposing the data is to create bursts datasets with the data contained in the measurements TIFF files (real and imaginary part scaled to int16 aa "distribution" digital number, not the original DN).

The step needed are:

1. parse burst information from the product annotation
2. build the burst window into the original TIFF file (line_min, sample_min, line_max, sample_max)
3. map the burst data to a xr.Dataset

Step 3. can be done quickly by creating a VRT on disk and opening it with rioxarray.

Longer term we wan to support products on read-only file-systems or zip files, but there is no way (that I found) to pass the VRT XML to GDAL without saving it first to a file.

Group names do not uniquely identify a burst, it possible (even if unlikely) that two relative orbits will have identical group names for different bursts, e.g. group="IW2/N433_W0120_VV". Adding the relative orbit to the group structure will make all group names longer and more complex, but unique: e.g. group="22/IW2/N433_W0120_VV" and group="169/IW2/N433_W0120_VV".

This is a trade-off. I'm not sure what naming scheme is best.

We could even accept both (causing even more confusion).

At the moment the only way to link the burst to the associated GCPs is to match the azimuth_times and slant_range_times.

It would be helpful to attach some information on localisation of the burst (computed to define the burst name) to the burst dataset.

I'm able to have this run when I git clone and pip install . inside the repository. But when I pip install xarray-sentinel, it successfully installs but throws the error

sentinel1.open_dataset( "S1A_IW_SLC__1SDV_20191130T235546_20191130T235613_030147_0371D0_AEE1.SAFE")
...
URLError: <urlopen error [Errno 2] No such file or directory: '/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/xarray_sentinel/resources/sentinel1/s1-level-1-product.xsd'>

$ ls /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/xarray_sentinel
conventions.py  esa_safe.py  __init__.py  __pycache__  sentinel1.py  version.py

The .xsd files aren't in the directory pip creates, and this is true even after disabling installs from wheel.

I think this might involve setting package_data=True and including a MANIFEST.in file to specify the non-python files to include.

Draft for open open_dataset inferface. open_dataset takes in input both the manifest.safe and the directory.

Open without a group:

manifest_path = (
    DATA_FOLDER
    / "S1B_IW_SLC__1SDV_20210401T052622_20210401T052650_026269_032297_EFA4.SAFE"
)
xr.open_dataset(manifest_path, engine="sentinel-1")

<xarray.Dataset>
Dimensions:  ()
Data variables:
    *empty*
Attributes: (12/13)
    constellation:              sentinel-1
    platform:                   sentinel-1b
    instrument:                 ['c-sar']
    sat:orbit_state:            descending
    sat:absolute_orbit:         26269
    sat:relative_orbit:         168
    ...                         ...
    sar:frequency_band:         C
    sar:instrument_mode:        IW
    sar:polarizations:          ['VV', 'VH']
    sar:product_type:           SLC
    xs:instrument_mode_swaths:  ['IW1', 'IW2', 'IW3']
    groups:                     ['IW1/gcp', 'IW1/attitude', 'IW1/orbit', 'IW2...]

Open with group = 'IW1':

manifest_path = (
    DATA_FOLDER
    / "S1B_IW_SLC__1SDV_20210401T052622_20210401T052650_026269_032297_EFA4.SAFE"
)
xr.open_dataset(manifest_path, engine="sentinel-1", group="IW1")  
<xarray.Dataset>
Dimensions:  ()
Data variables:
    *empty*
Attributes:
    groups:   ['orbit', 'attitude', 'gcp']

Open with group = 'IW1/orbit':

manifest_path = (
    DATA_FOLDER
    / "S1B_IW_SLC__1SDV_20210401T052622_20210401T052650_026269_032297_EFA4.SAFE"
)
xr.open_dataset(manifest_path, engine="sentinel-1", group="IW1/orbit")  # type: ignore

<xarray.Dataset>
Dimensions:  (time: 17)
Coordinates:
  * time     (time) datetime64[ns] 2021-04-01T05:25:19 ... 2021-04-01T05:27:59
Data variables:
    x        (time) float64 ...
    y        (time) float64 ...
    z        (time) float64 ...
    vx       (time) float64 ...
    vy       (time) float64 ...
    vz       (time) float64 ...
Attributes:
    Conventions:       CF-1.7
    reference_system:  Earth Fixed

TBC:

• groups to be displayed in case group=None:
  • [IW1, IW2...]
  • [IW1/orbit, IW1/attitude, IW1/gcp, IW2/orbit...]
  • [IW1, IW1/orbit, IW1/attitude, IW1/gcp, IW2, IW2/orbit...]
• attributes to expose in case group=None
• attributes to expose in case group='IW1'
• Support for zip? Nice to have but I would postpone the implementation.

Most metadata are distributed in XML files (e.g. manifest.safe, product/swath and calibration files) and the product zip file contains part of the XSD definition to use.

As the XSD schemas are required when using xmlschema we need a way to always have a set of files. Strategy is:

1. allow the user to specify a set of XSD files as a mapping from namespace to path to the schema file
2. look for the distributed schema files based on a predefined product folder structure e.g. top_dir/support/s1-level-1-product.xsd
3. use a mapping from namespace to shipped schema file

Until some releases ago we had full fsspec thanks to the following line working:

ds = xr.open_dataset(fs.open(urlpath), engine="rasterio")

This broke with some recent release of rioxarray or xarray, I think due to the fact that the open file is assumed to be a simple python object not a fsspec one.

At the moment a workaround is in place that attempt to use rasaterio remote URLs, but it is very fragile.

as type copies arrays when not specified otherwise see: https://numpy.org/doc/stable/reference/generated/numpy.ndarray.astype.html

• https://github.com/bopen/xarray-sentinel/blob/main/xarray_sentinel/sentinel1.py#L741
• https://github.com/bopen/xarray-sentinel/blob/main/xarray_sentinel/sentinel1.py#L746

Seems to be reason for tests failing in sarsen, will try working more on this and PR is "underway"

Useful enhancements:

• [ ] read the firstValidSample and lastValidSample tags and expose them (possibly making them more readable)
• [ ] provide and option to crop the burst to the border of the actual image