Digital Earth Australia notebooks and tools repository

As this repository continues to grow, a larger and larger amount of time is going to be spent re-running notebooks and verifying that all our functionality still works after every change to the DEA environments and our scripts/functions.

This seems like a natural fit for some kind of automated testing, e.g. to automatically re-run all notebooks after changes are pushed to develop to verify that updates haven't broken any notebooks before we ultimately update master.

~~Similarly, being able to automatically re-build a test version of the DEA User Guide from develop would also greatly assist in reducing formatting errors and re-writes required to generate the documentation.~~ (added as of #719 )

This issue is to capture discussion around the practicality of implementing automated testing on this repository. Any thoughts would be greatly appreciated!

This PR adds a dea_tools module to Tools/, which is installable with pip. It also updates every notebook to use that module instead of inserting Scripts/ into the path.

This is not meant to be a breaking change, so every notebook should still work. I ran every notebook that I could (a handful were too slow or too memory-hungry), so I have confirmed that they still work where possible.

The only breaking change that could happen is on the Sandbox, where user modules don't persist. The Sandbox will need to be updated to pip install this module from the most recent version (or cached version?) on load.

I don't want to merge this PR until we have resolved the Sandbox question (@Kirill888 may be the person to ask?), but I do want to kick-start the process of moving things over.

As discussed on Slack with @cbur24, once we have a functioning module for DEA, we can pretty easily build a DEAfrica component with a dea_tools dependency.

Here's what the old method of loading DEA scripts looked like:

import sys
sys.path.insert(1, '../Scripts')  # or ../../Scripts etc

from dea_plotting import rgb
from dea_spatialtools import xr_rasterize

New method that I introduce here:

from dea_tools.plotting import rgb
from dea_tools.spatial import xr_rasterize

Proposed changes

This PR adds a brand new Collection 3 Fractional Cover notebook, explaining how to inspect the available FC products and measurements, how to load data, and a simple best-practice example of analysing the data (including cloud masking from WO):

https://github.com/GeoscienceAustralia/dea-notebooks/blob/c3_dea_datasets/DEA_datasets/DEA_Fractional_Cover.ipynb

The format of the notebook is intended to match the other "DEA *" notebooks in DEA datasets. All content in the intro section is taken directly from CMI.

I'd love a quick review for typos/formatting/usability, but particularly that I've characterised the product and best approaches for loading/analysing it - if you have a chance before Friday that would be amazing!

Closes issues (optional)

• Closes Issue #667

Checklist (replace [ ] with [x] to check off)

• [X] Notebook created using the DEA-notebooks template
• [X] Remove any unused Python packages from Load packages
• [X] Remove any unused/empty code cells
• [X] Remove any guidance cells (e.g. General advice)
• [X] Ensure that all code cells follow the PEP8 standard for code. The jupyterlab_code_formatter tool can be used to format code cells to a consistent style: select each code cell, then click Edit and then one of the Apply X Formatter options (YAPF or Black are recommended).
• [X] Include relevant tags in the final notebook cell (refer to the DEA Tags Index, and re-use tags if possible)
• [X] Clear all outputs, run notebook from start to finish, and save the notebook in the state where all cells have been sequentially evaluated
• [ ] Test notebook on both the NCI and DEA Sandbox (flag if not working as part of PR and ask for help to solve if needed)
• [X] If applicable, update the Notebook currently compatible with the NCI|DEA Sandbox environment only line below the notebook title to reflect the environments the notebook is compatible with

Proposed changes

This notebook was originally written by @katedugdale. I have updated it for inclusion in the stable branch of dea-notebooks.

Checklist (replace [ ] with [x] to check off)

• [x] Notebook created using the DEA-notebooks template
• [x] Remove any unused Python packages from Load packages
• [x] Remove any unused/empty code cells
• [x] Remove any guidance cells (e.g. General advice)
• [x] Ensure that all code cells follow the PEP8 standard for code. The jupyterlab_code_formatter tool can be used to format code cells to a consistent style: select each code cell, then click Edit and then one of the Apply X Formatter options (YAPF or Black are recommended).
• [x] Include relevant tags in the final notebook cell (refer to the DEA Tags Index, and re-use tags if possible)
• [x] Clear all outputs, run notebook from start to finish, and save the notebook in the state where all cells have been sequentially evaluated
• [x] Test notebook on both the NCI and DEA Sandbox (flag if not working as part of PR and ask for help to solve if needed)
• [x] If applicable, update the Notebook currently compatible with the NCI|DEA Sandbox environment only line below the notebook title to reflect the environments the notebook is compatible with

Here's a notebook that generates polygons with Sentinel 2 MNDWI, Landsat 8 MNDWI, and WOfS, and compares their time series.

I haven't a) run it in sequential order, or b) run it on the NCI, because it takes ages to run and I want to make sure it's feature complete first.

Checklist (replace [ ] with [x] to check off)

• [x] Notebook created using the DEA-notebooks template
• [x] Remove any unused Python packages from Load packages
• [x] Remove any unused/empty code cells
• [x] Remove any guidance cells (e.g. General advice)
• [x] Ensure that all code cells follow the PEP8 standard for code. The jupyterlab_code_formatter tool can be used to format code cells to a consistent style: select each code cell, then click Edit and then one of the Apply X Formatter options (YAPF or Black are recommended).
• [x] Include relevant tags in the final notebook cell (refer to the DEA Tags Index, and re-use tags if possible)
• [x] Clear all outputs, run notebook from start to finish, and save the notebook in the state where all cells have been sequentially evaluated
• [ ] Test notebook on both the NCI and DEA Sandbox (flag if not working as part of PR and ask for help to solve if needed)

Background

#526 discusses issues with the current implementation of load_ard that can cause performance issues in combination with dask due to loading individual products then stacking them with xr.concatenate (potentially running into problems with mis-aligned chunks and peak memory issues).

Proposed changes

This PR aims to re-factor load_ard using the following workflow:

1. Uses dc.find_datasets() across all products first, building up a simple list of datasets named dataset_list
2. Uses dc.load(datasets=dataset_list, ...) to load datasets across all products in one go
3. Masks data without converting to floats until the final step using tools from odc.algo, avoiding issues with high peak memory usage

This change has greatly simplified the function code, and has also enabled some potentially exciting new features, including the new filter_func parameter. This lets you pass in a custom function which filters out individual datasets prior to them even being loaded using dc.load. For example, to load only summer months from multiple years:

query = {
    'x': (153.38, 153.47),
    'y': (-28.83, -28.92),
    'time': ('2015', '2018'),
    'measurements': ['nbart_red', 'nbart_green', 'nbart_blue'],
    'output_crs': 'EPSG:3577',
    'resolution': (-30, 30),
    'group_by': 'solar_day',
    'dask_chunks': {}
}

def month_filter(dataset, months=[12, 1, 2]):    
    # Return True if dataset month is in list of months
    return dataset.time.begin.month in months

ds = load_ard(dc=dc,
              products=['ga_ls5t_ard_3', 'ga_ls7e_ard_3', 'ga_ls8c_ard_3'],
              filter_func=month_filter,
              **query)

Closes issues (optional)

• Closes Issue #526

Proposed changes

This PR adds a much over-due DEA Datasets notebook that demonstrates how to load data from DEA Landsat Collection 3 Surface Reflectance products using the Digital Earth Australia datacube. Topics covered include:

• Inspecting the Landsat products and measurements available in the datacube
• Loading Landsat data for specific location and time
• Plotting Landsat data in true and false colour
• Applying a cloud mask using the oa_fmask band
• Advanced: Loading data in its native projection and resolution

Most of the markdown is based directly on CMI, and I've tried to direct users back towards CMI for more info at every opportunity.

Closes issues (optional)

• Closes Issue #665

Checklist (replace [ ] with [x] to check off)

• [X] Notebook created using the DEA-notebooks template
• [X] Remove any unused Python packages from Load packages
• [X] Remove any unused/empty code cells
• [X] Remove any guidance cells (e.g. General advice)
• [X] Ensure that all code cells follow the PEP8 standard for code. The jupyterlab_code_formatter tool can be used to format code cells to a consistent style: select each code cell, then click Edit and then one of the Apply X Formatter options (YAPF or Black are recommended).
• [X] Include relevant tags in the final notebook cell (refer to the DEA Tags Index, and re-use tags if possible)
• [X] Clear all outputs, run notebook from start to finish, and save the notebook in the state where all cells have been sequentially evaluated
• [x] Test notebook on both the NCI and DEA Sandbox (flag if not working as part of PR and ask for help to solve if needed)
• [x] If applicable, update the Notebook currently compatible with the NCI|DEA Sandbox environment only line below the notebook title to reflect the environments the notebook is compatible with

This pull request will add notebooks developed for the DEA Sandbox to the general notebook repository for DEA.

Still to do:

• test notebooks on GA-hosted sandbox to validate indexing
• modify notebook content/formating to fit with needs of GA-hosted sandbox (e.g. dask_average_pixel_example.ipynb needs explanatory text).

Happy to take any other feedback and update as necessary! I'll leave this as a draft PR until we're ready.

Supplementary data for ARD_overpass_predictor notebook in dea-notebooks/Frequently_used_code/

Proposed changes

Supplementary data for ARD_overpass_predictor notebook in dea-notebooks/Frequently_used_code/

Checklist (replace [ ] with [x] to check off)

• [x] Notebook created using the DEA-notebooks template
• [x] Remove any unused Python packages from Load packages
• [x] Remove any unused/empty code cells
• [x] Remove any guidance cells (e.g. General advice)
• [x] Ensure that all code cells follow the PEP8 standard for code. The jupyterlab_code_formatter tool can be used to format code cells to a consistent style: select each code cell, then click Edit and then one of the Apply X Formatter options (YAPF or Black are recommended).
• [x] Include relevant tags in the final notebook cell (refer to the DEA Tags Index, and re-use tags if possible)
• [x] Clear all outputs, run notebook from start to finish, and save the notebook in the state where all cells have been sequentially evaluated
• [x] Test notebook on both the NCI and DEA Sandbox (flag if not working as part of PR and ask for help to solve if needed)
• [x] If applicable, update the Notebook currently compatible with the NCI|DEA Sandbox environment only line below the notebook title to reflect the environments the notebook is compatible with

Proposed changes

Added a notebook that estimates WOfS errors and probabilities as part of the waterbodies toolkit - thinking of using this for waterbody edges later on.

Checklist (replace [ ] with [x] to check off)

• [x] Notebook created using the DEA-notebooks template
• [x] Remove any unused Python packages from Load packages
• [x] Remove any unused/empty code cells
• [x] Remove any guidance cells (e.g. General advice)
• [x] Ensure that all code cells follow the PEP8 standard for code. The jupyterlab_code_formatter tool can be used to format code cells to a consistent style: select each code cell, then click Edit and then one of the Apply X Formatter options (YAPF or Black are recommended).
• [x] Include relevant tags in the final notebook cell (refer to the DEA Tags Index, and re-use tags if possible)
• [x] Clear all outputs, run notebook from start to finish, and save the notebook in the state where all cells have been sequentially evaluated
• [ ] Test notebook on both the NCI and DEA Sandbox (flag if not working as part of PR and ask for help to solve if needed)
• [ ] If applicable, update the Notebook currently compatible with the NCI|DEA Sandbox environment only line below the notebook title to reflect the environments the notebook is compatible with

Proposed changes

Started redoing the xr_geomedian notebook and then decided to port over from DE Africa all things image compositing. Two new notebooks: Generating geomedian composites and Generating composites; five new functions for dea_datahandling: calc_geomedian, first, last, nearest, _select_along_axis

I haven't run these on the NCI.

Checklist (replace [ ] with [x] to check off)

• [X] Notebook created using the DEA-notebooks template
• [X] Remove any unused Python packages from Load packages
• [X] Remove any unused/empty code cells
• [X] Remove any guidance cells (e.g. General advice)
• [X] Ensure that all code cells follow the PEP8 standard for code. The jupyterlab_code_formatter tool can be used to format code cells to a consistent style: select each code cell, then click Edit and then one of the Apply X Formatter options (YAPF or Black are recommended).
• [X] Include relevant tags in the final notebook cell (refer to the DEA Tags Index, and re-use tags if possible)
• [X] Clear all outputs, run notebook from start to finish, and save the notebook in the state where all cells have been sequentially evaluated
• [ ] Test notebook on both the NCI and DEA Sandbox (flag if not working as part of PR and ask for help to solve if needed)
• [ ] If applicable, update the Notebook currently compatible with the NCI|DEA Sandbox environment only line below the notebook title to reflect the environments the notebook is compatible with

https://github.com/GeoscienceAustralia/dea-notebooks/blob/develop/.github/workflows/test_notebooks.yml#L31

(base) ubuntu@:~/dea-sandbox$ pip --use-feature=in-tree-build

Usage:   
  pip <command> [options]

option --use-feature: invalid choice: 'in-tree-build' (choose from 'fast-deps', 'truststore', 'no-binary-enable-wheel-cache')

Proposed changes

Work in progress...

Closes issues (optional)

• Closes Issue #000
• Closes Issue #000

Checklist (replace [ ] with [x] to check off)

• [x] Notebook created using the DEA-notebooks template
• [x] Remove any unused Python packages from Load packages
• [x] Remove any unused/empty code cells
• [x] Remove any guidance cells (e.g. General advice)
• [x] Ensure that all code cells follow the PEP8 standard for code. The jupyterlab_code_formatter tool can be used to format code cells to a consistent style: select each code cell, then click Edit and then one of the Apply X Formatter options (YAPF or Black are recommended).
• [x] Include relevant tags in the final notebook cell (refer to the DEA Tags Index, and re-use tags if possible)
• [x] Clear all outputs, run notebook from start to finish, and save the notebook in the state where all cells have been sequentially evaluated
• [ ] Test notebook on both the NCI and DEA Sandbox (flag if not working as part of PR and ask for help to solve if needed)
• [ ] If applicable, update the Notebook currently compatible with the NCI|DEA Sandbox environment only line below the notebook title to reflect the environments the notebook is compatible with

Add a new Sentinel-2 notebook which contains new up-to-date information, including the background from the Sentinel-2 Collection 3 Content Management Interface (CMI) pages.

To flag:

• Are we removing the old Sentinel-2 notebook (DEA_datasets/Sentinel_2.ipynb)? I have not included the removal of the old notebook as part of this PR.
• Sentinel-2B CMI links are not publicly accessible yet. Aaron followed up this morning as they are currently pending approval.
• @robbibt will add s2cloudless as an option for Sentinel-2 in the load_ard() function.

This PR adds load hints to our products that will allow users to load them into a default CRS and resolution without having to supply them manually using output_crs and resolution: https://github.com/GeoscienceAustralia/dea-config/pull/1117

We should document this in DEA Notebooks/user guide so that this change does not catch users unawares.

The DEA datasets folder of DEA Notebooks includes standardised notebooks that introduce each of our products and how to get started with them using the Datacube: https://docs.dea.ga.gov.au/notebooks/DEA_datasets/README.html

These notebooks include a set of standard sections to make it easy for users to follow along and compare between products:

1. Background: Taken directly from CMI 2. Description: A brief summary of what is included in the notebook 3. Getting started: Some setup steps like loading Python packages and connecting to the datacube 4. Available products and measurements: Inspecting the data using dc.list_products() and dc.list_mesurements() 5. Loading data: Simplest/minimal example of loading the data and plotting it 6. Example application: A small "demo" using the product, e.g. designed to show how it could be loaded and manipulated in a realistic workflow (e.g. loading the data, applying a cloud mask etc), and highlighting any common "gotchas" or things users should be aware of when using it

We should create a similar notebook for Sentinel-2 Collection 3, showing users what products and bands are available, how to load and plot the data, and perhaps how to use some of its special features (e.g. s2cloudless, filtering by GQA metadata, maturity etc).

You can see some nice examples of DEA dataset notebooks here; these could be a nice template for this work:

• https://docs.dea.ga.gov.au/notebooks/DEA_datasets/DEA_Water_Observations.html
• https://docs.dea.ga.gov.au/notebooks/DEA_datasets/DEA_Fractional_Cover.html

There is an ancient Sentinel-2 notebook here that may be useful for this, but it doesn't match the formatting of the other DEA dataset notebooks... we might be better off starting from scratch rather than adapting this

We should update this repository to include at least some nice examples of applying cloud masking using the new s2cloudless_mask band. This will provide users with code to get started with this new cloud mask.

An optional extra could be a simple notebook in Frequently_used_code that shows how to load and manipulate the quantitative s2cloudless_prob cloud probability layer (and any cool application of using it that we can think up)