• [x] Closes #1223
• [x] Tests added / passed
• [x] Passes git diff upstream/master | flake8 --diff
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

I think that a zarr backend could be the ideal storage format for xarray datasets, overcoming many of the frustrations associated with netcdf and enabling optimal performance on cloud platforms.

This is a very basic start to implementing a zarr backend (as proposed in #1223); however, I am taking a somewhat different approach. I store the whole dataset in a single zarr group. I encode the extra metadata needed by xarray (so far just dimension information) as attributes within the zarr group and child arrays. I hide these special attributes from the user by wrapping the attribute dictionaries in a "HiddenKeyDict", so that they can't be viewed or modified.

I have no tests yet (:flushed:), but the following code works.

from xarray.backends.zarr import ZarrStore
import xarray as xr
import numpy as np

ds = xr.Dataset(
    {'foo': (('y', 'x'), np.ones((100, 200)), {'myattr1': 1, 'myattr2': 2}),
     'bar': (('x',), np.zeros(200))},
    {'y': (('y',), np.arange(100)),
     'x': (('x',), np.arange(200))},
    {'some_attr': 'copana'}
).chunk({'y': 50, 'x': 40})

zs = ZarrStore(store='zarr_test')
ds.dump_to_store(zs)
ds2 = xr.Dataset.load_store(zs)
assert ds2.equals(ds)

There is a very long way to go here, but I thought I would just get a PR started. Some questions that would help me move forward.

1. What is "encoding" at the variable level? (I have never understood this part of xarray.) How should encoding be handled with zarr?
2. Should we encode / decode CF for zarr stores?
3. Do we want to always automatically align dask chunks with the underlying zarr chunks?
4. What sort of public API should the zarr backend have? Should you be able to load zarr stores via open_dataset? Or do we need a new method? I think .to_zarr() would be quite useful.
5. zarr arrays are extensible along all axes. What does this imply for unlimited dimensions?
6. Is any autoclose logic needed? As far as I can tell, zarr objects don't need to be closed.

• [x] closes #1084
• [x] passes git diff upstream/master | flake8 --diff
• [x] tests added / passed
• [x] whatsnew entry

This work in progress PR is a start on implementing a NetCDFTimeIndex, a subclass of pandas.Index, which closely mimics pandas.DatetimeIndex, but uses netcdftime._netcdftime.datetime objects. Currently implemented in the new index are:

• Partial datetime-string indexing (using strictly ISO8601-format strings, using a date parser implemented by @shoyer in https://github.com/pydata/xarray/issues/1084#issuecomment-274372547)
• Field-accessors for year, month, day, hour, minute, second, and microsecond, to enable groupby operations on attributes of date objects

This index is meant as a step towards improving the handling of non-standard calendars and dates outside the range Timestamp('1677-09-21 00:12:43.145225') to Timestamp('2262-04-11 23:47:16.854775807').

For now I have pushed only the code and some tests for the new index; I want to make sure the index is solid and well-tested before we consider integrating it into any of xarray's existing logic or writing any documentation.

Regarding the index, there are a couple remaining outstanding issues (that at least I'm aware of):

1. Currently one can create non-sensical datetimes using netcdftime._netcdftime.datetime objects. This means one can attempt to index with an out-of-bounds string or datetime without raising an error. Could this possibly be addressed upstream? For example:

In [1]: from netcdftime import DatetimeNoLeap

In [2]: DatetimeNoLeap(2000, 45, 45)
Out[2]: netcdftime._netcdftime.DatetimeNoLeap(2000, 45, 45, 0, 0, 0, 0, -1, 1)

2. I am looking to enable this index to be used in pandas.Series and pandas.DataFrame objects as well; this requires implementing a get_value method. I have taken @shoyer's suggested simplified approach from https://github.com/pydata/xarray/issues/1084#issuecomment-275963433, and tweaked it to also allow for slice indexing, so I think this is most of the way there. A remaining to-do for me, however, is to implement something to allow for integer-indexing outside of iloc, e.g. if you have a pandas.Series series, indexing with the syntax series[1] or series[1:3].

Hopefully this is a decent start; in particular I'm not an expert in writing tests so please let me know if there are improvements I can make to the structure and / or style I've used so far. I'm happy to make changes. I appreciate your help.

I think we can continue the discussion we have in #1426 about MultiIndex here.

In comment , @shoyer recommended to remove MultiIndex from public API.

I agree with this, as long as my codes work with this improvement.

I think if we could have a list of possible MultiIndex use cases here, it would be easier to deeply discuss and arrive at a consensus of the future API.

Current limitations of MultiIndex are

• It drops scalar coordinate after selection #1408, #1491
• It does not support to serialize to NetCDF #1077
• Stack/unstack behaviors are inconsistent #1431

use dask.array.apply_gufunc in xr.apply_ufunc for multiple outputs when dask='parallelized', add/fix tests

• [x] Closes #1815, closes #4015
• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Remaining Issues:

• [ ] fitting name for current dask_gufunc_kwargs
• [ ] rephrase dask docs to fit new behaviour
• [ ] combine output_core_dims and output_sizes, eg. xr.apply_ufunc(..., output_core_dims=[{"abc": 2]])

Many datasets have a two dimensional coordinate variable (e.g. longitude) which is different from the logical grid coordinates (e.g. nx, ny). (See #605.) For plotting purposes, this is solved by #608. However, we still might want to split / apply / combine over such coordinates. That has not been possible, because groupby only supports creating groups on one-dimensional arrays.

This PR overcomes that issue by using stack to collapse multiple dimensions in the group variable. A minimal example of the new functionality is

>>> da = xr.DataArray([[0,1],[2,3]], 
                coords={'lon': (['ny','nx'], [[30,40],[40,50]] ),
                        'lat': (['ny','nx'], [[10,10],[20,20]] )},
                dims=['ny','nx'])
>>> da.groupby('lon').sum()
<xarray.DataArray (lon: 3)>
array([0, 3, 3])
Coordinates:
  * lon      (lon) int64 30 40 50

This feature could have broad applicability for many realistic datasets (particularly model output on irregular grids): for example, averaging non-rectangular grids zonally (i.e. in latitude), binning in temperature, etc.

If you think this is worth pursuing, I would love some feedback.

The PR is not complete. Some items to address are

• [x] Create a specialized grouper to allow coarser bins. By default, if no grouper is specified, the GroupBy object uses all unique values to define the groups. With a high resolution dataset, this could balloon to a huge number of groups. With the latitude example, we would like to be able to specify e.g. 1-degree bins. Usage would be da.groupby('lon', bins=range(-90,90)).
• [ ] Allow specification of which dims to stack. For example, stack in space but keep time dimension intact. (Currently it just stacks all the dimensions of the group variable.)
• [x] A nice example for the docs.

As discussed in the meeting, we should issue a release soon with the new backend refactor and the new docs theme.

Here's a list of blockers:

• [x] #5231
• [x] #5073
• [x] #5235

Would be nice and look done:

• [x] #5244
• [x] #5258
• [x] #5101
• [x] ~#4866~ (we should let this sit on master for a while to find bugs)
• [x] #4902
• [x] ~#4972~ (this should probably also sit on master for a while)
• [x] #5227
• [x] #4740
• [x] #5149

Somewhat important, but no PR yet:

• [x] ~#5175~ (as pointed out by @shoyer, this is really a new feature, not a regression, it can wait)

@TomNicholas and @alexamici volunteered to handle this. I can be online at release time to help with things if needed.

Release instructions are here: https://github.com/pydata/xarray/blob/master/HOW_TO_RELEASE.md

IIRC they'll need to be added to the PyPI list and RTD list.

• [x] Closes #1444, closes #1436
• [x] Tests added / passed
• [x] Passes git diff master | flake8 --diff
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

xref https://github.com/pydata/xarray/issues/974#issuecomment-313977794

This pull request allows to append an xarray to an existing datastore.

• [x] Closes #2022
• [x] Tests will be added. Wanted to get an opinion if this is what is imagined by the community
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API To filter the data written to the array, the dimension over which the data will be appended has to be explicitly stated. If someone has an idea how to overcome this, I would be more than happy to incorporate the necessary changes into the PR. Cheers, Jendrik

Dask (https://github.com/dask/dask) currently provides on-node parallelism for medium-size data problems. However, large climate data sets will require multiple-node parallelism to analyze large climate data sets because this constitutes a big data problem. A likely solution to this issue is integration of distributed (https://github.com/dask/distributed) with dask. Distributed is now integrated with dask and its benefits are already starting to be realized, e.g., see http://matthewrocklin.com/blog/work/2016/02/26/dask-distributed-part-3.

Thus, this issue is designed to identify the steps needed to perform this integration, at a high-level. As stated by @shoyer, it will

definitely require some refactoring of the xarray backend system to make this work cleanly, but that's OK -- the xarray backend system is indicated as experimental/internal API precisely because we hadn't figured out all the use cases yet."

To be honest, I've never been entirely happy with the design we took there (we use inheritance rather than composition for backend classes), but we did get it to work for our use cases. Some refactoring with an eye towards compatibility with dask distributed seems like a very worthwhile endeavor. We do have the benefit of a pretty large test suite covering existing use cases.

Thus, we have the chance to make xarray big-data capable as well as provide improvements to the backend.

To this end, I'm starting this issue to help begin the design process following the xarray mailing list discussion some of us have been having (@shoyer, @mrocklin, @rabernat).

Task To Do List:

• [x] Verify asynchronous access error for to_netcdf output is resolved (e.g., https://github.com/pydata/xarray/issues/793)
• [x] LRU-cached file IO supporting serialization to robustly support HDF/NetCDF reads

This PR supersedes #1820 - see that PR for original discussion. See this gist to try out the new MultiIndex and options functionality.

• [x] Closes #1627, closes #1820
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

TODO:

• [x] Add support for Multi-indexes
• [x] Probably good to have some opt-in or fail back system in case where we (or users) know that the rendering will not work
• [x] Add some tests

Previously, DataStore did not judiciously close files, resulting in opening a large number of files that could result in an OSError related to too many files being open. This merge provides a solution for the netCDF, scipy, and h5netcdf backends.

This fixes GH6220 which makes it possible to use the fastpath for .groupby for monotonically increasing and decreasing values.

• [x] Closes #6220
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [ ] Closes #7243
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

What happened?

After upgrading numpy from 1.23.5 to 1.24.1, calling the unstack method on an xarray.DataArray with integer data produces the warning <__array_function__ internals>:200: RuntimeWarning: invalid value encountered in cast. I think this relates to "ongoing work to improve the handling and promotion of dtypes" (Numpy 1.24.0 Release Notes), and is catching the fact that the method attempts to provide nan as a fill value on an integer array.

What did you expect to happen?

In the case below, where there is no need for a fill value, I do not expect to get a warning.

Minimal Complete Verifiable Example

import xarray as xr
import numpy as np
# np.seterr(all='raise') # uncomment to convert warning to error

da = xr.DataArray(
    data=np.array([[0]], dtype=int),
    coords={'x': [0], 'y': [1]},
    )
da = da.stack({'z': ['x', 'y']})
da.unstack()

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

<__array_function__ internals>:200: RuntimeWarning: invalid value encountered in cast
<xarray.DataArray (x: 1, y: 1)>
array([[0]])
Coordinates:
  * x        (x) int64 0
  * y        (y) int64 1

Anything else we need to know?

No response

Environment

Workflow Run URL

xarray/tests/test_backends.py::TestNetCDF4Data::test_roundtrip_cftime_datetime_data: AssertionError: assert 'days since 1-01-01' == 'days since 0001-01-01'
  - days since 0001-01-01
  ?            ---
  + days since 1-01-01
xarray/tests/test_backends.py::TestNetCDF4ViaDaskData::test_roundtrip_cftime_datetime_data: AssertionError: assert 'days since 1-01-01' == 'days since 0001-01-01'
  - days since 0001-01-01
  ?            ---
  + days since 1-01-01
xarray/tests/test_backends.py::TestZarrDictStore::test_roundtrip_cftime_datetime_data: AssertionError: assert 'days since 1-01-01' == 'days since 0001-01-01'
  - days since 0001-01-01
  ?            ---
  + days since 1-01-01
xarray/tests/test_backends.py::TestZarrDirectoryStore::test_roundtrip_cftime_datetime_data: AssertionError: assert 'days since 1-01-01' == 'days since 0001-01-01'
  - days since 0001-01-01
  ?            ---
  + days since 1-01-01
xarray/tests/test_backends.py::TestZarrKVStoreV3::test_roundtrip_cftime_datetime_data: AssertionError: assert 'days since 1-01-01' == 'days since 0001-01-01'
  - days since 0001-01-01
  ?            ---
  + days since 1-01-01
xarray/tests/test_backends.py::TestZarrDirectoryStoreV3::test_roundtrip_cftime_datetime_data: AssertionError: assert 'days since 1-01-01' == 'days since 0001-01-01'
  - days since 0001-01-01
  ?            ---
  + days since 1-01-01
xarray/tests/test_backends.py::TestZarrDirectoryStoreV3::test_write_read_select_write: KeyError: 'var1'
xarray/tests/test_backends.py::TestZarrDirectoryStoreV3FromPath::test_roundtrip_cftime_datetime_data: AssertionError: assert 'days since 1-01-01' == 'days since 0001-01-01'
  - days since 0001-01-01
  ?            ---
  + days since 1-01-01
xarray/tests/test_backends.py::TestZarrDirectoryStoreV3FromPath::test_write_read_select_write: KeyError: 'var1'
xarray/tests/test_backends.py::TestScipyInMemoryData::test_roundtrip_cftime_datetime_data: AssertionError: assert 'days since 1-01-01' == 'days since 0001-01-01'
  - days since 0001-01-01
  ?            ---
  + days since 1-01-01
xarray/tests/test_backends.py::TestScipyFileObject::test_roundtrip_cftime_datetime_data: AssertionError: assert 'days since 1-01-01' == 'days since 0001-01-01'
  - days since 0001-01-01
  ?            ---
  + days since 1-01-01
xarray/tests/test_backends.py::TestScipyFilePath::test_roundtrip_cftime_datetime_data: AssertionError: assert 'days since 1-01-01' == 'days since 0001-01-01'
  - days since 0001-01-01
  ?            ---
  + days since 1-01-01
xarray/tests/test_backends.py::TestNetCDF3ViaNetCDF4Data::test_roundtrip_cftime_datetime_data: AssertionError: assert 'days since 1-01-01' == 'days since 0001-01-01'
  - days since 0001-01-01
  ?            ---
  + days since 1-01-01
xarray/tests/test_backends.py::TestNetCDF4ClassicViaNetCDF4Data::test_roundtrip_cftime_datetime_data: AssertionError: assert 'days since 1-01-01' == 'days since 0001-01-01'
  - days since 0001-01-01
  ?            ---
  + days since 1-01-01
xarray/tests/test_backends.py::TestGenericNetCDFData::test_roundtrip_cftime_datetime_data: AssertionError: assert 'days since 1-01-01' == 'days since 0001-01-01'
  - days since 0001-01-01
  ?            ---
  + days since 1-01-01
xarray/tests/test_backends.py::TestH5NetCDFData::test_roundtrip_cftime_datetime_data: AssertionError: assert 'days since 1-01-01' == 'days since 0001-01-01'
  - days since 0001-01-01
  ?            ---
  + days since 1-01-01
xarray/tests/test_backends.py::TestH5NetCDFFileObject::test_roundtrip_cftime_datetime_data: AssertionError: assert 'days since 1-01-01' == 'days since 0001-01-01'
  - days since 0001-01-01
  ?            ---
  + days since 1-01-01
xarray/tests/test_backends.py::TestH5NetCDFViaDaskData::test_roundtrip_cftime_datetime_data: AssertionError: assert 'days since 1-01-01' == 'days since 0001-01-01'
  - days since 0001-01-01
  ?            ---
  + days since 1-01-01
xarray/tests/test_calendar_ops.py::test_convert_calendar[2 failing variants]: TypeError: DatetimeArray._generate_range() got an unexpected keyword argument 'closed'
xarray/tests/test_calendar_ops.py::test_convert_calendar_360_days[4 failing variants]: TypeError: DatetimeArray._generate_range() got an unexpected keyword argument 'closed'
xarray/tests/test_calendar_ops.py::test_convert_calendar_360_days[2 failing variants]: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_calendar_ops.py::test_convert_calendar_missing[2 failing variants]: TypeError: DatetimeArray._generate_range() got an unexpected keyword argument 'closed'
xarray/tests/test_calendar_ops.py::test_convert_calendar_same_calendar: TypeError: DatetimeArray._generate_range() got an unexpected keyword argument 'closed'
xarray/tests/test_calendar_ops.py::test_interp_calendar[4 failing variants]: TypeError: DatetimeArray._generate_range() got an unexpected keyword argument 'closed'
xarray/tests/test_calendar_ops.py::test_interp_calendar_errors: TypeError: DatetimeArray._generate_range() got an unexpected keyword argument 'closed'
xarray/tests/test_cftime_offsets.py::test_date_range[4 failing variants]: TypeError: DatetimeArray._generate_range() got an unexpected keyword argument 'closed'
xarray/tests/test_cftime_offsets.py::test_date_range_errors: TypeError: DatetimeArray._generate_range() got an unexpected keyword argument 'closed'
xarray/tests/test_cftime_offsets.py::test_date_range_like[5 failing variants]: TypeError: DatetimeArray._generate_range() got an unexpected keyword argument 'closed'
xarray/tests/test_cftime_offsets.py::test_date_range_like_same_calendar: TypeError: DatetimeArray._generate_range() got an unexpected keyword argument 'closed'
xarray/tests/test_cftime_offsets.py::test_date_range_like_errors: TypeError: DatetimeArray._generate_range() got an unexpected keyword argument 'closed'
xarray/tests/test_cftimeindex.py::test_to_datetimeindex_out_of_range[9 failing variants]: Failed: DID NOT RAISE <class 'ValueError'>
xarray/tests/test_cftimeindex_resample.py::test_resample[729 failing variants]: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_cftimeindex_resample.py::test_calendars[5 failing variants]: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_cftimeindex_resample.py::test_origin[12 failing variants]: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_coding_times.py::test_should_cftime_be_used_source_outside_range: Failed: DID NOT RAISE <class 'ValueError'>
xarray/tests/test_computation.py::test_polyval_cftime[4 failing variants]: TypeError: DatetimeArray._generate_range() got an unexpected keyword argument 'closed'
xarray/tests/test_conventions.py::TestCFEncodedDataStore::test_roundtrip_cftime_datetime_data: AssertionError: assert 'days since 1-01-01' == 'days since 0001-01-01'
  - days since 0001-01-01
  ?            ---
  + days since 1-01-01
xarray/tests/test_dataarray.py::TestDataArray::test_sel_float: NotImplementedError: float16 indexes are not supported
xarray/tests/test_groupby.py::TestDataArrayResample::test_resample: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDataArrayResample::test_da_resample_func_args: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDataArrayResample::test_resample_first: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDataArrayResample::test_resample_bad_resample_dim: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDataArrayResample::test_resample_drop_nondim_coords: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDataArrayResample::test_resample_keep_attrs: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDataArrayResample::test_resample_skipna: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDataArrayResample::test_upsample: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDataArrayResample::test_upsample_nd: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDataArrayResample::test_upsample_tolerance: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDataArrayResample::test_upsample_interpolate: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDataArrayResample::test_upsample_interpolate_bug_2197: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDataArrayResample::test_upsample_interpolate_regression_1605: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDataArrayResample::test_upsample_interpolate_dask[2 failing variants]: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDataArrayResample::test_resample_base: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDataArrayResample::test_resample_offset: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDataArrayResample::test_resample_origin: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDatasetResample::test_resample_and_first: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDatasetResample::test_resample_min_count: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDatasetResample::test_resample_by_mean_with_keep_attrs: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDatasetResample::test_resample_loffset: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDatasetResample::test_resample_by_mean_discarding_attrs: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDatasetResample::test_resample_by_last_discarding_attrs: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDatasetResample::test_resample_drop_nondim_coords: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDatasetResample::test_resample_ds_da_are_the_same: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::TestDatasetResample::test_ds_resample_apply_func_args: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_groupby.py::test_resample_cumsum[2 failing variants]: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_units.py::TestDataArray::test_resample[2 failing variants]: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_units.py::TestDataset::test_resample[4 failing variants]: TypeError: Grouper.__init__() got an unexpected keyword argument 'base'
xarray/tests/test_variable.py::TestVariable::test_datetime64_conversion_scalar: AssertionError: assert numpy.datetime64('1970-01-01T00:00:00.946684800') == numpy.datetime64('2000-01-01T00:00:00.000000000')
 +  where numpy.datetime64('1970-01-01T00:00:00.946684800') = <xarray.Variable ()>\narray('1970-01-01T00:00:00.946684800', dtype='datetime64[ns]').values
xarray/tests/test_variable.py::TestVariable::test_0d_datetime: AssertionError: assert numpy.datetime64('1970-01-01T00:00:00.946684800') == numpy.datetime64('2000-01-01T00:00:00.000000000')
 +  where numpy.datetime64('1970-01-01T00:00:00.946684800') = <xarray.Variable ()>\narray('1970-01-01T00:00:00.946684800', dtype='datetime64[ns]').values
 +  and   numpy.datetime64('2000-01-01T00:00:00.000000000') = <class 'numpy.datetime64'>('2000-01-01', 'ns')
 +    where <class 'numpy.datetime64'> = np.datetime64
xarray/tests/test_variable.py::TestVariable::test_reduce_funcs: AssertionError: Left and right Variable objects are not identical

Differing values:
L
    array('2000-01-03T00:00:00.000000000', dtype='datetime64[ns]')
R
    array('1970-01-01T00:00:00.946857600', dtype='datetime64[ns]')

I want datatree to live in xarray main, as right now it's in a separate package but imports many xarray internals.

This presents a few questions:

1. At what stage is datatree "ready" to moved in here? At what stage should it become encouraged public API?
2. What's a good way to slowly roll the feature out?
3. How do I decrease the bus factor on datatree's code? Can I get some code reviews during the merging process? :pray:
4. Should I make a new CI environment just for testing datatree stuff?

Today @jhamman and @keewis suggested for now I make it so that you can from xarray import DataTree, using the current xarray-datatree package as an optional dependency. That way I can create a smoother on-ramp, get some more users testing it, but without committing all the code into this repo yet.

@pydata/xarray what do you think? Any other thoughts about best practices when moving a good few thousand lines of code into xarray?

• [x] First step towards moving solution of #4118 into this repository
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst