Hi there!

We have started using Terracotta in our K8S infrastructure on production. Basically we are serving the WSGI flask application (terracotta.server.app:app) using gunicorn alongside with an internal gRPC server which is taking internal requests and queries the terracotta HTTP endpoint for a singleband tile and returns it as a bytes object.

However, while the first 10-50 requests work fine, I now get this error from terracotta afterwards:

 [-] Exception on /singleband/some_path/25/10/506/313.png [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.8/dist-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.8/dist-packages/terracotta/server/flask_api.py", line 49, in inner
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/terracotta/server/singleband.py", line 121, in get_singleband
    return _get_singleband_image(keys, tile_xyz)
  File "/usr/local/lib/python3.8/dist-packages/terracotta/server/singleband.py", line 166, in _get_singleband_image
    image = singleband(parsed_keys, tile_xyz=tile_xyz, **options)
  File "/usr/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/usr/local/lib/python3.8/dist-packages/terracotta/handlers/singleband.py", line 43, in singleband
    tile_data = xyz.get_tile_data(
  File "/usr/local/lib/python3.8/dist-packages/terracotta/xyz.py", line 44, in get_tile_data
    return driver.get_raster_tile(
  File "/usr/local/lib/python3.8/dist-packages/terracotta/drivers/base.py", line 20, in inner
    return fun(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/terracotta/drivers/raster_base.py", line 557, in get_raster_tile
    future = executor.submit(retrieve_tile)
  File "/usr/lib/python3.8/concurrent/futures/process.py", line 629, in submit
    raise BrokenProcessPool(self._broken)
concurrent.futures.process.BrokenProcessPool: A child process terminated abruptly, the process pool is not usable anymore

The worst thing about this is that the flask application doesn't seem to actually error out. Instead, every subsequent request throws the error above. That's problematic as K8s then doesn't know that the pod needs to be restarted. However, on a longer sight, this also means that we could never cater for the amount of requests (around 50 RPS) we have using terracotta if this persists.

Has anyone encountered this yet?

When deploying a Lambda function through Zappa, using the newest version of Rasterio (1.0.22), we get a 500 for some tiles, while others load fine.

When reverting back to Rasterio 1.0.13 everything works fine.

zappa tail gives the following:

[1554888957364] [ERROR] 2019-04-10T09:35:57.326Z b24c3c65-85ed-4390-a18b-c0cf347feec0 Exception on /rgb/italy/2018/7/70/47.png [GET]
Traceback (most recent call last):
  File "rasterio/_base.pyx", line 213, in rasterio._base.DatasetBase.__init__
  File "rasterio/_shim.pyx", line 64, in rasterio._shim.open_dataset
  File "rasterio/_err.pyx", line 205, in rasterio._err.exc_wrap_pointer
rasterio._err.CPLE_OpenFailedError: '/vsis3/italy-composite/rasters/italy_2018_red.tif' not recognized as a supported file format.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/task/terracotta/drivers/raster_base.py", line 464, in _get_raster_tile
  src = es.enter_context(rasterio.open(path))
  File "/var/task/rasterio/env.py", line 423, in wrapper
  return f(*args, **kwds)
  File "/var/task/rasterio/__init__.py", line 216, in open
  s = DatasetReader(path, driver=driver, **kwargs)
  File "rasterio/_base.pyx", line 215, in rasterio._base.DatasetBase.__init__
rasterio.errors.RasterioIOError: '/vsis3/italy-composite/rasters/italy_2018_red.tif' not recognized as a supported file format.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/task/flask/app.py", line 2292, in wsgi_app
  response = self.full_dispatch_request()
  File "/var/task/flask/app.py", line 1815, in full_dispatch_request
  rv = self.handle_user_exception(e)
  File "/var/task/flask/app.py", line 1718, in handle_user_exception
  reraise(exc_type, exc_value, tb)
  File "/var/task/flask/_compat.py", line 35, in reraise
  raise value
  File "/var/task/flask/app.py", line 1813, in full_dispatch_request
  rv = self.dispatch_request()
  File "/var/task/flask/app.py", line 1799, in dispatch_request
  return self.view_functions[rule.endpoint](**req.view_args)
  File "/var/task/terracotta/server/flask_api.py", line 50, in inner
  return fun(*args, **kwargs)
  File "/var/task/terracotta/server/rgb.py", line 85, in get_rgb
  return _get_rgb_image(keys, tile_xyz=tile_xyz)
  File "/var/task/terracotta/server/rgb.py", line 135, in _get_rgb_image
  some_keys, rgb_values, stretch_ranges=stretch_ranges, tile_xyz=tile_xyz, **options
  File "/var/lang/lib/python3.6/contextlib.py", line 52, in inner
  return func(*args, **kwds)
  File "/var/task/terracotta/handlers/rgb.py", line 85, in rgb
  band_data = band_data_future.result()
  File "/var/lang/lib/python3.6/concurrent/futures/_base.py", line 432, in result
  return self.__get_result()
  File "/var/lang/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
  raise self._exception
  File "/var/lang/lib/python3.6/concurrent/futures/thread.py", line 56, in run
  result = self.fn(*self.args, **self.kwargs)
  File "/var/task/cachetools/__init__.py", line 87, in wrapper
  v = method(self, *args, **kwargs)
  File "/var/lang/lib/python3.6/contextlib.py", line 52, in inner
  return func(*args, **kwds)
  File "/var/task/terracotta/drivers/raster_base.py", line 466, in _get_raster_tile
  raise IOError('error while reading file {}'.format(path))
OSError: error while reading file s3://italy-composite/rasters/italy_2018_red.tif

We have a project where we want to use Terracotta to serve up some huge watermasks. There's no way we can load an entire file into memory and do computations (a 32gb machine fails when computing the metadata), this is of course no problem for serving the files, as they are cloud-optimized.

However, the metadata computation when creating the database still assumes that the entire file fits into memory and then some. So we should use Dask to chunk the computations when sizes exceed the memory limit.

To speed up the common case (where files fit into memory) we could do this only when a MemoryError is thrown. Or we could set a memory limit that we think is reasonable and always chunk the files such that we never exceed that and then maybe decrease it if we hit a MemoryError. Thoughts?

Implement the compositional TerracottaDriver structure proposed in https://github.com/DHI-GRAS/terracotta/pull/240#issuecomment-1013688976.

In a separate PR (going into #240) for better diffs.

~~Draft, WIP, PR. Just made to keep track of things, and for the tests to run. Can be ignored for now.~~

Edit: PR for improving the structure and codebase of the drivers. Was originally just for replacing the database specific code with a more general SQLAlchemy implementation. But has now also turned into a bigger, more general refactor of the entire driver code.

Great job so far! Here's the things I stumbled upon:

Documentation

• Be consistent: timestep vs timestamp
• I don't think we need to explain the layout of the option files; an example is sufficient.
• I think it is tremendously helpful to see example responses of the API calls early on.

Configuration

• Why split path and regex? Just have path_regex.
• Not sure about the yes/no syntax for boolean settings. How does e.g. Apache or Nginx handle that?

CLI

• Config path could be a positional argument
• Please wrap the config path in os.expanduser for us poor windows souls
• 💡: Accept rasters from the command line to quickly serve up anything: terracotta *.tif (then open a leaflet map in the browser, with the data already added as a layer, for the ultimate wow effect :smile:)

API

• I don't think the API queries should include terracotta. You would either run this as a Flask app on its own port, or configure the proxy in your webserver.
• Using a non-timestep API endpoint for a timestepped dataset causes an uncaught exception (500 server error; should give "Bad Request" or so)

I'll have a look at the actual code and do some profiling later. I'll update this issue with my findings.

The usecase that lead to this change is overlaying cloud coverage. See screenshot below. Cloud coverage information is extrapolated from GOES-16 ABI channel 13 and displayed on top of Bing Aerial Maps. Allowing a fourth channel in the predefined color maps allows us to have a varying transparency without sending a bulky json custom colormap with each api request. The varying transparency gives us the nice feathered edges for the clouds.

Use a multiprocessing.Pool to optimize raster files in parallel (each raster file is applied individually). Closes #55

Currently uses multiprocessing.cpu_count() number of threads. This seems to be the number of logical (hyper-threading) cores: I'm unsure whether the number of physical cores might be preferable.

Currently has some caveats: 1: The sub-progress bars are removed, such that only the overall progress is shown. (And the filename postfix is now showing the just-completed file, rather than the currently-being-processed file).

2: When raising an exception because optimized files already exists (and neither --overwrite nor --skip-existing flags are set), the entire trace-stack is now shown, where it previously just showed a nice little message.

3: The test_reoptimize test in tests/scripts/test_optimize_rasters.py fails. Not because the code performs wrongly, but because of the way exceptions inside threads are handled. When submitting a task to the Pool, an error_callback function is given, which receives any errors which occurs in the thread (and handles them in the main-thread). But apparently pytest recognizes that an error has been thrown inside the threads, and fails the test based on that. If a preprocessed file already exists, and terracotta optimize-rasters is run without --overwrite or --skip-existing flags, then the expected behaviour is to throw an exception. But somehow pytest expects the exception to be raised differently, or something like that? And I can't figure out how to make the test accept the new exception flow, such that the test just tests whether the code handles preexisting optimized files properly, and not where/how the exception is raised.

The SQLAlchemy broke setting MYSQL_USER and MYSQL_PASSWORD settings. This PR fixes that.

It also renames the fields from MYSQL_* to the more generic SQL_*, in anticipation of the coming PostgreSQL driver (and potential future others). This is currently just a plain rename, thus breaking compatibility with deployments using the previous fields. We need to decide whether that is okay, or whether there should be some deprecation warning system first.

Ideally, we also ought to make some tests for this. I have apparently previously broken this, without noticing. That shouldn't be possible in the future. Feel free to just add such tests -- it may take a while before I'm back to do it.

Is there a way or a workaround with terracotta that can help me serve XYZ tiles of a given polygon from a large tile? So the case is, I have a large tile but I want only a polygon section of it to be served, rest be masked out invalid, dynamically? I'm using Leaflet maps to visualise the raster tiles.

Make CORS allowed origins configurable for metadata and tiles via new environment variables:

• TC_ALLOWED_ORIGINS_METADATA = '["*"]'
• TC_ALLOWED_ORIGINS_TILES = '[]'

Where empty list [] means no allowed origins.

The changes are backward-compatible, so even when the user omits the new environment variables but updates Terracotta, the behavior will be the same as before: all origins allowed for /metadata, no cross-origin headers for tiles.

Closes #160

Hi all, I am trying to deploy terracotta to AWS but having issues. While trying to debug I realised that there is a discrepancy between the conda version and the pip version of terracotta, despite both being 0.7.5.

in the sqlite database installed with pip the keys table is called keys, while in the conda version the table is called key_names with the primary key also called "key_name" instead of "key". This obviously creates problems when trying to run the db with terracotta serve -d

I am seeing some weird behaviour when trying to use the RelationalMetaStore class, specifically the db connection handling. Since the connection is created once and yielded down the call stack it seems we might run into some issues when trying to do the commit or the rollback if any of the downstream calls errored out. We essentially end up with a lot of AttributeError: 'NoneType' object has no attribute 'rollback'/'commit', which seems weird to me since these are only set to None in the finally block in the top-most connect call.

I tried adding a little if self.connected: check before commiting or rollbacking which seems to reduce errors and smooth loading of the PNGs, but that seems a little icky since we are in a if not self.connected loop.

Am I looking at this wrong? Has anyone tried using the RelationalMetaStore implementation and seen/not seen this before? I tried thinking about a reproducible example but I am struggling to figure out how that could be done.

I wanted to use the lovely postgresql driver implemented by @nickeopti sometime back in March when it struck me that these changes werent included in the newest pypi release. Would the changes since 0.7.5 warrant a new release? 🙏

Hello

I manage to use geotiffs as per the examples, that is making single band r/g/b files.

Is it possible to directly use the RGB geotiffs?

Thanks

best regards

Gilles

Hi All,

I need to find the best way to run the terracotta optimize-raster and also importing the terracotta files in another python lambda to run the ingestion-script. Is it possible to run the Terracotta CLI/API through the deployed aws lambda system ?

Thanks you in advance for your time.

Experimental support for WMTS, including the Dimension tag. This is at the moment pretty much a working proof-of-concept, initially made just to test the feasibility. A lot of polishing is required before it may become ready to merge.

But first, is this a feature we want to add? It is suprisingly simple, and doesn't add much complexity, as it primarily relies on existing functionality. It essentially just provides an endpoint for fetching GetCapabilities -- the rest is handled as usual.

As far as I can tell the implementation adheres to the WMTS standard. I've cross-checked with Titiler as a reference point, and the processes seem comparable (apart from their implementation being much more generalised). The implementation is tested in QGIS on small test rasters.

Remaining tasks and questions (if we want to add it):

1. Support usage of all the normal endpoints for rasters; /rgb, /compute(?), as well as /singleband, along with all their options. This is currently hardcoded to /singleband without any options.
2. Dynamically add zoom levels instead of hardcoding them in the wmts.xml template file? If doing so, data is available here http://schemas.opengis.net/tms/1.0/json/examples/.
3. Are there any restrictions on zoom levels? I see Titiler deduce zoom level restrictions from data, somehow. Does Terracotta have such limits?
4. Is it correctly understood that WorldWebMercatorQuad is the correct reference system?
5. I've used xml.etree.ElementTree to build the capabilities XML file. It seems like the proper way for me, but I'm not satisfied with the code. I think it is too convoluted, and frankly hard to read. I'm actually leaning towards inserting elements as strings in the template instead. Thoughts on this?
6. Where shall static template/data files be stored? Regarding the wmts.xml and possibly WebMercatorQuad.json file(s).
7. Currently the dimension option blindly trusts the user that the supplied key makes sense as a dimension (ie that the raster files have the same bounds). Shall we verify this?
8. Documentation.
9. Testing. No efforts have been made here yet.