Fiona reads and writes geographic data files

Overview

Fiona

Fiona reads and writes geographic data files and thereby helps Python programmers integrate geographic information systems with other computer systems. Fiona contains extension modules that link the Geospatial Data Abstraction Library (GDAL).

https://travis-ci.org/Toblerity/Fiona.png?branch=master https://ci.appveyor.com/api/projects/status/github/Toblerity/Fiona?svg=true https://coveralls.io/repos/Toblerity/Fiona/badge.png

Fiona is designed to be simple and dependable. It focuses on reading and writing data in standard Python IO style and relies upon familiar Python types and protocols such as files, dictionaries, mappings, and iterators instead of classes specific to GDAL's OpenGIS Reference Implementation (OGR). Fiona can read and write real-world data using multi-layered GIS formats and zipped virtual file systems and integrates readily with other Python GIS packages such as pyproj, Rtree, and Shapely.

Fiona is supported only on CPython versions 3.6+.

Why the name "Fiona"? Because Fiona is OGR's neat and nimble API for Python programmers. And a Shrek reference made us laugh.

For more details, see:

Usage

Collections

Records are read from and written to file-like Collection objects returned from the fiona.open() function. Records are mappings modeled on the GeoJSON format. They don't have any spatial methods of their own, so if you want to do anything fancy with them you will probably need Shapely or something like it. Here is an example of using Fiona to read some records from one data file, change their geometry attributes, and write them to a new data file.

import fiona

# Open a file for reading. We'll call this the "source."

with fiona.open('tests/data/coutwildrnp.shp') as src:

    # The file we'll write to, the "destination", must be initialized
    # with a coordinate system, a format driver name, and
    # a record schema.  We can get initial values from the open
    # collection's ``meta`` property and then modify them as
    # desired.

    meta = src.meta
    meta['schema']['geometry'] = 'Point'

    # Open an output file, using the same format driver and
    # coordinate reference system as the source. The ``meta``
    # mapping fills in the keyword parameters of fiona.open().

    with fiona.open('test_write.shp', 'w', **meta) as dst:

        # Process only the records intersecting a box.
        for f in src.filter(bbox=(-107.0, 37.0, -105.0, 39.0)):

            # Get a point on the boundary of the record's
            # geometry.

            f['geometry'] = {
                'type': 'Point',
                'coordinates': f['geometry']['coordinates'][0][0]}

            # Write the record out.

            dst.write(f)

# The destination's contents are flushed to disk and the file is
# closed when its ``with`` block ends. This effectively
# executes ``dst.flush(); dst.close()``.

Reading Multilayer data

Collections can also be made from single layers within multilayer files or directories of data. The target layer is specified by name or by its integer index within the file or directory. The fiona.listlayers() function provides an index ordered list of layer names.

for layername in fiona.listlayers('tests/data'):
    with fiona.open('tests/data', layer=layername) as src:
        print(layername, len(src))

# Output:
# ('coutwildrnp', 67)

Layer can also be specified by index. In this case, layer=0 and layer='test_uk' specify the same layer in the data file or directory.

for i, layername in enumerate(fiona.listlayers('tests/data')):
    with fiona.open('tests/data', layer=i) as src:
        print(i, layername, len(src))

# Output:
# (0, 'coutwildrnp', 67)

Writing Multilayer data

Multilayer data can be written as well. Layers must be specified by name when writing.

with open('tests/data/cowildrnp.shp') as src:
    meta = src.meta
    f = next(src)

with fiona.open('/tmp/foo', 'w', layer='bar', **meta) as dst:
    dst.write(f)

print(fiona.listlayers('/tmp/foo'))

with fiona.open('/tmp/foo', layer='bar') as src:
    print(len(src))
    f = next(src)
    print(f['geometry']['type'])
    print(f['properties'])

    # Output:
    # ['bar']
    # 1
    # Polygon
    # OrderedDict([('PERIMETER', 1.22107), ('FEATURE2', None), ('NAME', 'Mount Naomi Wilderness'), ('FEATURE1', 'Wilderness'), ('URL', 'http://www.wilderness.net/index.cfm?fuse=NWPS&sec=wildView&wname=Mount%20Naomi'), ('AGBUR', 'FS'), ('AREA', 0.0179264), ('STATE_FIPS', '49'), ('WILDRNP020', 332), ('STATE', 'UT')])

A view of the /tmp/foo directory will confirm the creation of the new files.

$ ls /tmp/foo
bar.cpg bar.dbf bar.prj bar.shp bar.shx

Collections from archives and virtual file systems

Zip and Tar archives can be treated as virtual filesystems and Collections can be made from paths and layers within them. In other words, Fiona lets you read and write zipped Shapefiles.

for i, layername in enumerate(
        fiona.listlayers('zip://tests/data/coutwildrnp.zip'):
    with fiona.open('zip://tests/data/coutwildrnp.zip', layer=i) as src:
        print(i, layername, len(src))

# Output:
# (0, 'coutwildrnp', 67)

Fiona can also read from more exotic file systems. For instance, a zipped shape file in S3 can be accessed like so:

with fiona.open('zip+s3://mapbox/rasterio/coutwildrnp.zip') as src:
    print(len(src))

# Output:
# 67

Fiona CLI

Fiona's command line interface, named "fio", is documented at docs/cli.rst. Its fio info pretty prints information about a data file.

$ fio info --indent 2 tests/data/coutwildrnp.shp
{
  "count": 67,
  "crs": "EPSG:4326",
  "driver": "ESRI Shapefile",
  "bounds": [
    -113.56424713134766,
    37.0689811706543,
    -104.97087097167969,
    41.99627685546875
  ],
  "schema": {
    "geometry": "Polygon",
    "properties": {
      "PERIMETER": "float:24.15",
      "FEATURE2": "str:80",
      "NAME": "str:80",
      "FEATURE1": "str:80",
      "URL": "str:101",
      "AGBUR": "str:80",
      "AREA": "float:24.15",
      "STATE_FIPS": "str:80",
      "WILDRNP020": "int:10",
      "STATE": "str:80"
    }
  }
}

Installation

Fiona requires Python versions 3.6+ and GDAL version 1.11-3.0. To build from a source distribution you will need a C compiler and GDAL and Python development headers and libraries (libgdal1-dev for Debian/Ubuntu, gdal-dev for CentOS/Fedora).

To build from a repository copy, you will also need Cython to build C sources from the project's .pyx files. See the project's requirements-dev.txt file for guidance.

The Kyngchaos GDAL frameworks will satisfy the GDAL/OGR dependency for OS X, as will Homebrew's GDAL Formula (brew install gdal).

Python Requirements

Fiona depends on the modules six, cligj, and munch. Pip will fetch these requirements for you, but users installing Fiona from a Windows installer must get them separately.

Unix-like systems

Assuming you're using a virtualenv (if not, skip to the 4th command) and GDAL/OGR libraries, headers, and gdal-config program are installed to well known locations on your system via your system's package manager (brew install gdal using Homebrew on OS X), installation is this simple.

$ mkdir fiona_env
$ virtualenv fiona_env
$ source fiona_env/bin/activate
(fiona_env)$ pip install fiona

If gdal-config is not available or if GDAL/OGR headers and libs aren't installed to a well known location, you must set include dirs, library dirs, and libraries options via the setup.cfg file or setup command line as shown below (using git). You must also specify the version of the GDAL API on the command line using the --gdalversion argument (see example below) or with the GDAL_VERSION environment variable (e.g. export GDAL_VERSION=2.1).

(fiona_env)$ git clone git://github.com/Toblerity/Fiona.git
(fiona_env)$ cd Fiona
(fiona_env)$ python setup.py build_ext -I/path/to/gdal/include -L/path/to/gdal/lib -lgdal install --gdalversion 2.1

Or specify that build options and GDAL API version should be provided by a particular gdal-config program.

(fiona_env)$ GDAL_CONFIG=/path/to/gdal-config pip install fiona

Windows

Binary installers are available at http://www.lfd.uci.edu/~gohlke/pythonlibs/#fiona and coming eventually to PyPI.

You can download a binary distribution of GDAL from here. You will also need to download the compiled libraries and headers (include files).

When building from source on Windows, it is important to know that setup.py cannot rely on gdal-config, which is only present on UNIX systems, to discover the locations of header files and libraries that Fiona needs to compile its C extensions. On Windows, these paths need to be provided by the user. You will need to find the include files and the library files for gdal and use setup.py as follows. You must also specify the version of the GDAL API on the command line using the --gdalversion argument (see example below) or with the GDAL_VERSION environment variable (e.g. set GDAL_VERSION=2.1).

$ python setup.py build_ext -I<path to gdal include files> -lgdal_i -L<path to gdal library> install --gdalversion 2.1
$ set GDAL_VERSION=3.0
$ pip install --install-option=build_ext --install-option="-I<drive letter>:\\<path to gdal include files>\\include" --install-option="-lgdal_i" --install-option="-L<drive letter>:\\<path to gdal lib files>\\libs" fiona

Note: The GDAL DLL (gdal111.dll or similar) and gdal-data directory need to be in your Windows PATH otherwise Fiona will fail to work.

The Appveyor CI build uses the GISInternals GDAL binaries to build Fiona. This produces a binary wheel for successful builds, which includes GDAL and other dependencies, for users wanting to try an unstable development version. The Appveyor configuration file may be a useful example for users building from source on Windows.

Development and testing

Building from the source requires Cython. Tests require pytest. If the GDAL/OGR libraries, headers, and gdal-config program are installed to well known locations on your system (via your system's package manager), you can do this:

(fiona_env)$ git clone git://github.com/Toblerity/Fiona.git
(fiona_env)$ cd Fiona
(fiona_env)$ pip install cython
(fiona_env)$ pip install -e .[test]
(fiona_env)$ py.test

Or you can use the pep-518-install script:

(fiona_env)$ git clone git://github.com/Toblerity/Fiona.git
(fiona_env)$ cd Fiona
(fiona_env)$ ./pep-518-install

If you have a non-standard environment, you'll need to specify the include and lib dirs and GDAL library on the command line:

(fiona_env)$ python setup.py build_ext -I/path/to/gdal/include -L/path/to/gdal/lib -lgdal --gdalversion 2 develop
(fiona_env)$ py.test
Comments
  • Circular imports for Python 3.8 on Windows if DLLs are already loaded

    Circular imports for Python 3.8 on Windows if DLLs are already loaded

    WIth fiona 1.8.15, the issue is fixed now in the conda-forge builds, only the Windows py3.8 build is still failing: https://github.com/conda-forge/fiona-feedstock/pull/167

    The failure is with the general error message of partially initialized module, so doesn't give much pointers (at least to me):

    running run_test.py
    Traceback (most recent call last):
      File "D:\bld\fiona_1599200296823\test_tmp\run_test.py", line 6, in <module>
        import fiona
      File "D:\bld\fiona_1599200296823\_test_env\lib\site-packages\fiona\__init__.py", line 84, in <module>
        import fiona._loading
      File "D:\bld\fiona_1599200296823\_test_env\lib\site-packages\fiona\_loading.py", line 44, in <module>
        import fiona.ogrext
      File "fiona/ogrext.pyx", line 29, in init fiona.ogrext
      File "D:\bld\fiona_1599200296823\_test_env\lib\site-packages\fiona\env.py", line 14, in <module>
        with fiona._loading.add_gdal_dll_directories():
    AttributeError: partially initialized module 'fiona' has no attribute '_loading' (most likely due to a circular import)
    

    Originally posted by @jorisvandenbossche in https://github.com/Toblerity/Fiona/issues/941#issuecomment-686982973

    opened by rbuffat 45
  • [WIP] Add support for FlatGeobuf

    [WIP] Add support for FlatGeobuf

    It looks as Fiona is unable to write any records with the FlatGeobuf driver, but also the log does not show any errors. The metadata seems to be saved, but nothing else. Reading of FlatGeobuf seems to be fine though. I currently don't see where the problem could be.

    @tomplex @kylebarron did you had any luck with writing data?

    @bjornharrtell if I'm not mistaken, you are the author of the gdal FlatGeobuf driver. Maybe you have an idea why this is happening. But don't feel obligated to have a look at this! The relevant code for writing would be here: https://github.com/Toblerity/Fiona/blob/maint-1.8/fiona/ogrext.pyx#L1211-L1284

    opened by rbuffat 33
  • Unable to open EPSG support file gcs.csv

    Unable to open EPSG support file gcs.csv

    This issue has the same symptoms as https://github.com/mapbox/rasterio/issues/1539:

    In [1]: import fiona                                                                                                                                                                                                
    
    In [2]: ds = fiona.open('aerialway-line.shp')                                                                                                                                                                       
    
    In [3]: ds.crs                                                                                                                                                                                                      
    ERROR 4: Unable to open EPSG support file gcs.csv.  Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.
    ERROR 6: No translation for an empty SRS to PROJ.4 format is known.
    Out[3]: {}
    
    In [4]: fiona.__version__                                                                                                                                                                                           
    Out[4]: '1.8.0'
    

    The open function is decorated by ensure_env_with_credentials but it seems doesn't work.

    opened by drnextgis 28
  • CRS Axis Order issues with 1.8.9

    CRS Axis Order issues with 1.8.9

    Problem Description

    When reading in: https://raw.githubusercontent.com/geopandas/geopandas/master/geopandas/tests/data/overlay/overlap/df1_df2_overlap-union.geojson

    With previous versions of Fiona, the WKT string was read in as:

    GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Latitude",NORTH],AXIS["Longitude",EAST],AUTHORITY["EPSG","4326"]]
    

    However, currently (1.8.9) it is read in as:

    GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Longitude",EAST],AXIS["Latitude",NORTH]]
    

    Everything is mostly the same except for the axis order at the end.

    I believe this is due to this line: https://github.com/Toblerity/Fiona/blob/960568d2fede452a042e5c9f315fb93e3cd56baa/fiona/_crs.pyx#L65

    This is problematic for a couple of reasons:

    1. It does not preserve the original WKT string/projection and mangles it
    2. What if the user actually modified their data to be in the axis order mentioned in the WKT string? How would they tell Fiona that they did so?

    Recommended Solution

    Add an always_xy flag to crs_to_wkt() and only use it when creating a CRS to be used in the transformation. It would also be a good idea to add an always_xy flag to the transformation operations for the case where the user modified the underlying data to match the axis order specified in the CRS. For Fiona 1.8.x, the always_xy flag could be omitted from the transformation operations and just default to always_xy=True in crs_to_wkt()

    Thoughts?

    opened by snowman2 27
  • Driver not found (GPKG)

    Driver not found (GPKG)

    Expected behavior and actual behavior.

    GPKG driver is avaliable

    Steps to reproduce the problem.

    import fiona
    from fiona._drivers import GDALEnv
    env = GDALEnv()
    
    "GPKG" in env.start().drivers().keys()
    True
    

    Operating system

    Linux, Ubuntu 17.10

    Fiona version and provenance

    >>> fiona.__version__
    '1.7.10.post1'
    >>> 
    

    from pip

    opened by jachym 26
  • Add shim module and support for GDAL 3

    Add shim module and support for GDAL 3

    I propose to merge this and make a 1.8.9 release without fixing the Appveyor build so that we can ease the PROJ 4/6 pain of users. Let's tackle the Appveyor updates in the master branch.

    Summary of the PR:

    • Adds a shim module for GDAL 3, introduces two new functions (osr_get_name and osr_set_traditional_axis_mapping_strategy), and adds no-op versions of the functions to the older shims.
    • Everywhere we initialize an OGRSpatialReferenceH object using OSRImportFrom*, we set its axis order to the "traditional" one to preserve the existing axis order.
    • We disable two tests for GDAL 3 because towgs84 and wktext don't surface in GDAL's WKT output anymore.

    @hobu you're interested in this, I know. Want to give it a quick scan? I realize that some of the Cython files are archaic and a bit opaque.

    @snorfalorpagus any objections?

    opened by sgillies 19
  • Segmentation fault with simple geometry operations using Shapely and Fiona

    Segmentation fault with simple geometry operations using Shapely and Fiona

    Hi,

    I am having trouble making sense of an issue I'm having using Shapely and Fiona. I keep getting segmentation faults. even when doing very simple tasks. I've narrowed it down to being a problem having shapely and fiona loaded at the same time, which makes either package much less useful.

    I've included a minimal working example below. The first runs just fine with only shapely loaded. The second produces a segmentation fault running the same processes with shapely and fiona loaded.

    Example 1, works:

    In [1]: %paste
    from shapely import geometry
    shp = geometry.LinearRing([(0,0),(1,0),(0,1)])
    shp.centroid.coords.xy
    Out[1]: (array('d', [0.35355339059327373]), array('d', [0.35355339059327373]))
    

    Example 2, segfault:

    In [1]: %paste
    import fiona
    from shapely import geometry
    shp = geometry.LinearRing([(0,0),(1,0),(0,1)])
    shp.centroid.coords.xy
    ## -- End pasted text --
    /Users/wiar9509/anaconda/bin/python.app: line 3:   849 Segmentation fault: 11  /Users/wiar9509/anaconda/python.app/Contents/MacOS/python "$@"
    

    I am running this in iPython, using an Anaconda install. My Python information is: Python 2.7.12 |Anaconda custom (x86_64)| (default, Jul 2 2016, 17:43:17)

    My iPython version is:

    engr2-2-77-37-dhcp:~ wiar9509$ conda list ipython
    # packages in environment at /Users/wiar9509/anaconda:
    #
    ipython                   4.2.0                    py27_1  
    ipython_genutils          0.1.0                    py27_0 
    

    Fiona version:

    In [2]: fiona.__version__
    Out[2]: '1.7.0'
    

    Shapely version:

    In [3]: shapely.__version__
    Out[3]: '1.5.16'
    

    Please let me know how I might be able to solve this issue. I am excited to use these packages!

    Billy

    packaging 
    opened by armstrwa 19
  • Intermittent / transient fiona.open errors using AWS S3 + lambda

    Intermittent / transient fiona.open errors using AWS S3 + lambda

    Expected behavior and actual behavior.

    I'm using fiona==1.8.6 on AWS lambda to open shapefiles on S3. I'm seeing occasional clusters of fiona.open errors and was wondering (1) whether anyone else has seen similar behavior, and (2) whether there is anything I can do to reduce the frequency of these errors.

    The stack traces look like this:

    my_shapefile = fiona.open(shapefile_uri)
    File "/var/task/fiona/__init__.py", line 249, in open
    enabled_drivers=enabled_drivers, **kwargs)
    File "/var/task/fiona/collection.py", line 160, in __init__
    self.session.start(self, **kwargs)
    File "fiona/ogrext.pyx", line 468, in fiona.ogrext.Session.start
    File "fiona/_shim.pyx", line 73, in fiona._shim.gdal_open_vector
    fiona.errors.DriverError: '/vsis3/my_aws_bucket/my_reference_data/index.shp' not recognized as a supported file format.
    

    In case it matters, index.shp in the stack trace above is a US national index shapefile pointing to state-specific child shapefiles.

    The stack traces can also look like this:

    my_shapefile = fiona.open(shapefile_uri)
    File "/var/task/fiona/env.py", line 406, in wrapper
    return f(*args, **kwargs)
    File "/var/task/fiona/__init__.py", line 253, in open
    layer=layer, enabled_drivers=enabled_drivers, **kwargs)
    File "/var/task/fiona/collection.py", line 159, in __init__
    self.session.start(self, **kwargs)
    File "fiona/ogrext.pyx", line 484, in fiona.ogrext.Session.start
    File "fiona/_shim.pyx", line 80, in fiona._shim.gdal_open_vector
    fiona.errors.DriverError: Failed to read all values for 33144 records in .shx file: No such file or directory.
    

    These errors occur infrequently but are clustered in time -- see below.

    Steps to reproduce the problem.

    I'm not able to reproduce the problem consistently. My AWS lambda code opens the same S3 shapefile every time the lambda is invoked. The fiona.open call succeeds most of the time (say, >95%), but I occasionally get the errors described above (and these errors tend to be clustered in time).

    Operating system

    AWS lambda with runtime = "python3.6" (see https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html).

    Fiona and GDAL version and provenance

    I've gotten this error with both fiona==1.8.6 and fiona==1.8a2.

    opened by atorch 18
  • FionaValueError: No dataset found at path

    FionaValueError: No dataset found at path

    I suddenly keep getting:

    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "C:\Python27\lib\site-packages\fiona\__init__.py", line 175, in open
        enabled_drivers=enabled_drivers)
      File "C:\Python27\lib\site-packages\fiona\collection.py", line 147, in __init__
        self.session.start(self)
      File "fiona\ogrext.pyx", line 444, in fiona.ogrext.Session.start (fiona/ogrext.c:9253)
    FionaValueError: No dataset found at path '\test.shp' using drivers: *
    

    I use fiona through rasterstats (https://github.com/perrygeo/python-rasterstats). I get this with every .shp files. Even those that used to work.

    The problem is that I don't know exactly what changed. I've updated to: Python 2.7.13 And from Gohlke: Fiona 1.7.1 GDAL 2.0.2 (because the 2.1.2 version resulted in Python crashes) From gisinternals GDAL 2.1.2 and it started not working.

    I used to use: Python 2.7.11 And from Gohlke: Fiona 1.6.3 GDAL 2.0.2 From gisinternals GDAL 2.1.0

    The problem is when I uninstall everything (even GDAL from gisinternals) and go back to 1.6.3, the problem persists. ogrext.pyx doesn't seems to be anywhere (only .pyd) but I guess that's normal?

    I apologize already for probably not giving enough information. Many thanks in advance.

    opened by vanderlist 18
  • Windows CI

    Windows CI

    A Windows CI could be useful to test Fionas behavior on windows platform or in the future to build binary wheels for windows.

    I started playing with appveyor: https://github.com/rbuffat/Fiona/blob/win/appveyor.yml, respectively: https://ci.appveyor.com/project/rbuffat/fiona @micahcochran the current status is that fiona can built but at runtime the following error is thrown:

    Traceback (most recent call last):
    File "c:\python27.10-x64\lib\site-packages\nose\suite.py", line 209, in run
        self.setUp()
    File "c:\python27.10-x64\lib\site-packages\nose\suite.py", line 292, in setUp
        self.setupContext(ancestor)
    File "c:\python27.10-x64\lib\site-packages\nose\suite.py", line 315, in setupContext
        try_run(context, names)
    File "c:\python27.10-x64\lib\site-packages\nose\util.py", line 471, in try_run
        return func()
    File "C:\projects\fiona\tests\__init__.py", line 39, in setup
        create_jsonfile(jsonfile)
    File "C:\projects\fiona\tests\__init__.py", line 20, in create_jsonfile
        import fiona
    File "build\bdist.win-amd64\egg\fiona\__init__.py", line 72, in <module>
    File "build\bdist.win-amd64\egg\fiona\collection.py", line 7, in <module>
    File "build\bdist.win-amd64\egg\fiona\ogrext.py", line 7, in <module>
    File "build\bdist.win-amd64\egg\fiona\ogrext.py", line 6, in __bootstrap__
    ImportError: DLL load failed: %1 is not a valid Win32 application.
    

    My current research revealed that this can be caused by various reasons such as missing files, missing paths, incompatible compiled dll's etc.

    intermediate 
    opened by rbuffat 18
  • remove per-feature logging to improve performance issue#685

    remove per-feature logging to improve performance issue#685

    I saw the issue #685, and so I took a stab at it based on the discussion to see if I could help move things forward.

    I commented out the debug print statements instead of deleting them so that you don't have to re-create them when you are debugging (and an official decision on what to do with them has not been reached as far as I know). However, that presents the danger of being re-introduced by accident when committing changes after debugging.

    Using this example:

    from itertools import chain, repeat
    import fiona
    def test_to_file():
        with fiona.Env(),fiona.open("../tests/data/coutwildrnp.shp") as collection:
            features = chain.from_iterable(repeat(list(collection), 2000))
            with fiona.open("/tmp/out.gpkg", "w", schema=collection.schema, crs=collection.crs, driver="GPKG") as dst:
                dst.writerecords(features)
    

    And timing it:

    %%timeit
    test_to_file()
    

    Before: 1min 3s ± 2.32 s per loop (mean ± std. dev. of 7 runs, 1 loop each) After: 15.3 s ± 1.13 s per loop (mean ± std. dev. of 7 runs, 1 loop each)

    opened by snowman2 17
  • Support add new columns to gpkg in append mode

    Support add new columns to gpkg in append mode

    Hi, I'm using Fiona from GeoPandas, actually there is the append mode, but if the data we want to append has columns that does not exists will fails. I'm constructing a big gpkg file but I don't have a way to know all the columns beforehand.

      File "fiona/ogrext.pyx", line 1280, in fiona.ogrext.WritingSession.writerecs
    ValueError: Record does not match collection schema: dict_keys(list_with_column_names)
    

    Thx!

    opened by latot 0
  • Update and streamline the readme

    Update and streamline the readme

    I've made the readme fresher, removing mentions of obsolete GDAL installers, and much shorter, removing some old examples and details about installation from source that are better off in a more advanced doc.

    Updating the readme example suggested that Feature &c should be exported in fiona/__init__.py and that the Geometry class should know about __geo_interface__ to integrate better with Shapely.

    opened by sgillies 0
  • Fail to read an online zipped ESRI GDB file with Fiona 1.8.22, but success with Fiona 1.8.13

    Fail to read an online zipped ESRI GDB file with Fiona 1.8.22, but success with Fiona 1.8.13

    I expected to read an online zipped ESRI GDB file with Fiona 1.8.22, it failed, while when I try it in another environment with Fiona 1.8.13, it succeeded.

    Steps to reproduce the problem.

    import fiona import geopandas as gp path="https://raw.githubusercontent.com/UrbanGISer/Test/main/Case05new.gdb.zip" c = fiona.open(path,layer=0) gdf = gp.GeoDataFrame.from_features(c) gdf.crs=c.crs.get('init') gdf.head()

    Operating system

    Win 11

    Fiona and GDAL version and provenance

    Fiona 1.8.22 with print (osgeo.gdal.version) 3.5.2 Fiona 1.8.13 with print (osgeo.gdal.version) 3.0.2

    needs discussion 
    opened by UrbanGISer 3
  • Update README.rst with infos regarding to GDAL embedded wheels for windows

    Update README.rst with infos regarding to GDAL embedded wheels for windows

    Expected behavior and actual behavior.

    No bug here, but an out of date documentation (see. discussion on fiona.groups.io).

    The actual documentation doesn't acknowledge that there are now windows wheels on pypi and that they include a GDAL shared library.

    The documentation might also benefit from a paragraph stating what could and what couldn't be done regarding to diverse GDAL's bindings (osgeo and all it's dependencies).

    bug docs 
    opened by tgrandje 0
  • test_no_append_driver_cannot_append fails

    test_no_append_driver_cannot_append fails

    https://gist.github.com/risicle/2631e23beaf492e95d6d1ef44d3d789e

    Operating system

    NixOS 22.11

    Fiona and GDAL version and provenance

    Fiona 1.8.22 installed via Nix GDAL 3.6.0.1 installed via Nix

    opened by dotlambda 1
Releases(1.9b1)
  • 1.9b1(Dec 13, 2022)

    The fist 1.9 beta release is ready for early testing. All of the features planned for 1.9.0 are in this release.

    The binary wheels on PyPI include GDAL 3.5.3, GEOS 3.11.1, and PROJ 9.0.1. To try the wheels, run pip install --pre fiona.

    New features

    • Add listdir and listlayers method to io.MemoryFile (resolving #754).
    • Add support for TIN and triangle geometries (#1163).
    • Add an allow_unsupported_drivers option to fiona.open() (#1126).
    • Added support for the OGR StringList field type (#1141).

    Changes and bug fixes

    • Missing and unused imports have been added or removed.
    • Make sure that errors aren't lost when a collection can't be saved properly (#1169).
    • Ensure that ZipMemoryFile have the proper GDAL name after creation so that we can use listdir() (#1092).
    • The fiona._loading module, which supports DLL loading on Windows, has been moved into __init__.py and is no longer used anywhere else (#1168).
    • Move project metadata to pyproject.toml (#1165).
    • Update drvsupport.py to reflect new format capabilities in GDAL 3.6.0 (#1122).
    • Remove debug logging from env and _env modules.
    Source code(tar.gz)
    Source code(zip)
  • 1.9a3(Oct 18, 2022)

    Builds now require Cython >= 0.29.29 because of https://github.com/cython/cython/issues/4609 (see https://github.com/Toblerity/Fiona/issues/1143).

    Wheels include GDAL 3.5.2, PROJ 9.0.1, and GEOS 3.11.0 and are available for Python versions 3.7-3.11.

    Source code(tar.gz)
    Source code(zip)
  • 1.8.22(Oct 14, 2022)

    Builds now require Cython >= 0.29.29 because of https://github.com/cython/cython/issues/4609 (see #1143).

    Wheels include GDAL 3.4.3, PROJ 8.2.1, and GEOS 3.10.2.

    Source code(tar.gz)
    Source code(zip)
  • 1.9a2(Jun 10, 2022)

    The second 1.9 pre-release is ready for early testing. Not all of the features planned for 1.9.0 are in this release, and some of the features here may yet be modified. Please pay close attention to the deprecations and packaging changes.

    The binary wheels on PyPI include GDAL 3.5.0 and PROJ 9.0.0, the latest versions of each. To try the wheels, run pip install --pre fiona.

    Deprecations:

    • Fiona's API methods will accept feature and geometry dicts in 1.9.0, but this usage is deprecated. Instances of Feature and Geometry will be required in 2.0.
    • The precision keyword argument of fiona.transform.transform_geom is deprecated and will be removed in version 2.0.
    • Deprecated usage has been eliminated in the project. Fiona's tests pass when run with a -Werror::DeprecationWarning filter.

    Changes:

    • Fiona's FionaDeprecationWarning now sub-classes DeprecationWarning.
    • Some test modules have beeen re-formatted using black.

    New features:

    • Fiona Collections now carry a context exit stack into which we can push fiona Envs and MemoryFiles (#1059).
    • Fiona has a new CRS class, like rasterio's, which is compatible with the CRS dicts of previous versions (#714).
    Source code(tar.gz)
    Source code(zip)
  • 1.9a1(May 19, 2022)

    The first 1.9 pre-release is ready for early testing. Not all of the features planned for 1.9.0 are in this release, and some of the features here may yet be modified. Please pay close attention to the deprecations and packaging changes.

    The binary wheels on PyPI include GDAL 3.5.0 and PROJ 9.0.0, the latest versions of each.

    Deprecations:

    • The fiona.drivers() function has been deprecated and will be removed in version 2.0. It should be replaced by fiona.Env().
    • The new fiona.meta module will be renamed to fiona.drivers in version 2.0.

    Packaging:

    • Source distributions contain no C source files and require Cython to create them from .pyx files (#1096).

    Changes:

    • Shims for various versions of GDAL have been removed and are replaced by Cython compilation conditions (#1093).
    • Use of CURL_CA_BUNDLE environment variable is replaced by a more specific GDAL/PROJ_CURL_CA_BUNDLE (#1095).
    • Fiona's feature accessors now return instances of fiona.model.Feature instead of Python dicts (#787). The Feature class is compatible with code that expects GeoJSON-like dicts but also provides id, geometry, and properties attributes. The last two of these are instances of fiona.model.Geometry and fiona.model.Properties.
    • GDAL 3.1.0 is the minimum GDAL version.
    • Drop Python 2, and establish Python 3.7 as the minimum version (#1079).
    • Remove six and reduce footprint of fiona.compat (#985).

    New features:

    • The appropriate format driver can be detected from filename in write mode (#948).
    • Driver metadata including dataset open and dataset and layer creations options are now exposed through methods of the fiona.meta module (#950).
    • CRS WKT format support (#979).
    • Add 'where' SQL clause to set attribute filter (#961, #1097).

    Bug fixes:

    • Env and Session classes have been updated for parity with rasterio and to resolve a credential refresh bug (#1055).
    Source code(tar.gz)
    Source code(zip)
  • 1.8.21(Feb 7, 2022)

    Changes:

    • Driver mode support tests have been made more general and less susceptible to driver quirks involving feature fields and coordinate values (#1060).
    • OSError is raised on attempts to open a dataset in a Python file object in "a" mode (see #1027).
    • Upgrade attrs, cython, etc to open up Python 3.10 support (#1049).

    Bug fixes:

    • Allow FieldSkipLogFilter to handle exception messages as well as strings (reported in #1035).
    • Clean up VSI files left by MemoryFileBase, resolving #1041.
    • Hard-coded "utf-8" collection encoding added in #423 has been removed (#1057).
    Source code(tar.gz)
    Source code(zip)
  • 1.8.13.post1(Feb 22, 2020)

    This release is being made to improve binary wheel compatibility with shapely 1.7.0. There have been no changes to the fiona package code since 1.8.13.

    Source code(tar.gz)
    Source code(zip)
  • 1.8.0(Oct 31, 2018)

    Fiona 1.8.0 is on PyPI today. Congratulations to all 46 developers (see the credits) file and many thanks to everyone who took the time to report a bug or test a new feature.

    Much of the motivation for this version has been provided by the GeoPandas project. Working with Joris Van den Bossche et al. on various issues has been a pleasure.

    There are no known breaking changes in 1.8.0. Python warnings should be expected in several cases of class and method deprecation.

    • The fiona.drivers() context manager is being replaced by fiona.Env(), which also registers format drivers and has the same properties as the GDAL configuration manager in Rasterio.
    • Collection slicing will be disallowed in a future version of Fiona to remove the confusion between mapping and list semantics for Collection objects. Code such as fiona.open(“example.shp”)[1:10] should be changed to list(fiona.open(“example.shp”))[1:10].

    This version has new features, including a set from the Rasterio project.

    • Fiona has an increased and configurable transaction size for record writes, which makes the GeoPackage format fully usable in Fiona.
    • The “http” and “https” URI schemes for datasets are now fully supported, providing direct access to vector data on the web. Support for an “s3” does the same for data in AWS S3 buckets.
    • Support for “zip”, “zip+https”, and “zip+s3” datasets allows users to access records in zipped Shapefiles (for example) without needing to unzip them, whether on a local file system, on the web, or in S3.
    • New MemoryFile and ZipMemoryFile classes provide easy access to datasets in streams of bytes.

    Major refactoring was required to bring new features over from Rasterio and to modernize our use of Cython. This was a huge lift, largely done by Joshua Arnott. Elliott Sales de Andrade took the lead on finishing the migration of Fiona’s tests to pytest.

    Source code(tar.gz)
    Source code(zip)
Solving the Traveling Salesman Problem using Self-Organizing Maps

Solving the Traveling Salesman Problem using Self-Organizing Maps This repository contains an implementation of a Self Organizing Map that can be used

Diego Vicente 3.1k Dec 31, 2022
An API built to format given addresses using Python and Flask.

An API built to format given addresses using Python and Flask. About The API returns properly formatted data, i.e. removing duplicate fields, distingu

1 Feb 27, 2022
WebGL2 powered geospatial visualization layers

deck.gl | Website WebGL2-powered, highly performant large-scale data visualization deck.gl is designed to simplify high-performance, WebGL-based visua

Vis.gl 10.5k Jan 08, 2023
Stitch image tiles into larger composite TIFs

untiler Utility to take a directory of {z}/{x}/{y}.(jpg|png) tiles, and stitch into a scenetiff (tif w/ exact merc tile bounds). Future versions will

Mapbox 38 Dec 16, 2022
Specification for storing geospatial vector data (point, line, polygon) in Parquet

GeoParquet About This repository defines how to store geospatial vector data (point, lines, polygons) in Apache Parquet, a popular columnar storage fo

Open Geospatial Consortium 449 Dec 27, 2022
Deal with Bing Maps Tiles and Pixels / WGS 84 coordinates conversions, and generate grid Shapefiles

PyBingTiles This is a small toolkit in order to deal with Bing Tiles, used i.e. by Facebook for their Data for Good datasets. Install Clone this repos

Shoichi 1 Dec 08, 2021
gjf: A tool for fixing invalid GeoJSON objects

gjf: A tool for fixing invalid GeoJSON objects The goal of this tool is to make it as easy as possible to fix invalid GeoJSON objects through Python o

Yazeed Almuqwishi 91 Dec 06, 2022
Enable geospatial data mining through Google Earth Engine in Grasshopper 3D, via its most recent Hops component.

AALU_Geo Mining This repository is produced for a masterclass at the Architectural Association Landscape Urbanism programme. Requirements Rhinoceros (

4 Nov 16, 2022
A bot that tweets info and location map for new bicycle parking added to OpenStreetMap within a GeoJSON boundary.

Bike parking tweepy bot app A twitter bot app that searches for bicycle parking added to OpenStreetMap. Relies on AWS Lambda/S3, Python3, Tweepy, Flas

Angelo Trivisonno 1 Dec 19, 2021
Color correction plugin for rasterio

rio-color A rasterio plugin for applying basic color-oriented image operations to geospatial rasters. Goals No heavy dependencies: rio-color is purpos

Mapbox 111 Nov 15, 2022
This program analizes films database with adresses, and creates a folium map with closest films to the coordinates

Films-map-project UCU CS lab 1.2, 1st year This program analizes films database with adresses, and creates a folium map with closest films to the coor

Artem Moskovets 1 Feb 09, 2022
A simple python script that, given a location and a date, uses the Nasa Earth API to show a photo taken by the Landsat 8 satellite. The script must be executed on the command-line.

What does it do? Given a location and a date, it uses the Nasa Earth API to show a photo taken by the Landsat 8 satellite. The script must be executed

Caio 42 Nov 26, 2022
Use Mapbox GL JS to visualize data in a Python Jupyter notebook

Location Data Visualization library for Jupyter Notebooks Library documentation at https://mapbox-mapboxgl-jupyter.readthedocs-hosted.com/en/latest/.

Mapbox 620 Dec 15, 2022
python toolbox for visualizing geographical data and making maps

geoplotlib is a python toolbox for visualizing geographical data and making maps data = read_csv('data/bus.csv') geoplotlib.dot(data) geoplotlib.show(

Andrea Cuttone 976 Dec 11, 2022
r.cfdtools 7 Dec 28, 2022
Create Siege configuration files from Cloud Optimized GeoTIFF.

cogeo-siege Documentation: Source Code: https://github.com/developmentseed/cogeo-siege Description Create siege configuration files from Cloud Optimiz

Development Seed 3 Dec 01, 2022
A Jupyter - Leaflet.js bridge

ipyleaflet A Jupyter / Leaflet bridge enabling interactive maps in the Jupyter notebook. Usage Selecting a basemap for a leaflet map: Loading a geojso

Jupyter Widgets 1.3k Dec 27, 2022
Calculate the area inside of any GeoJSON geometry. This is a port of Mapbox's geojson-area for Python

geojson-area Calculate the area inside of any GeoJSON geometry. This is a port of Mapbox's geojson-area for Python. Installation $ pip install area U

Alireza 87 Dec 14, 2022
A public data repository for datasets created from TransLink GTFS data.

TransLink Spatial Data What: TransLink is the statutory public transit authority for the Metro Vancouver region. This GitHub repository is a collectio

Henry Tang 3 Jan 14, 2022
A proof-of-concept jupyter extension which converts english queries into relevant python code

Text2Code for Jupyter notebook A proof-of-concept jupyter extension which converts english queries into relevant python code. Blog post with more deta

DeepKlarity 2.1k Dec 29, 2022