NOTE: I closed this PR by mistake. I'll re-open this. This pull request is for catering to this [issue](Allow consuming JSON data #1334). Currently, we support 2 kinds of json datasets:

1. Where the dataset's rows are present in a certain key of json. For example refer to this example, here the certain_key is data.
2. Where the dataset's rows are present in certain key of differnent parts of the json. For example refer to this example, here the certain key is laureates.
3. [WIP] Where the json is in the form of list, example. Current implementation for this is commented out, since its getting stuck in a recursion loop.

Updated internal variable names to match that of datapackage spec #765 The following changes were done for the variable names

tags -> keywords
nulls -> missingValues
name -> title
shortname -> name

The changes were done in the following files -

retriever/lib/compile.py
retriever/lib/datapackage.py
retriever/lib/engine.py
retriever/lib/parse_script_to_json.py
retriever/lib/templates.py
retriever/lib/tools.py
scripts/bioclim.py
scripts/biomass_allometry_db.py
scripts/breed_bird_survey.py
scripts/breed_bird_survey_50stop.py
scripts/forest_inventory_analysis.py
scripts/gentry_forest_transects.py
scripts/npn.py
scripts/plant_life_hist_eu.py
scripts/prism_climate.py
scripts/vertnet.py
scripts/wood_density.py
scripts/*.json(almost all datapackages) transition missingValues -> missing_values
test/test_retriever.py
retriever/__main__.py

@henrykironde I have made the changes after updating it with the master branch

I have started to look into and working on the project : Expansion of Spatial Data Support to the Data Retriever.

@ethanwhite @henrykironde Please let me know of any aspects in particular I should be prioritising over others.

Would it be okay if I carried on the discussion through this Issue?

I'm in the process of trying to get the EcoData Retriever fully functional on OSX (since so many of the awesome ecological informaticsy people I know use Macs). As I've mentioned elsewhere it looks like building from source now works, at least when using homebrew (http://ecodataretriever.org/getting_started.html).

What I'm working on now is getting the .app working so that you don't need to be comfortable in the shell (and have XCode installed) to use the Retriever. I have a version that is working on the machine that I built it on (our only Mac) and was wondering if some kind Mac folks like @karthik, @sckott, @emhart, @sarahsupp and @dfalster might have a few minutes to give it a trial run.

The file is here: https://www.dropbox.com/s/26b1pj91mqucc0l/retriever.zip

Basically I'm just looking for folks to unzip it, double click on it, and see if:

1. It opens at all.
2. You can install things successfully when setting the database management system to CSV and sqlite (these don't have any external dependencies).
3. If you have either MySQL or PostgreSQL installed if it works with them. (MySQL is a bit fragile at the moment. It is currently working for most datasets, but not all, so just try a few if you get errors).
4. Report back.

Thanks in advance. And, yes, I wrote this issue... on a Mac.

Currently we only support ingesting delimited tabular data. It is increasingly common for tabular style data to be distributed in JSON files and it would be nice to also be able to consume this. We would probably just convert it to CSV as a starting point and then process it using our standard pipeline.

There are a few not particularly active packages for doing this, but the code to do it is so simple enough that since none of the packages seem to be widely adopted we might be better off just writing and maintaining it ourselves.

(no rush on this, just a thought while looking at a cool dataset that's only available in JSON: http://data.unpaywall.org/products/snapshot)

@ethanwhite @henrykironde I have made the following changes

• name -> title
• shortname -> name
• tags -> keywords
• nulls -> missingValues

I wanted to ask to change missingValues to missing_values as the original one seems to be in camel case and not in pep8 naming convention , if you allow I would change it. I am still finding such internal name cases which could be updated. I guess the code is clean. Kindly go through it once and if there are any suggestion I would start working on them

Sir I have removed the hard coded assignments for encoding and added a field of encoding. I haven't edited the existing scripts for the existing data. Should I do them manually? Kindly review the code and tell me any changes needed for the code. @ethanwhite @henrykironde

Hey all,

I've mostly gotten the eBird data into a PostgreSQL/PostGIS database, and I thought I'd share my code with you in case you wanted to integrate it into something more robust with EcoDataRetriever. If you know how to optimize it better, I'd love to hear what you come up with.

If you do decide to include it, please acknowledge Matt Jones and Jim Regetz, since they helped me through this.

Let me know if you have any questions!

Dave

PS the "world" data set unzips to be 50 gigabytes, so you'll probably want to work with something smaller...

-- Data file available via http://ebird.org/ebird/data/download

-- commands to extract the text file from the tarball:
   -- tar xvf ebd_relMay-2013.tar
   -- gunzip ebd_relMay-2013.txt.gz
-- WARNING: The resulting file is almost 50 gigabytes!

-- In retrospect, there's probably some premature optimization for some of these columns: if the data set changes,
-- it might be safer to use longer varchar arguments.
CREATE TABLE eBird (
  GLOBAL_UNIQUE_IDENTIFIER     char(50),      -- always 45-47 characters needed (so far)
  TAXONOMIC_ORDER              numeric,       -- Probably not needed
  CATEGORY                     varchar(20),   -- Probably 10 would be safe
  COMMON_NAME                  varchar(70),   -- Some hybrids have really long names
  SCIENTIFIC_NAME              varchar(70),   --  ''
  SUBSPECIES_COMMON_NAME       varchar(70),   --  ''
  SUBSPECIES_SCIENTIFIC_NAME   varchar(70),   --  ''
  OBSERVATION_COUNT            varchar(8),    -- Someone saw 1.3 million Auklets.
                                              -- Unfortunately, it can't be an integer 
                                              -- because some are just presence/absence
  BREEDING_BIRD_ATLAS_CODE     char(2),       -- need to confirm that these are always length 2
  AGE_SEX                      text,          -- Potentially long, but almost always blank
  COUNTRY                      varchar(50),   -- long enough for "Saint Helena, Ascension and Tristan da Cunha"
  COUNTRY_CODE                 char(2),       -- alpha-2 codes
  STATE_PROVINCE               varchar(50),   -- no idea if this is long enough? U.S. Virgin Islands may be almost 30
  SUBNATIONAL1_CODE            char(10),      -- looks standardized at 5 characters?
  COUNTY                       varchar(50),   -- who knows how long it could be
  SUBNATIONAL2_CODE            char(12),      -- looks standardized at 9 characters?
  IBA_CODE                     char(16),
  LOCALITY                     text,          -- unstructured/potentially long
  LOCALITY_ID                  char(10),      -- maximum observed so far is 8
  LOCALITY_TYPE                char(2),       -- short codes
  LATITUDE                     real,          -- Is this the appropriate level of precision?
  LONGITUDE                    real,          --    ''
  OBSERVATION_DATE             date,          -- Do I need to specify YMD somehow?
  TIME_OBSERVATIONS_STARTED    time,          -- How do I make this a time?
  TRIP_COMMENTS                text,          -- Comments are long, unstructured, 
  SPECIES_COMMENTS             text,          --    and inconsistent, but sometimes interesting
  OBSERVER_ID                  char(12),      -- max of 9 in the data I've seen so far
  FIRST_NAME                   text,          -- Already have observer IDs
  LAST_NAME                    text,          -- ''
  SAMPLING_EVENT_IDENTIFIER    char(12),      -- Probably want to index on this.
  PROTOCOL_TYPE                varchar(50),   -- Needs to be at least 30 for sure.
  PROJECT_CODE                 varchar(20),   -- Needs to be at least 10 for sure.
  DURATION_MINUTES             int,           -- bigint?
  EFFORT_DISTANCE_KM           real,          -- precision?
  EFFORT_AREA_HA               real,          -- precision?
  NUMBER_OBSERVERS             int,           -- just a small int
  ALL_SPECIES_REPORTED         int,           -- Seems to always be 1 or 0.  Maybe I could make this Boolean?
  GROUP_IDENTIFIER             varchar(10),   -- Appears to be max of 7 or 8
  APPROVED                     int,           -- Can be Boolean?
  REVIEWED                     int,           -- Can be Boolean?
  REASON                       char(17),      -- May need to be longer if data set includes unvetted data
  X                            text           -- Blank
);


COPY eBird
  FROM '/home/dharris/eBird/ebd_relMay-2013.txt' 
  HEADER
  CSV
  QUOTE E'\5'       -- The file has unbalanced quotes. Using an obscure character as a quote mark instead.
  DELIMITER E'\t';


-- Note: it's probably slightly faster to load postgis and add a geographic column first (see below).
-- I'm keeping the original ordering in this document for accuracy's sake.
CREATE INDEX ON eBird (sampling_event_identifier)

-- Example query: SELECT SCIENTIFIC_NAME FROM eBird WHERE SAMPLING_EVENT_IDENTIFIER = 'S9605852';
-- Example query: SELECT count(SCIENTIFIC_NAME) FROM eBird WHERE SAMPLING_EVENT_IDENTIFIER = 'S9605852';


CREATE EXTENSION postgis;
ALTER TABLE eBird ADD COLUMN geog geography(POINT,4326); -- I hope 4326 is correct...
UPDATE eBird SET geog = ST_GeogFromText('POINT(' || longitude || ' ' ||  latitude || ')');
CREATE INDEX geog_index ON eBird USING GIST (geog); 

-- Example query: find all the species within 1000 of my dorm:
-- SELECT SCIENTIFIC_NAME FROM eBird WHERE ST_DWithin(geog, ST_GeographyFromText('SRID=4326;POINT(-119.6972 34.4208)'), 1000);

-- Slightly fancier version:
-- SELECT DISTINCT SCIENTIFIC_NAME, COMMON_NAME FROM eBird 
--   WHERE ST_DWithin(geog, ST_GeographyFromText('SRID=4326;POINT(-119.855385 34.417239)'), 1000) 
--   ORDER BY SCIENTIFIC_NAME;

(Edited to add some amazing PostGIS queries and some better commets, etc.)

PS: After poking around a bit more, it looks like I should have used doubles rather than reals to store lat/lon. I had misread the documentation about how much precision was used for reals.

It is not uncommon for a data source to go down (e.g. #902) or for a download to fail for some reason (e.g., #863). We should catch these, not cache the data that comes down (which is sometimes a corrupt file and sometimes a 404 html page), and report to the user that the source appears to be down and that they should try again and if it still fails later let us know.

Updated internal variable names to match that of datapackage spec #765 The following changes were done for the variable names

tags -> keywords
nulls -> missingValues
name -> title
shortname -> name

The changes were done in the following files -

retriever/lib/compile.py
retriever/lib/datapackage.py
retriever/lib/engine.py
retriever/lib/parse_script_to_json.py
retriever/lib/templates.py
retriever/lib/tools.py
scripts/bioclim.py
scripts/biomass_allometry_db.py
scripts/breed_bird_survey.py
scripts/breed_bird_survey_50stop.py
scripts/forest_inventory_analysis.py
scripts/gentry_forest_transects.py
scripts/npn.py
scripts/plant_life_hist_eu.py
scripts/prism_climate.py
scripts/vertnet.py
scripts/wood_density.py
scripts/*.json(almost all datapackages) transition missingValues -> missing_values
test/test_retriever.py
retriever/__main__.py

Hi @henrykironde Sorry, I was off schedule last days so I couldn't work on this issue as I told you. This should solve #1019 but is this the right place for the method?

For contributors who want to take part in the hacktoberfest, please check the issue lists from the various projects

Retriever: https://github.com/weecology/retriever/issues Retriever-recipes: https://github.com/weecology/retriever-recipes/issues Rdataretriever: https://github.com/ropensci/rdataretriever/issues Retriever.jl: https://github.com/weecology/Retriever.jl/issues

Example packages:
1: Package file: https://github.com/weecology/retriever-recipes/blob/main/scripts/usda_agriculture_plants_database.py Sample url: https://plants.sc.egov.usda.gov/csvdownload?plantLst=plantCompleteList

2: package file: https://github.com/weecology/retriever-recipes/blob/main/scripts/aquatic_animal_excretion.py url: https://esajournals.onlinelibrary.wiley.com/action/downloadSupplement?doi=10.1002%2Fecy.1792&file=ecy1792-sup-0001-DataS1.zip

display_all_rdatasets_names takes list of package_name insted of taking a string of package_name as a parameter

>>> display_all_rdataset_names("aer")
List of all available Rdatasets in packages: aer
No package named 'a' found in Rdatasets
No package named 'e' found in Rdatasets
No package named 'r' found in Rdatasets

>>> display_all_rdataset_names(["aer"])
List of all available Rdatasets in packages: ['aer']
Package: aer              Dataset: affairs                   Script Name: rdataset-aer-affairs
Package: aer              Dataset: argentinacpi              Script Name: rdataset-aer-argentinacpi
Package: aer              Dataset: bankwages                 Script Name: rdataset-aer-bankwages
Package: aer              Dataset: benderlyzwick             Script Name: rdataset-aer-benderlyzwick
Package: aer              Dataset: bondyield                 Script Name: rdataset-aer-bondyield
Package: aer              Dataset: cartelstability           Script Name: rdataset-aer-cartelstability
Package: aer              Dataset: caschools                 Script Name: rdataset-aer-caschools
Package: aer              Dataset: chinaincome               Script Name: rdataset-aer-chinaincome
Package: aer              Dataset: cigarettesb               Script Name: rdataset-aer-cigarettesb
Package: aer              Dataset: cigarettessw              Script Name: rdataset-aer-cigarettessw
Package: aer              Dataset: collegedistance           Script Name: rdataset-aer-collegedistance
Package: aer              Dataset: consumergood              Script Name: rdataset-aer-consumergood
Package: aer              Dataset: cps1985                   Script Name: rdataset-aer-cps1985
Package: aer              Dataset: cps1988                   Script Name: rdataset-aer-cps1988
....

NOTES

Expected behavior and actual behavior.

While I am having gdal 3.2.2, if I try to import ogr in a script dealing with ".shp" files, it doesn't import, but if I downgrade my gdal to 3.0.2 I'm able to import ogr and script run successfully

ogr_not_working

ogr_working

Operating system

Ubuntu 20.04 bit

GDAL version and provenance

GDAL 3.3.2 version from ubuntugis-unstable PPA

We have added some API to the retriever. Some of the APIs, like Tidycensus, can be run and tested on the retriever dashboard.

You can clone the retrieverdash project and test locally using the developer docs for the dashboard https://retrieverdash.readthedocs.io/developer.html#setting-up-locally.

When working locally, first you will need to have the APIs working well on the retriever. Use the DEV LIST in the retriever dashboard to test only the required scripts.