Hi, Any chance someone can add docker support + sql docker support like in this cookiecutter django project?

The benefits are:

1. reproducible environment for running the code -> easier to deply
2. reproducible database (if needed)

I am new to Docker&Cookie-cutter, otherwise I would do this myself.

Related to #143 #76

Steps to reproduce

• Install Miniconda
• Install Jupyter lab
• Install cookiecutter
• Install nb_conda_kernels
• Create conda environment with conda create -n yourenvname python=3
• Configure environment with

conda activate yourenvname
conda install ...

• Return to base environment with conda deactivate
• Create new project with cookiecutter https://github.com/drivendata/cookiecutter-data-science
• Execute make data
• Confirm success with

make test_environment  
python3 test_environment.py  
Development environment passes all tests!  

• Execute jupyter lab
• In Jupyter Lab, navigate to myproject>notebooks
• From jupyter lab launcher create a new notebook using the conda environment yourenvname
• Execute first cell with

# OPTIONAL: Load the "autoreload" extension so that code can change
%load_ext autoreload

# OPTIONAL: always reload modules so that as you change code in src, it gets loaded
%autoreload 2

from src.data import make_dataset

Error

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-f47036c946b3> in <module>
      5 get_ipython().run_line_magic('autoreload', '2')
      6 
----> 7 from src.data import make_dataset

ModuleNotFoundError: No module named 'src'

The error does not occur when using a python3 notebook instead of a notebook with the environment kernel.

It appears that the project package is not recognized by the notebook because it is not installed in the environment kernel.

I saw the recommendation to install jupyter in the environment, but that seems to be contrary to the design of jupyter lab/notebook and environments. One wants to have one jupyter install in the base environment so one can traverse all projects but still isolate notebooks within kernels.

This issue plus the conversations on #164 #118 #83 suggest that environments are a source of confusion and complexity. It would be nice if there were a way to let the user choose the package manager then reference environment stuff encapsulated by package manager. That would make the maintenance problem easier rather than either trying to do complex conditional logic in the makefile or forcing the user to debug incorrect assumptions about their starting environment.

In the meantime, perhaps someone could suggest a command to import src into an existing environment.

Anaconda has a yml-based specification file for indicating dependencies. We should use this instead of pip if we are using conda to manage virtual environments

• Add a configuration-time question to choose between conda and virtualenv explicitly
• Use an environment.yml to specify the conda dependencies
• Install dependencies at venv-creation time, since we already have this info

As an added bonus, this makes it a little easier to add pipenv support by extending the VIRTUALENV choices

I just started using this cookiecutter and I'm wondering how people are using this directory structure in order to generate figures and reports.

Here's what I'm doing currently:

• do analysis and generate interesting figure, save them to /reports/figures/
• write up final jupyter notebook report from within /notebooks/reports/, any reference to figures are going to be ../../reports/figures/fig.png
• export the report as report.html and place in /reports/

The issue now, is that when I view the report.html, the figures don't have the proper path. How are people getting around this?

Hi there! Loved the project, this really reflects the maturity of data science projects and where we are standing. So good!

I rise this issue as I was wondering if the current structure can be adapted to an api-driven project. This is, a project in which the analysis and data flow may be related to an api definition.

If yes, what would it be? So we can document it (or point me out where it is) If not, why? Some books have recommended having an api flow for analysis and process so our results and analysis are available for our mates in engineering. Even allowing for an easy scale up.

Thank you so much!

We've seen a lot of potential features for this where we need to handle forking paths gracefully. By default cookiecutter can't do this (see https://github.com/audreyr/cookiecutter/pull/848). It's been years, so we can't reasonably expect this to change upstream...

This implements a monkey-patching workaround to enable this behavior. It introduces a couple of major changes, so here are my recommendations.

Here are the big differences for a user:

• we now need to run ccds <path to repo> instead of cookiecutter <path to repo>
• the options and their defaults are changing
• as a consequence of supporting more environments/dependency managers at setup time, we stop supporting them at execution time. this means that teams will have to pick one of each and stick to it across developers. I think this is pretty widespread already. (i.e. make create_environment will only support one of the options rather than multiple like it does now)

Implementation details are:

• Add setup.py to make this package installable and give it a CLI
• add monkey_patch.py to patch the cookiecutter functions that we need to handle our use case
• support a list of dictionaries in cookiecutter.json that let's a use pick an option and then sub-options
• use post_gen_project.py to create the environment file based on a standard list of libraries. (#5)

There is also work for a number of longstanding items in this branch as well:

• we now support multiple storage providers (#120)
• we allow user to choose package manager (#118)
• we allow user to choose a dependency format
• src is now {{ cookiecutter.module_name }} (#140)
• remove test_environment.py which is just cruft IMO

Done:

• [x] tag current master as v1 so anyone relying on the current flow/structure can continue to use it easily
• [x] implement the rails for the high priority items
• [x] make comprehensive tests run on CI/CD to de-risk
• [x] support multiple dependency formats (environment.yml, requirements.txt, pipenv)
• [x] default pydata dependencies options
• [x] add support for azure/google cloud
• [x] add ccds command and make cookiecutter-data-science a proper package

Remaining items

Cookiecutter default structure

• [ ] user supplied config files
• [ ] revise generated python package boilerplate (make optional)
• [ ] mkdocs in place of sphinx
• [ ] add lint command to Makefile

Cookiecutter options

• [ ] add deon - WIP #244
• [ ] add nbautoexport - WIP #244
• [ ] formatting commands / configuration (e.g., black)
• [ ] test suite configuration (e.g., pytest install and make commands)
• [ ] add

Infrastructure

• [ ] tests passing on windows
• [ ] release command and make PyPI release

Docs

• [ ] add documentation for new installation (pip install cookiecutter-data-science) and new initiation (ccds <path to repo>)
• [ ] add table of options with links to the project documentation in our docs
• [ ] update screencast and add screenshots ( #197 ) of the new flow
• [ ] add options for

Part of our Data Days for Good initiative was to work on this repo. One of the requested additions was fleshing out the setup tests a little bit more to test for what would happen if user input was introduced. This was mentioned by Isaac. I have separated out the fixture and put it in it's own file, which is discoverable by pytest and mentioned in the docs. I have also increased the scope of creating the temp directory to be a class scope so it doesn't have to create it and tear it down after every function call. Lastly I also test for certain conditions given user input.

Adding DVC support can benefit this framework in two ways:

1. Provide version control backend for data files - including intermediate results, models, etc. As you know git is not the best solution for storing data files. Storing and versioning data is crucial for reproducibility.
2. Serve as an easier (to learn) alternative to makefiles. DVC allows for tracking execution steps, dependencies between them, etc. All from CLI, no need to learn a new language (make).

I think both projects can benefit and get more visibility from this. Let me know what you think.

How to reproduce:

1. cookiecutter https://github.com/drivendata/cookiecutter-data-science
2. Give it your bucket name
3. Look at line 8 of the Makefile - there's your bucket name in plaintext.

:-( 👎

Hi,

as the data folders are not included in the Github repository, they are not there when the project is cloned. But the Makefile contains commands related to /data (and scripts could). I included a rule for creating the missing folders.

Besides Cookiecutter there is also PyScaffold which provides a standardised project scaffold for Python. Since version 3.0 PyScaffold also has an extension system and thus it would be possible to provide this cool template also easily for PyScaffold.

I see several advantages over Cookiecutter:

• Currently src seems to be a Python package which is not a good practice if you want to combine several Data Science projects which use this template. One would end up with packages overwriting each others.
• PyScaffold is best for having a standard-compliant Python project scaffold. One could thus focus on providing additional folders and structure within this scaffold.
• One would have a modern, descriptive way of configuration with the help of setup.cfg instead of setup.py.
• PyScaffold would also allow to combine a Data Science structure with other extensions for instance having Tox support for unit testing, precommit for automised tasks etc.
• PyScaffold allows updating scaffolds which is important in the fast changing world of Python packaging.

Small change to allow gsutil to do rsync recursively, syncing all the folders.

Context: We had some issues using it with gcloud, since make sync_data_up would not upload all the data folders. Adding the -r flag fixes this issue.

https://github.com/drivendata/cookiecutter-data-science/blob/b4c0c12243653c493c188239af14c835b9768fbc/%7B%7B%20cookiecutter.repo_name%20%7D%7D/Makefile#L62

In v2, sync_data_up incorrectly lists the bucket as the source directory, rather than the local data/ folder. The order should be filpped.

In addition, there is inconsistent use of environment variables. sync_data_down uses the templatized AWS profile from the cookiecutter form:

https://github.com/drivendata/cookiecutter-data-science/blob/b4c0c12243653c493c188239af14c835b9768fbc/%7B%7B%20cookiecutter.repo_name%20%7D%7D/Makefile#L51

while sync_data_up uses the PROFILE environment variable which is not set in the Makefile.

https://github.com/drivendata/cookiecutter-data-science/blob/b4c0c12243653c493c188239af14c835b9768fbc/%7B%7B%20cookiecutter.repo_name%20%7D%7D/Makefile#L63

This PR allows a ccds user to select mkdocs, sphinx, or none as their default code documentation tool.

If sphinx is chosen, provides the output of sphinx-quickstart in /docs/ (this is the current behavior)

If mkdocs is chosen, provides the output of mkdocs new {{ project_name }} in /docs/

If none is chosen, provides no default documentation tool, but keeps an empty /docs/ folder.

Notes

Adding tests for this brings the number of configurations to 60. With the number of possible configurations continuing to grow exponentially, we may want to consider how we generate test configs and what is appropriate.

Attempting to run a build on the latest v2 branch yields the following:

option --use-feature: invalid choice: '2020-resolver' (choose from 'fast-deps', 'truststore', 'no-binary-enable-wheel-cache')

Pip 22.3 removes that option since this resolver is now default.

Closes #49

Provides additional prompt to make boilerplate python module files and folders disappear, with the exception of __init__.py to make the module a python module.

The current workflow assumes that the user has not created a directory for the project and does not already have a repo for it.

I think I'm not alone in preferring to create the repo on GitHub, clone it, and then run ccds. I would be good to support that.

Or at least provide instructions for how to work with an existing repo.