Zipline uses IEX API to get benchmark data in benchmarks.py:

def get_benchmark_returns(symbol):
    """
    Get a Series of benchmark returns from IEX associated with `symbol`.
    Default is `SPY`.

    Parameters
    ----------
    symbol : str
        Benchmark symbol for which we're getting the returns.

    The data is provided by IEX (https://iextrading.com/), and we can
    get up to 5 years worth of data.
    """
    r = requests.get(
        'https://api.iextrading.com/1.0/stock/{}/chart/5y'.format(symbol)
    )
    data = r.json()

    df = pd.DataFrame(data)

    df.index = pd.DatetimeIndex(df['date'])
    df = df['close']

    return df.sort_index().tz_localize('UTC').pct_change(1).iloc[1:]

However, according to the IEX FAQ page, the chart api was already removed on June 15, 2019. Currently, using this api to try to download any stock data such as SPY will return nothing but an HTTP 403 error. The functions of deprecated APIs are now transferred to their new API, IEX Cloud, which requires a unique token per user in any request. Any idea how to fix this issue in the long run?

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

• Operating System: Linux hostname 2.6.18-238.12.1.el5 #1 SMP Tue May 31 13:22:04 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
• Python Version: Python 3.5.1
• Python Bitness: 32
• How did you install Zipline: pip
• Python packages:

alembic==0.9.1 appdirs==1.4.3 bcolz==0.12.1 Bottleneck==1.3.0.dev0 click==6.7 contextlib2==0.5.5 cyordereddict==1.0.0 Cython==0.25.2 decorator==4.0.11 empyrical==0.2.2 intervaltree==2.1.0 Logbook==1.0.0 lru-dict==1.1.6 Mako==1.0.6 MarkupSafe==1.0 multipledispatch==0.4.9 networkx==1.11 numexpr==2.6.2 numpy==1.12.1 packaging==16.8 pandas==0.18.1 pandas-datareader==0.3.0.post0 patsy==0.4.1 pyparsing==2.2.0 python-dateutil==2.6.0 python-editor==1.0.3 pytz==2017.2 requests==2.13.0 requests-file==1.4.2 requests-ftp==0.3.1 scipy==0.19.0 setuptools-scm==1.15.5 six==1.10.0 sortedcontainers==1.5.7 SQLAlchemy==1.1.9 statsmodels==0.8.0 tables==3.4.2 toolz==0.8.2 zipline==1.1.0

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

I just ran the buyapple.py example that described on http://www.zipline.io/beginner-tutorial.html, but got some errors, can you help to check ?

/home/kevinyuan/dev/zipline> bin/zipline ingest Downloading Bundle: quantopian-quandl [####################################] 100% Writing data to /home/kevinyuan/.zipline/data/quantopian-quandl/2017-05-01T16;36;49.048566.

/home/kevinyuan/dev/zipline> bin/zipline run -f examples/buyapple.py -s 2000-1-1 -e 2001-12-31 [2017-05-01 16:33:28.023652] INFO: Loader: Cache at /home/kevinyuan/.zipline/data/^GSPC_benchmark.csv does not have data from 1990-01-02 00:00:00+00:00 to 2017-04-27 00:00:00+00:00. Downloading benchmark data for '^GSPC'. [2017-05-01 16:33:28.076307] ERROR: Loader: failed to cache the new benchmark returns Traceback (most recent call last): File "/share/dev/tools/lib/python3.5/urllib/request.py", line 1240, in do_open h.request(req.get_method(), req.selector, req.data, headers) File "/share/dev/tools/lib/python3.5/http/client.py", line 1083, in request self.send_request(method, url, body, headers) File "/share/dev/tools/lib/python3.5/http/client.py", line 1128, in send_request self.endheaders(body) File "/share/dev/tools/lib/python3.5/http/client.py", line 1079, in endheaders self._send_output(message_body) File "/share/dev/tools/lib/python3.5/http/client.py", line 911, in _sendoutput self.send(msg) File "/share/dev/tools/lib/python3.5/http/client.py", line 854, in send self.connect() File "/share/dev/tools/lib/python3.5/http/client.py", line 1237, in connect serverhostname=server_hostname) File "/share/dev/tools/lib/python3.5/ssl.py", line 376, in wrap_socket context=self) File "/share/dev/tools/lib/python3.5/ssl.py", line 747, in _init self.do_handshake() File "/share/dev/tools/lib/python3.5/ssl.py", line 983, in do_handshake self.sslobj.do_handshake() File "/share/dev/tools/lib/python3.5/ssl.py", line 628, in do_handshake self._sslobj.dohandshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home-tahoe-n2/kevinyuan/dev/zipline/lib/python3.5/site-packages/zipline/data/loader.py", line 247, in ensure_benchmark_data last_date, File "/home-tahoe-n2/kevinyuan/dev/zipline/lib/python3.5/site-packages/zipline/data/benchmarks.py", line 59, in get_benchmark_returns squeeze=True, # squeeze tells pandas to make this a Series File "/home-tahoe-n2/kevinyuan/dev/zipline/lib/python3.5/site-packages/pandas/io/parsers.py", line 562, in parser_f return read(filepath_or_buffer, kwds) File "/home-tahoe-n2/kevinyuan/dev/zipline/lib/python3.5/site-packages/pandas/io/parsers.py", line 301, in read compression=kwds.get('compression', None)) File "/home-tahoe-n2/kevinyuan/dev/zipline/lib/python3.5/site-packages/pandas/io/common.py", line 308, in get_filepath_or_buffer req = urlopen(str(filepath_or_buffer)) File "/share/dev/tools/lib/python3.5/urllib/request.py", line 162, in urlopen return opener.open(url, data, timeout) File "/share/dev/tools/lib/python3.5/urllib/request.py", line 465, in open response = self.open(req, data) File "/share/dev/tools/lib/python3.5/urllib/request.py", line 483, in _open '_open', req) File "/share/dev/tools/lib/python3.5/urllib/request.py", line 443, in _callchain result = func(*args) File "/share/dev/tools/lib/python3.5/urllib/request.py", line 1283, in https_open context=self.context, check_hostname=self.check_hostname) File "/share/dev/tools/lib/python3.5/urllib/request.py", line 1242, in doopen raise URLError(err) urllib.error.URLError: <urlopen error [SSL: CERTIFICATEVERIFY_FAILED] certificate verify failed (_ssl.c:645)> Traceback (most recent call last): File "/share/dev/tools/lib/python3.5/urllib/request.py", line 1240, in doopen h.request(req.get_method(), req.selector, req.data, headers) File "/share/dev/tools/lib/python3.5/http/client.py", line 1083, in request self.send_request(method, url, body, headers) File "/share/dev/tools/lib/python3.5/http/client.py", line 1128, in send_request self.endheaders(body) File "/share/dev/tools/lib/python3.5/http/client.py", line 1079, in endheaders self._send_output(message_body) File "/share/dev/tools/lib/python3.5/http/client.py", line 911, in _sendoutput self.send(msg) File "/share/dev/tools/lib/python3.5/http/client.py", line 854, in send self.connect() File "/share/dev/tools/lib/python3.5/http/client.py", line 1237, in connect serverhostname=server_hostname) File "/share/dev/tools/lib/python3.5/ssl.py", line 376, in wrap_socket context=self) File "/share/dev/tools/lib/python3.5/ssl.py", line 747, in _init self.do_handshake() File "/share/dev/tools/lib/python3.5/ssl.py", line 983, in do_handshake self.sslobj.do_handshake() File "/share/dev/tools/lib/python3.5/ssl.py", line 628, in do_handshake self._sslobj.dohandshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "bin/zipline", line 11, in sys.exit(main()) File "/home-tahoe-n2/kevinyuan/dev/zipline/lib/python3.5/site-packages/click/core.py", line 722, in call return self.main(args, **kwargs) File "/home-tahoe-n2/kevinyuan/dev/zipline/lib/python3.5/site-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/home-tahoe-n2/kevinyuan/dev/zipline/lib/python3.5/site-packages/click/core.py", line 1066, in invoke return process_result(sub_ctx.command.invoke(sub_ctx)) File "/home-tahoe-n2/kevinyuan/dev/zipline/lib/python3.5/site-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, *ctx.params) File "/home-tahoe-n2/kevinyuan/dev/zipline/lib/python3.5/site-packages/click/core.py", line 535, in invoke return callback(args, **kwargs) File "/home-tahoe-n2/kevinyuan/dev/zipline/lib/python3.5/site-packages/zipline/main.py", line 97, in _ return f(args, **kwargs) File "/home-tahoe-n2/kevinyuan/dev/zipline/lib/python3.5/site-packages/click/decorators.py", line 17, in new_func return f(get_current_context(), *args, *kwargs) File "/home-tahoe-n2/kevinyuan/dev/zipline/lib/python3.5/site-packages/zipline/main.py", line 240, in run environ=os.environ, File "/home-tahoe-n2/kevinyuan/dev/zipline/lib/python3.5/site-packages/zipline/utils/run_algo.py", line 132, in run env = TradingEnvironment(asset_db_path=connstr) File "/home-tahoe-n2/kevinyuan/dev/zipline/lib/python3.5/site-packages/zipline/finance/trading.py", line 101, in init self.bm_symbol, File "/home-tahoe-n2/kevinyuan/dev/zipline/lib/python3.5/site-packages/zipline/data/loader.py", line 164, in load_market_data trading_day, File "/home-tahoe-n2/kevinyuan/dev/zipline/lib/python3.5/site-packages/zipline/data/loader.py", line 247, in ensure_benchmark_data last_date, File "/home-tahoe-n2/kevinyuan/dev/zipline/lib/python3.5/site-packages/zipline/data/benchmarks.py", line 59, in get_benchmark_returns squeeze=True, # squeeze tells pandas to make this a Series File "/home-tahoe-n2/kevinyuan/dev/zipline/lib/python3.5/site-packages/pandas/io/parsers.py", line 562, in parser_f return read(filepath_or_buffer, kwds) File "/home-tahoe-n2/kevinyuan/dev/zipline/lib/python3.5/site-packages/pandas/io/parsers.py", line 301, in _read compression=kwds.get('compression', None)) File "/home-tahoe-n2/kevinyuan/dev/zipline/lib/python3.5/site-packages/pandas/io/common.py", line 308, in get_filepath_or_buffer req = _urlopen(str(filepath_or_buffer)) File "/share/dev/tools/lib/python3.5/urllib/request.py", line 162, in urlopen return opener.open(url, data, timeout) File "/share/dev/tools/lib/python3.5/urllib/request.py", line 465, in open response = self.open(req, data) File "/share/dev/tools/lib/python3.5/urllib/request.py", line 483, in open '_open', req) File "/share/dev/tools/lib/python3.5/urllib/request.py", line 443, in _callchain result = func(args) File "/share/dev/tools/lib/python3.5/urllib/request.py", line 1283, in https_open context=self.context, check_hostname=self.check_hostname) File "/share/dev/tools/lib/python3.5/urllib/request.py", line 1242, in doopen raise URLError(err) urllib.error.URLError: <urlopen error [SSL: CERTIFICATEVERIFY_FAILED] certificate verify failed (_ssl.c:645)>

running buyapple.py fails in python 2.7.10 but runs in python 3.4.3 for zipline 8.0rc. The part of code that fails is data[symbol('AAPL')] found in

record(AAPL=data[symbol('AAPL')].price)

the error is

Traceback (most recent call last):
  File "C:\python2710_64\Scripts\run_algo.py", line 4, in <module>
    __import__('pkg_resources').run_script('zipline==0.8.0rc1', 'run_algo.py')
  File "C:\Python2710_64\lib\site-packages\pkg_resources\__init__.py", line 729, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "C:\Python2710_64\lib\site-packages\pkg_resources\__init__.py", line 1642, in run_script
    exec(code, namespace, namespace)
  File "c:\python2710_64\lib\site-packages\zipline-0.8.0rc1-py2.7-win-amd64.egg\EGG-INFO\scripts\run_algo.py", line 23, in <module>
    run_pipeline(print_algo=True, **parsed)
  File "C:\Python2710_64\lib\site-packages\zipline-0.8.0rc1-py2.7-win-amd64.egg\zipline\utils\cli.py", line 246, in run_pipeline
    perf = algo.run(source, overwrite_sim_params=overwrite_sim_params)
  File "C:\Python2710_64\lib\site-packages\zipline-0.8.0rc1-py2.7-win-amd64.egg\zipline\algorithm.py", line 529, in run
    for perf in self.gen:
  File "C:\Python2710_64\lib\site-packages\zipline-0.8.0rc1-py2.7-win-amd64.egg\zipline\gens\tradesimulation.py", line 120, in transform
    self.algo.instant_fill,
  File "C:\Python2710_64\lib\site-packages\zipline-0.8.0rc1-py2.7-win-amd64.egg\zipline\gens\tradesimulation.py", line 314, in _process_snapshot
    new_orders = self._call_handle_data()
  File "C:\Python2710_64\lib\site-packages\zipline-0.8.0rc1-py2.7-win-amd64.egg\zipline\gens\tradesimulation.py", line 343, in _call_handle_data
    self.simulation_dt,
  File "C:\Python2710_64\lib\site-packages\zipline-0.8.0rc1-py2.7-win-amd64.egg\zipline\utils\events.py", line 194, in handle_data
    event.handle_data(context, data, dt)
  File "C:\Python2710_64\lib\site-packages\zipline-0.8.0rc1-py2.7-win-amd64.egg\zipline\utils\events.py", line 212, in handle_data
    self.callback(context, data)
  File "C:\Python2710_64\lib\site-packages\zipline-0.8.0rc1-py2.7-win-amd64.egg\zipline\algorithm.py", line 306, in handle_data
    self._handle_data(self, data)
  File "buyapple.py", line 26, in handle_data
    record(AAPL=data[symbol('AAPL')].price)
  File "C:\Python2710_64\lib\site-packages\zipline-0.8.0rc1-py2.7-win-amd64.egg\zipline\protocol.py", line 518, in __getitem__
    return self._data[name]
KeyError: Equity(0, symbol='AAPL', asset_name='', exchange='', start_date=None, end_date=None, first_traded=None)

Environment: Windows 7 64 bit, Python 3.4, Python 2.7, zipline 0.8.0rc Appears to be with zipline.protocol.BarData and zipline.assets._assets.Equity

Kind regards

David Bieber

This PR adds an estimates quarter loader to be used with Pipeline.

Some open questions are:

• What other tests do we need to add?
• How do we ensure that the num_quarters attribute has been added to the dynamically generated dataset by the time load_adjusted_array is called?

Fix benchmark downloading from Google with pandas-datareader. This issue was originally brought up here.

We now get benchmark data from Google instead of Yahoo, as seen here.

However, it appears that as of only a week or two ago, Google changed the URL from which they are serving their financial data, causing pandas datareader to break. This is also preventing us from rebuilding the test_examples data. (For more info see the original post above).

• Create a restricted list manager that takes in information about restricted sids and stores in memory upon instantiation
• Create a restrictions controller that aggregates and provides restrictions information from multiple restricted list managers
• Make this restrictions controller available to BarData so that can_trade can take into account these restrictions
• Register each restricted list manager as a trading control using set_do_not_order_list so that we may log/fail when we try to order a restricted sid

Dear Zipline Maintainers,

Environment

• Operating System: Linux 3d62a3c1924c 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
• Python Version: Python 3.5.2 :: Anaconda 4.3.0 (64-bit)
• Python Bitness: 64
• How did you install Zipline: pip
• Python packages:

alabaster==0.7.9
alembic==0.9.2
anaconda-client==1.6.0
anaconda-navigator==1.4.3
appdirs==1.4.3
APScheduler==3.3.1
arctic==1.38.0
argcomplete==1.0.0
astroid==1.4.9
astropy==1.3
Babel==2.3.4
backports.shutil-get-terminal-size==1.0.0
bayesian-optimization==0.4.0
bcolz==0.12.1
beautifulsoup4==4.5.3
bitarray==0.8.1
blaze==0.10.1
bokeh==0.12.4
boto==2.45.0
Bottleneck==1.2.1
butils==0.2.13
cachetools==2.0.0
cffi==1.9.1
chardet==2.3.0
chest==0.2.3
click==6.7
cloudpickle==0.2.2
clyent==1.2.2
colorama==0.3.7
conda==4.3.9
conda-build==2.1.3
conda-verify==2.0.0
configobj==5.0.6
contextlib2==0.5.5
coverage==4.3.4
coveralls==1.1
cryptography==1.7.1
cycler==0.10.0
cyordereddict==1.0.0
Cython==0.25.2
cytoolz==0.8.2
dask==0.13.0
data==0.3.7
datashape==0.5.4
deap==1.1.0
decorator==4.0.11
dill==0.2.5
docopt==0.6.2
docutils==0.13.1
dynd==0.7.3.dev1
empyrical==0.2.2
enum34==1.1.6
et-xmlfile==1.0.1
fastcache==1.0.2
filelock==2.0.7
Flask==0.12
Flask-Cors==3.0.2
future==0.16.0
gevent==1.2.1
greenlet==0.4.11
h5py==2.6.0
HeapDict==1.0.0
hmmlearn==0.2.0
idna==2.2
imagesize==0.7.1
inspyred==1.0.1
intervaltree==2.1.0
ipykernel==4.5.2
ipython==5.1.0
ipython-genutils==0.1.0
ipywidgets==5.2.2
isort==4.2.5
itsdangerous==0.24
jdcal==1.3
jedi==0.9.0
Jinja2==2.9.4
jsonschema==2.5.1
jupyter==1.0.0
jupyter-client==4.4.0
jupyter-console==5.0.0
jupyter-core==4.2.1
jupyterhub==0.7.2
lazy-object-proxy==1.2.2
learning==0.4.19
llvmlite==0.15.0
locket==0.2.0
Logbook==1.0.0
lru-dict==1.1.6
lxml==3.7.2
lz4==0.8.2
Mako==1.0.6
MarkupSafe==1.0
matplotlib==2.0.0
minepy==1.2.0
mistune==0.7.3
mockextras==1.0.2
mpmath==0.19
multipledispatch==0.4.9
nb-anacondacloud==1.2.0
nb-conda==2.0.0
nb-conda-kernels==2.0.0
nbconvert==4.2.0
nbformat==4.2.0
nbpresent==3.0.2
networkx==1.11
nltk==3.2.2
nose==1.3.7
notebook==4.3.1
numba==0.30.1
numexpr==2.6.2
numpy==1.12.1
numpydoc==0.6.0
odo==0.5.0
openpyxl==2.4.1
packaging==16.8
pamela==0.3.0
pandas==0.18.1
pandas-datareader==0.4.0
partd==0.3.7
pathlib2==2.2.0
patsy==0.4.1
pep8==1.7.0
pexpect==4.2.1
pickleshare==0.7.4
Pillow==4.0.0
pkginfo==1.4.1
ply==3.9
prompt-toolkit==1.0.9
psutil==5.0.1
ptyprocess==0.5.1
py==1.4.32
pyasn1==0.1.9
PyBrain==0.3.3
pycosat==0.6.1
pycparser==2.17
pycrypto==2.6.1
pycurl==7.43.0
pyflakes==1.5.0
pyfolio==0.7.0
Pygments==2.1.3
pylint==1.6.4
pymongo==3.4.0
pyOpenSSL==16.2.0
pypandoc==1.3.3
pyparsing==2.2.0
pytest==3.0.5
python-dateutil==2.6.0
python-editor==1.0.3
pytz==2017.2
PyWavelets==0.5.2
PyYAML==3.12
pyyawt==0.1.1
pyzmq==16.0.2
QtAwesome==0.4.3
qtconsole==4.2.1
QtPy==1.2.1
redis==2.10.5
requests==2.14.2
requests-file==1.4.2
requests-ftp==0.3.1
rope-py3k==0.9.4.post1
rpy2==2.8.5
scikit-image==0.12.3
scikit-learn==0.18.1
scipy==0.19.0
seaborn==0.7.1
simplegeneric==0.8.1
singledispatch==3.4.0.3
six==1.10.0
snowballstemmer==1.2.1
sockjs-tornado==1.0.3
sortedcontainers==1.5.7
Sphinx==1.5.1
sphinx-rtd-theme==0.1.9
spyder==3.1.2
SQLAlchemy==1.1.10
statsmodels==0.8.0
sympy==1.0
TA-Lib==0.4.10
tables==3.4.2
terminado==0.6
toolz==0.8.2
tornado==4.4.2
TPOT==0.6.8
tqdm==4.11.2
traitlets==4.3.1
tsfresh==0.5.0
tzlocal==1.3
unicodecsv==0.14.1
update-checker==0.16
wcwidth==0.1.7
Werkzeug==0.11.15
widgetsnbextension==1.2.6
wrapt==1.10.8
xgboost==0.6a2
xlrd==1.0.0
XlsxWriter==0.9.6
xlwt==1.2.0
zipline==1.1.0

Error retrieving H15 interest rates - ValueError: 'Time Period' is not in list

• If I try to instantiate a TradingEnvironment, I get a ValueError. The code worked until yesterday and stopped some hours ago. I'm not sure whether on Th or Fr (CEST).

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)

...
env = TradingEnvironment(bm_symbol=self.benchmark, exchange_tz=self.exchange_tz,
--> 539                                  trading_calendar=cal)
...

/opt/conda/envs/develop/lib/python3.5/site-packages/zipline/finance/trading.py in __init__(self, load, bm_symbol, exchange_tz, trading_calendar, asset_db_path)
     94             trading_calendar.day,
     95             trading_calendar.schedule.index,
---> 96             self.bm_symbol,
     97         )
     98 

/opt/conda/envs/develop/lib/python3.5/site-packages/zipline/data/loader.py in load_market_data(trading_day, trading_days, bm_symbol)
    169         first_date,
    170         last_date,
--> 171         now,
    172     )
    173     benchmark_returns = br[br.index.slice_indexer(first_date, last_date)]

/opt/conda/envs/develop/lib/python3.5/site-packages/zipline/data/loader.py in ensure_treasury_data(bm_symbol, first_date, last_date, now)
    317 
    318     try:
--> 319         data = loader_module.get_treasury_data(first_date, last_date)
    320         data.to_csv(path)
    321     except (OSError, IOError, HTTPError):

/opt/conda/envs/develop/lib/python3.5/site-packages/zipline/data/treasuries.py in get_treasury_data(start_date, end_date)
     74         parse_dates=['Time Period'],
     75         na_values=['ND'],  # Presumably this stands for "No Data".
---> 76         index_col=0,
     77     ).loc[
     78         start_date:end_date

/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in parser_f(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, escapechar, comment, encoding, dialect, tupleize_cols, error_bad_lines, warn_bad_lines, skipfooter, skip_footer, doublequote, delim_whitespace, as_recarray, compact_ints, use_unsigned, low_memory, buffer_lines, memory_map, float_precision)
    644                     skip_blank_lines=skip_blank_lines)
    645 
--> 646         return _read(filepath_or_buffer, kwds)
    647 
    648     parser_f.__name__ = name

/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in _read(filepath_or_buffer, kwds)
    387 
    388     # Create the parser.
--> 389     parser = TextFileReader(filepath_or_buffer, **kwds)
    390 
    391     if (nrows is not None) and (chunksize is not None):

/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in __init__(self, f, engine, **kwds)
    728             self.options['has_index_names'] = kwds['has_index_names']
    729 
--> 730         self._make_engine(self.engine)
    731 
    732     def close(self):

/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in _make_engine(self, engine)
    921     def _make_engine(self, engine='c'):
    922         if engine == 'c':
--> 923             self._engine = CParserWrapper(self.f, **self.options)
    924         else:
    925             if engine == 'python':

/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in __init__(self, src, **kwds)
   1434                 raise ValueError("Usecols do not match names.")
   1435 
-> 1436         self._set_noconvert_columns()
   1437 
   1438         self.orig_names = self.names

/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in _set_noconvert_columns(self)
   1484                         _set(k)
   1485                 else:
-> 1486                     _set(val)
   1487 
   1488         elif isinstance(self.parse_dates, dict):

/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in _set(x)
   1474 
   1475             if not is_integer(x):
-> 1476                 x = names.index(x)
   1477 
   1478             self._reader.set_noconvert(x)

ValueError: 'Time Period' is not in list

What steps have you taken to resolve this already?

• If I run the corresponding code manually, I get the same error. https://github.com/quantopian/zipline/blob/master/zipline/data/treasuries.py#L61

df = pd.read_csv(
        "http://www.federalreserve.gov/datadownload/Output.aspx"
        "?rel=H15"
        "&series=bf17364827e38702b42a58cf8eaa3f78"
        "&lastObs="
        "&from="  # An unbounded query is ~2x faster than specifying dates.
        "&to="
        "&filetype=csv"
        "&label=omit"
        "&layout=seriescolumn"
        "&type=package",
        skiprows=1,  # First row is a useless header.
        parse_dates=['Time Period'],
        na_values=['ND'],  # Presumably this stands for "No Data".
        index_col=0,
)

• The request to the hard encoded link (to obtain the H15 interest rates) fails: "Unable to find the output file. Please contact administrator for assistance." http://www.federalreserve.gov/datadownload/Output.aspx?rel=H15&series=bf17364827e38702b42a58cf8eaa3f78&lastObs=&from=&to=&filetype=csv&label=omit&layout=seriescolumn&type=package

• I played around with the Data Download Program and compared the parameters of the generated request strings. https://www.federalreserve.gov/releases/h15/

• It seems as if label=omit is not accepted anymore. If I omit the label parameter or set it to include, it seems to work. http://www.federalreserve.gov/datadownload/Output.aspx?rel=H15&series=bf17364827e38702b42a58cf8eaa3f78&lastObs=&from=&to=&filetype=csv&label=include&layout=seriescolumn&type=package

• I am curious if I am the only one having this problem. If not, how can we fix this problem permanently?

Sincerely,

Rudi

code snippet:

class TestAssetFinder(TestCase):
    @with_environment()
    def setUp(self, env=None):
        self.identifier = 'f'
        metadata={0: {'asset_type': 'future', 'symbol':
            self.identifier}}
        self.asset_finder = AssetFinder(metadata=metadata)
        env.update_asset_finder(identifiers=[self.identifier])

    @with_environment()
    def test_future_metadata(self, env=None):
        asset = env.asset_finder.lookup_generic(self.identifier,
                                                   datetime.now())
        self.assertIsInstance(asset, Future)

output

AssertionError: (Equity(0, symbol='f', asset_name='', exchange='', start_date=0, end_date=Timestamp('2015-08-20 00:00:00+0000', tz='UTC'), first_traded=None), []) is not an instance of <class 'zipline.assets._assets.Future'>

cc @jfkirk @StewartDouglas @yankees714 @dhexus

Hi guys,

Please update http://www.zipline.io/ because dual_moving_avg.py doesn't exist, its dual_moving_average.py. This strategy work perfectly in quantopian environment but not with zipline CLI. I suspect the ichart.finance.yahoo path to be broken (see details at the end).

The only example that works or don't crash is buyapple.
dual_ema_talib.py -> ImportError: No module named talib pairtrade.py-> ValueError: You must define a handle_data function. olmar.py-> AttributeError: 'TradingAlgorithm' object has no attribute 'symbol' buyapple_analyze.py-> ValueError: You must define a handle_data function.

Thanks,

Fraka6

zipline.version Out[39]: '0.7.0'

python scripts/run_algo.py -f zipline/examples/dual_moving_average.py --symbols AAPL --start 2011-1-1 --end 2012-1-1 -o dma.pickle ... [2015-07-30 04:14:04.415502] WARNING: Loader: No benchmark data found for date range. start_date=2015-07-30 00:00:00+00:00, end_date=2015-07-30 04:14:04.295081, url=http://ichart.finance.yahoo.com/table.csv?a=6&c=2015&b=30&e=30&d=6&g=d&f=2015&s=%5EGSPC [2015-07-30 04:14:24.025207] INFO: Performance: Simulated 252 trading days out of 252. [2015-07-30 04:14:24.025510] INFO: Performance: first open: 2011-01-03 14:31:00+00:00 [2015-07-30 04:14:24.025611] INFO: Performance: last close: 2011-12-30 21:00:00+00:00 Traceback (most recent call last): ... File "scripts/run_algo.py", line 25, in File "pandas/hashtable.pyx", line 694, in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:12231) KeyError: 'AAPL'

Instead of having separate ExchangeCalendar and TradingSchedule objects, we now just have TradingCalendar. The TradingCalendar keeps track of each session (defined as a contiguous set of minutes between an open and a close). It's also responsible for handling the grouping logic of any given minute to its containing session, or the next/previous session if it's not a market minute for the given calendar.

(Unfortunately, the branch name is a misnomer. no 24/5 yet.)

Background

Zipline currently requires two special data inputs for simulations: "Benchmark Returns", which are used to calculate the "Alpha" and "Beta" metrics, among other things, and "Treasury Curves", which were at one time used as the "Risk Free Rate", which was part of the Sharpe Ratio calculation.

Since these inputs are required by all simulations, we implicitly fetch them from third party API sources if they're not provided by users. We get treasury data from the US Federal Reserve's API, and we get benchmarks from IEX.

Problems

Implicitly fetching benchmarks and treasuries causes many problems:

• Implicitly fetching means that running simulations requires an internet connection. We try to make this less painful by caching downloaded results and re-using them when possible, but this is only a partial fix, and it means that many users don't notice the implicit download until it starts causing mysterious problems.

• The APIs we fetch from sometimes fail, leading to confusing behavior for users and spurious bug reports for Zipline maintainers.

• The APIs we fetch from sometimes change in incompatible ways, which breaks older versions of Zipline. This is currently the case for the IEX API we use to fetch benchmarks, resulting in issues like:

  • https://github.com/quantopian/zipline/issues/2480
  • https://github.com/quantopian/zipline/issues/2607
  • https://github.com/quantopian/zipline/issues/2445

• Our default benchmark is US-centric. We default to using SPY as the benchmark, which only makes sense in the US (and even then, only makes sense if you have also historical dividends for SPY, which many users don't have).

Proposed Solution

I think we should remove these implicit dependencies from Zipline. Treasuries we should just remove, since they're not actually used anymore. Figuring out what to do with benchmarks is a bit trickier.

Treasuries

Removing treasuries is relatively straightforward because we no longer actually use them. A quick scan of our GitHub issues turns up these issues that should be fixed by the removal:

• https://github.com/quantopian/zipline/issues/324
• https://github.com/quantopian/zipline/issues/119
• https://github.com/quantopian/zipline/issues/144
• https://github.com/quantopian/zipline/issues/2422

I've opened a PR at https://github.com/quantopian/zipline/pull/2626 to finally remove all traces of the treasury subsystem.

Benchmarks

Benchmarks are a bit trickier. The benchmark is used in the calculation of the "alpha" and "beta" metrics, and many users are generally interested in comparing the returns of their strategy against a particular benchmark (often an ETF or index of some kind). We also don't currently have a way to specify a benchmark from the command line, or to define a benchmark asset for a particular bundle.

I think there are a few things we could do to improve the situation here:

1. We could add the ability to define a benchmark explicitly when running Zipline via the CLI. We already have the ability to do this internally, but there's no supported way to control the benchmark via the CLI or via an extension. I think this is necessary pretty much no matter what.

   • Optionally, we could also make it required that the user tell us what their benchmark is. This would remove the need for implicit fetching of the benchmark. Users who don't care could pass a dummy benchmark (e.g., of all zero returns).

2. Make the benchmark optional. Making the benchmark optional would result in alpha, beta, and any other benchmark-dependent risk metrics not being populated in zipline's output. The tricky thing here is to do this in a way that doesn't result in performance degradataion when running with a benchmark. I think we either should do this or make the benchmark asset required.

3. (Short Term) We can fix our IEX API calls for benchmark data to use the updated APIs. This doesn't fix the systemic maintenance issues associated with the benchmark, but it would at least fix Zipline being straight-up broken for many people, which is its current status. I think the main challenge here is that IEX now requires an API token to work at all, which means we need to provide some mechanism for the user to pass in their API token.

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

Now that you know a little about me, let me tell you about the issue I am having:

I am trying to set up the zipline enviornment, its been incredibly time confusing and stressful, as I am a new prog. now when I try to ingest or run a program I get the following syntax error:

(env_zipline) [email protected] VSC % /Users/avant/opt/anaconda3/envs/env_zipline/bin/python /Users/avant/Desktop/VSC/agcap/ma.py Traceback (most recent call last): File "/Users/avant/Desktop/VSC/agcap/ma.py", line 1, in from zipline.api import order_target, record, symbol File "/Users/avant/opt/anaconda3/envs/env_zipline/lib/python3.6/site-packages/zipline/init.py", line 29, in from .utils.run_algo import run_algorithm File "/Users/avant/opt/anaconda3/envs/env_zipline/lib/python3.6/site-packages/zipline/utils/run_algo.py", line 20, in from zipline.data import bundles File "/Users/avant/opt/anaconda3/envs/env_zipline/lib/python3.6/site-packages/zipline/data/bundles/init.py", line 2, in from . import quandl # noqa File "/Users/avant/opt/anaconda3/envs/env_zipline/lib/python3.6/site-packages/zipline/data/bundles/quandl.py", line 16, in from . import core as bundles File "/Users/avant/opt/anaconda3/envs/env_zipline/lib/python3.6/site-packages/zipline/data/bundles/core.py", line 20, in from zipline.assets.asset_db_migrations import downgrade File "/Users/avant/opt/anaconda3/envs/env_zipline/lib/python3.6/site-packages/zipline/assets/asset_db_migrations.py", line 1, in from alembic.migration import MigrationContext File "/Users/avant/opt/anaconda3/envs/env_zipline/lib/python3.6/site-packages/alembic/init.py", line 3, in from . import context File "/Users/avant/opt/anaconda3/envs/env_zipline/lib/python3.6/site-packages/alembic/context.py", line 1, in from .runtime.environment import EnvironmentContext File "/Users/avant/opt/anaconda3/envs/env_zipline/lib/python3.6/site-packages/alembic/runtime/environment.py", line 1 from future import annotations ^ SyntaxError: future feature annotations is not defined

Description of Issue

• What did you expect to happen?
• What happened instead?

Here is how you can reproduce this issue on your machine:

Reproduction Steps

1.run any program or try to ingest 2. 3. ...

What steps have you taken to resolve this already?

many hours of trying different versions of python/pandas and the rest of the libs. many GitHub/stackoverflow articles.

...

Anything else?

...

Sincerely, `$ whoami

I also attached a screenshot `

Patching CVE-2007-4559

Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

Dear Zipline Maintainers, go to your https://pypi.org/project/zipline/#description page. try the links. specifically the 'Zipline Install Documentation' link. that link, as well as many others, go to a useless advert for zipline

not a confidence builder

There are small typos in:

• tests/test_assets.py
• zipline/data/minute_bars.py
• zipline/errors.py
• zipline/finance/controls.py

Fixes:

• Should read constraint rather than contraint.
• Should read slippage rather than slipage.
• Should read identifier rather than identifer.
• Should read considered rather than consdered.

Semi-automated pull request generated by https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md

Any issues raised for in this Git are unlikely to be resolved, since Quantopian has folded and the staff that have moved to Robinhood have not been involved for a few years, so it's currently unlikely that anything will happen here.

However, thanks to the foresight of the Quantopian founders and devs who made this open source, other contributors have taken the banner and have improved this package from where Quantopian left it.

@stefan-jansen maintains a forked, updated version of this package here: https://github.com/stefan-jansen/zipline-reloaded

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

• What did you expect to happen?
• What happened instead?

Here is how you can reproduce this issue on your machine:

Reproduction Steps

...

What steps have you taken to resolve this already?

...

Anything else?

...

Sincerely, $ whoami