One last question for today ;)

The 'simulation' method for forecast is very simple and fast. Just curious why it isn't built to forecast beyond the last index of the dataset?

is it possible to, say, if you want to forecast 30 days beyond the current dataset, you could simply add 30 days to the dataset with np.nan or 0 and set the last_obs to the last day of actual data? I've tried this and the forecasts come out alot different than the in-sample ones. Seems like the residuals from the entire dataset are getting incorporated into a few things, like compute_variance maybe, whether they are actually being used to fit or not.

Anyway, just wondering if there's a best way to do this. Kind of looking like, if one wants to do, say, 1,000 sims of 30 days forward forecasted, a new model would need to be instantiated and fit 30x1000 = 30,000 times .....

Hi, I recently start to use your library for modelling time series. It works perfectly fine under single thread. But if I use ARX model and fit a lot of them in a multithreading task. Then I randomly got either

`The optimizer returned code 9. The message is: Iteration limit exceeded See scipy.optimize.fmin_slsqp for code meaning.

ConvergenceWarning)`

or The optimizer returned code 5 problems. So I get nan for the final results. Is it normal? Is ARX thread safe?

Hi, I'm using Python 3.7. In order to compute the value at risk, I have to forecast FIGARCH and calculate the daily conditional mean and standard deviation. To do that, I used the package ARCH which contains the FIGARCH model. Please find my code below:

import numpy as np
import pandas as pd
import scipy as sp
from scipy.stats import stats,norm
import scipy.stats as stats
df = pd.read_excel ('E:\\Book1.xlsx',sheet_name="Sheet3",header=0)
ret_co1 = df.CO1.pct_change().dropna()
mu_co1 = np.mean(ret_co1)
sigma_co1 = np.std(ret_co1)

from arch.univariate import FIGARCH
mod_4 = FIGARCH(p=1, q=1, power=2.0, truncation=1000)
resid_4 = ret_co1 - mu_co1
residu_4 = resid_4.values
var_4 = np.full((1,4595),sigma_co1)
vari_4 = var_4[0]
bc_4 = mod_4.backcast(resid_4)
bn_4 = mod_4.bounds(resid_4)
para=[1,1]
fig_4 = mod_4.compute_variance(parameters=para, resids=residu_4, sigma2=vari_4, backcast=bc_4, var_bounds=bn_4)

ret_co1 are my returns mu_co1 is my mean residu_4 is my vector of mean residuals I computed as resids = returns - E(returns) vari_4 is the vector of conditional variance where I put the same variance on each element of the matrix para[1,1] is for p=1 and q=1

The message error I got is: "a bytes-like object is required, not 'list'" and came from the File "C:\Users\Dimitry\Anaconda3\lib\site-packages\arch\univariate\volatility.py", line 2519, in compute_variance var_bounds) Can you help me to fix this error please?

We run AR-GARCH model on two different platforms with python3.5.3 and arch 4.6.0 package. One is Windows 7 Enterprise x64, the other is CentOS release 6.4 (Final) x86_64 GNU/Linux. The prediction results of the two platforms are very different on some sequences. for example:

from arch.univariate import ARX from arch.univariate import GARCH

L = [0.024537471, 0.011513695, 1.38838E-4, 0.034646548, -0.010029519, 0.013727814, -0.002833435, 0.012121772, -0.045976405, 0.00194646, 9.60483E-4, -0.041520158, 0.022433862, -0.012950044, 0.001248923] ar_asset = ARX(L, lags=[1]) ar_asset_model = ar_asset.fit() mean_asset_pre = ar_asset_model.forecast(horizon=1).mean[-1:]['h.1'].values[0] print('mean_asset_pre is:', mean_asset_pre)

The result on Windows 7 Enterprise x64 is "mean_asset_pre is: -0.01279744", but the result on CentOS release 6.4 (Final) x86_64 GNU/Linux is "mean_asset_pre is: -57747.628851".

Add variance forecasting to the forecast function[

• [x] Constant Variance Forecasting
• [x] GARCH Forecasting
• [ ] EGARCH Forecasting
• [x] HARCH Forecasting
• [x] EWMA Forecasting
• [x] EWMA2006 Forecasting
• [x] Documentation
• [x] IPython notebook
• [x] Hedgehog plot
• [ ] Test first_obs, last_obs and hold_back interaction with forecasting

Minimal example to show the current behaviour:

In [1]: from arch.bootstrap import IIDBootstrap

In [2]: IIDBootstrap([1, 2, 3], [4, 5])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-46a2c8a96fe3> in <module>()
----> 1 IIDBootstrap([1, 2, 3], [4, 5])

~/projects/automattic/data-science/conda_env/lib/python3.6/site-packages/arch/bootstrap/base.py in __init__(self, *args, **kwargs)
    155         for arg in all_args:
    156             if len(arg) != self._num_items:
--> 157                 raise ValueError("All inputs must have the same number of "
    158                                  "elements in axis 0")
    159         self._index = np.arange(self._num_items)

ValueError: All inputs must have the same number of elements in axis 0

With real-life data, sample sizes of different groups often vary. What is the reason for not supporting different sample sizes in IIDBootstrap? Is there a workaround?

Re-opening https://github.com/bashtage/arch/pull/328

Added dist.moment(h) for each univariate.Distribution subclass. Also added dist.partial_moment(h, z), which can be useful when working with models that have nonzero asymmetric dependence. Both methods have closed-form solutions for all distributions.

The test code compares the methods against numbers obtained by numeric integration.

Is it possible to use ARMAResults calculated using statsmodels as a mean model so that a volatility process can be added?

I wasn't sure how this might work but I was thinking of something along the lines of this:

from arch.univariate.mean import ARCHModel
from arch.univariate import ARCH

model = ARCHModel(returns)
model.mean = arma_res  # statsmodels ARMAResults
model.volatility = ARCH()

results = model.fit(update_freq=5)

Hi -

I tried to install first via

pip install arch ...

I got this error:

ValueError: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216 from C header, got 192 from PyObject

Then I tried to install via

pip install arch --install-option="--no-binary" ... which resulted in almost the same error:

AttributeError: type object 'arch.univariate.recursions.array' has no attribute 'reduce_cython'

Can you help, pls ?

Get cStringIO and range imports from arch.compat.python instead of statsmodels.compat

Get cache_readonly from pandas (cython implementation, better testing)

Use dict instead of resettable_cache. I know in sm all usages of resettable_cache are unnecessary; am just assuming the same is true here.

Hi, the arch model with t distribution works fine with simulated date, but it has very unstable estimate with real data, such as stock daily returns. I tried with a AR(2)-GARCH(1,1)-t model for SP 500 stocks with 5 year daily returns. The AR coefficients of some stock are way beyond 1, like 10 or 1000, which means an non-stationary process, or an error. Then I tried with one stock estimation starting with 1000 observations, and add one obs each time to the sample. The coefficients estimated each time are quite volatile and become over 1 sometimes. If I switch to AR-GARCH-normal, the coefficient estimates are very stable. So I think it has problem with MLE estimation of t distribution. It is challenging to estimate GARCH-t model. Matlab has this function and does fairly well in latest versions. Would you be able to improve that? We can discuss about the estimation methods but I found it difficult to change your code myself. Thank you

Daniel

This PR was automatically created by Snyk using the credentials of a real user.

Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

Changes included in this PR

• Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
  • requirements-dev.txt

Sphinx 1.8.6 has requirement docutils<0.18,>=0.11, but you have docutils 0.18.1.
notebook 5.7.16 requires terminado, which is not installed.
nbformat 4.4.0 requires jsonschema, which is not installed.
jupyter 1.0.0 requires qtconsole, which is not installed.
ipython 5.10.0 requires simplegeneric, which is not installed.

Vulnerabilities that will be fixed

By pinning:

Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- | 551/1000
Why? Recently disclosed, Has a fix available, CVSS 5.3 | Regular Expression Denial of Service (ReDoS)
SNYK-PYTHON-SETUPTOOLS-3180412 | setuptools:
39.0.1 -> 65.5.1
| No | No Known Exploit

(*) Note that the real score may have changed since the PR was raised.

Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.

Check the changes in this PR to ensure they won't cause issues with your project.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

For more information: 🧐 View latest project report

🛠 Adjust project settings

📚 Read more about Snyk's upgrade and patch logic

Learn how to fix vulnerabilities with free interactive lessons:

🦉 Regular Expression Denial of Service (ReDoS)

I am trying to understand -- when forecasting using GARCH, how do the simulated "values" compare to the "residual_variances". From a few experiments, I cannot get them to match.

To be clear, I'm looking at these two properties in the ARCHModelForecast object:

• simulations.values
• simulations.residual_variances

From my understanding of the docs, the volatility series is "simulated" via drawing shocks e_t ~ N(0, 1) many times. So there are (by default) 1000 "volatility paths", which are then averaged out to be the expected volatility. Each of these paths also correspond to a sequence of values (corresponding to epsilon_t in the docs), which is just white noise times sqrt(current modeled volatility).

However, when I try to compare the following two, I don't get the same answer.

Calculation 1

• Compute.fit( .. ).forecast( 5 days ).simulations.residual_variances
• Take the mean of the last row (corresponding to the expected volatility of the 5th day volatility forecast)

Calculation 2

• Compute .fit(..).forecast( 5 days ).simulations.values
• This results in a giant dataframe whcih is of size (num simulations) x ( 5 days )
• Take the last row (corresponding to the 5th day simulated return)
• Square these and take the mean (corresponding to E[ epsilon^2 ] as written in the docs

These two don't always give the same result. Any ideas why? I can provide more detail if anything is unclear. Have gone through source code too, but no luck

This PR was automatically created by Snyk using the credentials of a real user.

Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

Changes included in this PR

• Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
  • requirements-dev.txt

Sphinx 1.8.6 has requirement docutils<0.18,>=0.11, but you have docutils 0.18.1.
seaborn 0.9.1 requires numpy, which is not installed.
scipy 1.2.3 requires numpy, which is not installed.
pytest-cov 2.12.1 requires coverage, which is not installed.
pandas 0.24.2 requires numpy, which is not installed.
notebook 5.7.16 requires terminado, which is not installed.
nbformat 4.4.0 requires jsonschema, which is not installed.
matplotlib 2.2.5 requires numpy, which is not installed.
jupyter 1.0.0 requires qtconsole, which is not installed.
ipython 5.10.0 requires simplegeneric, which is not installed.

Vulnerabilities that will be fixed

By pinning:

Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- | 441/1000
Why? Recently disclosed, Has a fix available, CVSS 3.1 | Regular Expression Denial of Service (ReDoS)
SNYK-PYTHON-SETUPTOOLS-3113904 | setuptools:
39.0.1 -> 65.5.1
| No | No Known Exploit

(*) Note that the real score may have changed since the PR was raised.

Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.

Check the changes in this PR to ensure they won't cause issues with your project.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

For more information: 🧐 View latest project report

🛠 Adjust project settings

📚 Read more about Snyk's upgrade and patch logic

Learn how to fix vulnerabilities with free interactive lessons:

🦉 Regular Expression Denial of Service (ReDoS)

This PR was automatically created by Snyk using the credentials of a real user.

Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

Changes included in this PR

• Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
  • doc/requirements.txt

Sphinx 1.8.6 has requirement docutils<0.18,>=0.11, but you have docutils 0.18.1.
notebook 5.7.16 requires terminado, which is not installed.
nbformat 4.4.0 requires jsonschema, which is not installed.
nbconvert 5.6.1 has requirement mistune<2,>=0.8.1, but you have mistune 2.0.3.
jupyter 1.0.0 requires qtconsole, which is not installed.
Jinja2 2.11.3 requires MarkupSafe, which is not installed.
ipython 5.10.0 requires simplegeneric, which is not installed.

Vulnerabilities that will be fixed

By pinning:

Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- | 479/1000
Why? Has a fix available, CVSS 5.3 | Regular Expression Denial of Service (ReDoS)
SNYK-PYTHON-MISTUNE-2940625 | mistune:
0.8.4 -> 2.0.3
| No | No Known Exploit

(*) Note that the real score may have changed since the PR was raised.

Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.

Check the changes in this PR to ensure they won't cause issues with your project.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

For more information: 🧐 View latest project report

🛠 Adjust project settings

📚 Read more about Snyk's upgrade and patch logic

Learn how to fix vulnerabilities with free interactive lessons:

🦉 Regular Expression Denial of Service (ReDoS)

Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

Changes included in this PR

• Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
  • doc/requirements.txt

Sphinx 1.8.6 has requirement docutils<0.18,>=0.11, but you have docutils 0.18.1.
notebook 5.7.15 requires terminado, which is not installed.
nbformat 4.4.0 requires jsonschema, which is not installed.
nbconvert 5.6.1 has requirement mistune<2,>=0.8.1, but you have mistune 2.0.3.
jupyter 1.0.0 requires qtconsole, which is not installed.
Jinja2 2.11.3 requires MarkupSafe, which is not installed.
ipython 5.10.0 requires simplegeneric, which is not installed.

Vulnerabilities that will be fixed

By pinning:

Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- | 479/1000
Why? Has a fix available, CVSS 5.3 | Regular Expression Denial of Service (ReDoS)
SNYK-PYTHON-MISTUNE-2940625 | mistune:
0.8.4 -> 2.0.3
| No | No Known Exploit

(*) Note that the real score may have changed since the PR was raised.

Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.

Check the changes in this PR to ensure they won't cause issues with your project.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

For more information: 🧐 View latest project report

🛠 Adjust project settings

📚 Read more about Snyk's upgrade and patch logic

Learn how to fix vulnerabilities with free interactive lessons:

🦉 Regular Expression Denial of Service (ReDoS)

One feature I think would be useful is bootstrap methods for nonstationary time-series. I am currently looking at the Local block bootstrap (LBB), which was relatively straightforward to implement.

The LBB works by assuming that the time-series is almost stationary, but with slowly changing properties (e.g. the seasons in a year). Then, it creates bootstrap samples by sampling blocks near each other and stitching them together.

I have created a prototype, which seems to work, but I don't have any unit tests yet.

import numpy as np
from arch.bootstrap.base import IIDBootstrap, _get_random_integers, ArrayLike, RandomState, Generator, Int64Array


class LocalBlockBootstrap(IIDBootstrap):
    _name = "Local Block Bootstrap"

    def __init__(
        self,
        block_size: int,
        max_step: int,
        *args: ArrayLike,
        random_state: RandomState | None = None,
        seed: None | int | Generator | RandomState = None,
        **kwargs: ArrayLike,
    ) -> None:
        super().__init__(*args, random_state=random_state, seed=seed, **kwargs)
        self.block_size: int = block_size
        self.max_step: int = max_step
        self._parameters = [block_size, max_step]

    def clone(
        self,
        *args: ArrayLike,
        seed: None | int | Generator | RandomState = None,
        **kwargs: ArrayLike,
    ) -> 'LocalBlockBootstrap':

        block_size = self._parameters[0]
        max_step = self._parameters[1]
        return self.__class__(block_size, max_step, *args, random_state=None, seed=seed, **kwargs)

    def update_indices(self) -> Int64Array:
        num_blocks = self._num_items // self.block_size
        if num_blocks * self.block_size < self._num_items:
            num_blocks += 1

        m = np.arange(num_blocks)
        lower = np.maximum(0, self.block_size * m - self.max_step - 1)
        upper = np.minimum(self._num_items - self.block_size, self.block_size * m + self.max_step - 1)
        step = upper - lower
        indices = lower + _get_random_integers(self.generator, step, size=len(step))

        indices = indices[:, np.newaxis] + np.arange(self.block_size)[np.newaxis, :]
        indices = indices.flatten()

        if indices.shape[0] > self._num_items:
            return indices[: self._num_items]
        else:
            return indices