As @yy910616 pointed out in #13, our implementation of STL decomposition was wrong and it was basically classic seasonal decomposition (trend as moving average, seasonal as average across periods, plus residual). In this PR, we want to correct it.

• [x] Merge the original NaiveSeasonalAD (a special version of classic decomposition where the trend is preassumed constant 0) with original STLDecomposition (which is in fact classic seasonal decomposition as @yy910616 pointed out), and make it ClassicSeasonalDecomposition
• [ ] ~Implement the real STLDecomposition~
• [x] Updated docs

Updated: for item 2, we decided to hold on for statsmodels new release (details in the thread).

We like to start using type hints for better practice of python programming. For a first-time contributor, this is probably a nice starting point, as you will go through every part of the code base and familiarize yourself with the code structure.

To-do's:

• [ ] Add type hints to all functions.
• [ ] Modify docstrings accordingly, so sphinx-autodoc will automatically grab type info from type hints.
• [ ] Add unit tests (with mypy?) for type checking.

Have fun!

First of all, thx for the great tool ^^

here's the code that produce this RuntimeError:

tmdl.py

from adtk.detector import ThresholdAD
from adtk.detector import QuantileAD
from adtk.detector import InterQuartileRangeAD
from adtk.detector import PersistAD
from adtk.detector import LevelShiftAD
from adtk.detector import VolatilityShiftAD
from adtk.detector import SeasonalAD
from adtk.detector import AutoregressionAD

def get_detector(adname="ThresholdAD"):
    detectors = {"ThresholdAD": ThresholdAD,
                 "QuantileAD": QuantileAD,
                 "InterQuartileRangeAD": InterQuartileRangeAD,
                 "PersistAD": PersistAD,
                 "LevelShiftAD": LevelShiftAD,
                 "VolatilityShiftAD": VolatilityShiftAD,
                 "SeasonalAD": SeasonalAD,
                 "AutoregressionAD": AutoregressionAD,
                 }
   return detectors.get(adname)

\# using adtk anomoly detectors
def ad_detector(dname, train_data=None, test_data=None, **kwargs):
    Ad = get_detector(dname)
    ad = Ad(**kwargs)
    train_anoms = ad.fit_detect(train_data)
    test_anoms = ad.detect(test_data)
    return train_anoms, test_anoms

I wrote these functions to help me quickly doing some experiment with different detectors by changing the detector's name in the main() function. That's what I thought. --!

main.py

...(functons read the data)

s_train, s_test = split_train_test(data, mode=split_mode, n_splits=n_splits)
train_anoms, test_anoms = [], []
for train, test in zip(s_train, s_test):  # the Error show up in this for loop
        train_anom, test_anom = tmdl.ad_detector(dname='SeasonalAD',
                                                 train_data=train,
                                                 test_data=test.squeeze(),
                                                 c=1, side='both')
        # collect the results
        train_anoms.append(train_anom)
        test_anoms.append(test_anom)

When ran this piece of code, it reported RuntimeError: the model must be trained first.

Last but not least, when I followed the Quick Start, the machine did not complain anything.

Any help would be appreciated.

Code is as follows:

from adtk.transformer import ClassicSeasonalDecomposition
s_transformed = ClassicSeasonalDecomposition().fit_transform(s).rename("Seasonal decomposition residual")
plot(pd.concat([s, s_transformed], axis=1), ts_markersize=1);

my data frame has multiple numeric ( float & int columns with date as index ).

and I am getting following error:

ValueError Traceback (most recent call last) in 1 from adtk.transformer import ClassicSeasonalDecomposition ----> 2 s_transformed = ClassicSeasonalDecomposition().fit_transform(s).rename("Seasonal decomposition residual") 3 plot(pd.concat([s, s_transformed], axis=1), ts_markersize=1);

~/jbooks/notebooks/lib/python3.6/site-packages/adtk/_transformer_base.py in fit_predict(self, ts) 94 95 """ ---> 96 self.fit(ts) 97 return self.predict(ts) 98

~/jbooks/notebooks/lib/python3.6/site-packages/adtk/_transformer_base.py in fit(self, ts) 47 48 """ ---> 49 self._fit(ts) 50 51 def predict(

~/jbooks/notebooks/lib/python3.6/site-packages/adtk/_base.py in _fit(self, ts) 172 # fit model for each column 173 for col in df.columns: --> 174 self._models[col].fit(df[col]) 175 self._fitted = 2 176 else:

~/jbooks/notebooks/lib/python3.6/site-packages/adtk/_transformer_base.py in fit(self, ts) 47 48 """ ---> 49 self._fit(ts) 50 51 def predict(

~/jbooks/notebooks/lib/python3.6/site-packages/adtk/_base.py in _fit(self, ts) 152 if isinstance(ts, pd.Series): 153 s = ts.copy() # type: pd.Series --> 154 self._fit_core(s) 155 self._fitted = 1 156 elif isinstance(ts, pd.DataFrame):

~/jbooks/notebooks/lib/python3.6/site-packages/adtk/transformer/_transformer_1d.py in _fit_core(self, s) 684 s.index.is_monotonic_increasing or s.index.is_monotonic_decreasing 685 ): --> 686 raise ValueError("Time series must have a monotonic time index. ") 687 # remove starting and ending nans 688 s = s.loc[s.first_valid_index() : s[::-1].first_valid_index()].copy()

ValueError: Time series must have a monotonic time index.

Any help is highly appreciated...

First, big thanks for a nice library! Very useful!

I am trying to follow your Quick Start for SeasonalAD, but I am encountering a problem. My timeseries seem be to unequally spaced (e.g. 09:05, 09:15, 9:30, 9:55). Hence the SeasonalAD complains: RuntimeError: Series does not follow any known frequency (e.g. second, minute, hour, day, week, month, year, etc. How to overcome this? I have tried rounding my series to 15min, removing duplicates and resampling.

s_train.index = s_train.index.round('15min')
s_train = s_train[~s_train.index.duplicated()]
s_train = s_train.asfreq('15min')

Obviously nothing worked. Any ideas how to solve this? I wish to retain as much granularity as possible.

I use VolatilityShiftAD and set 'side=both/positive/negative' to see how different about them, but the results of them are exactly same. VolatilityShiftAD can not detect the negative anomaly.

('seismic-new.csv': copy the nomaly data which from 'seismic.csv' to the end of file)

pipnet = VolatilityShiftAD(window=20)
s = pd.read_csv('~/data/seismic-new.csv', index_col="Time", parse_dates=True, squeeze=True)
s = validate_series(s)

anomalies = pipnet.pipe_.fit_detect(s,return_intermediate=True)

plot(anomalies["diff_abs"])
#plot(s, anomaly=anomalies, anomaly_color='red')
plt.savefig("VSAD-std.png")

anomalies:

diff_abs(std):

I use Excel to calculate std at two time '08-05 15:05:00'(left window < 15:05:00, right window > 15:05:00) and '08-05 16:56:00', the result is different from the VolatilityShiftAD.

Hi in the quickstart guide you point ot the nyc taxi dataset on the Numenta Anomaly Benchmark but do not provide the links to the files required (or transformations to generat them). I have located training.csv as NAB/data/realKnownCause/nyc_taxi.csv but have not located known_anomalies.csv Thanks

Hello,

I am trying to using adtk to find outliers in my data. I used the below code. outlier_detector = OutlierDetector(LocalOutlierFactor(contamination=0.05)) anomalies = outlier_detector.fit_detect(df_anomaly) print(anomalies) My output is type bool where I see something like this log_time 2022-02-04 09:48:07 False 2022-02-06 16:16:19 False 2022-02-06 16:21:20 True 2022-02-06 16:26:20 False 2022-02-06 16:31:20 True ...
2022-02-07 05:56:23 False 2022-02-07 06:01:23 False 2022-02-07 06:06:23 False 2022-02-07 06:11:23 False 2022-02-07 06:16:23 False

Is this a dataframe? How can I get list of values which have True (which I believe are outliers)? I need to plot those outliers on my plot.

Any help will be really appreciated.

I have created a simple example. ​

from adtk.detector import LevelShiftAD
from adtk.visualization import plot
import pandas

d1 = [1, 10000, 1,     10000, 10000, 10000, 10000, 10000, 10000, 10000, 10000, 10000, 10000, 1]
d2 = [1, 10000, 10000, 10000, 10000, 10000, 10000, 10000, 10000, 10000, 10000, 10000, 10000, 1]
s = pandas.Series(d1, index=pandas.date_range("2021-01-01", periods=len(d)))

level_shift_ad = LevelShiftAD(c=6.0, side='both', window=2)
anomalies = level_shift_ad.fit_detect(s)

plot(s, anomaly=anomalies, anomaly_color='red');

With d2 two anomalies are detected. With d1 no anomalies are detected.

Why? Or maybe the question must be: What should I look at to understand? :)

Thanks

Hey folks,

I am trying to import LevelShiftAD but it seems there are some import problems within this module. This is what I get when I try to import it

     10 from scipy import linalg, fft as sp_fft
     11 from scipy.fft._helper import _init_nd_shape_and_axes
---> 12 from scipy._lib._util import prod as _prod
     13 import numpy as np
     14 from scipy.special import lambertw

ImportError: cannot import name 'prod'

I am running Python 3.6.9 on a virtual environment. Any clue of how I could fix it? I've already tried it by downgrading scipy to 1.2 and 1.3.

Thanks!

I fit seasonal detector and transformer to a multivariate time series following the documentation.

https://adtk.readthedocs.io/en/stable/api/transformers.html#adtk.transformer.ClassicSeasonalDecomposition

https://adtk.readthedocs.io/en/stable/notebooks/demo.html#SeasonalAD

When I fit a pandas Dataframe with several columns (multivariate time series) to these detector/transformer, the seasonal_ component does not exist anymore.

Would it be possible to get a seasonal_ dataframe in that case ?

First of all, thanks a lot for providing such an wonderful package.

I was using Quantile AD, and I thought it was marking all values above high quantile and lower quantile as outlier (like the one threshold criteria would do, when appropriate quantile bounds are provided).

However it seems like the method is not working in such a way, as there are several outlier-regarded middle values between very high unmarked values.

I was trying to read the source code, but I could not really get the logic (I think I got the wrong idea, as from my understanding, the above phenomena should not happen).

Can you please provide me some explanation about this? I really would appreciate.

I really would like to attach the plot I have, but due to several security issues of the company, I cannot do so.

Sometimes an anomaly could lie in the fact that the timeseries value is not changing at all, that is the previous value is exactly the same as the next one.

Is it possible to adapt the PersistAD detector to identify these points as anomalies?

I was looking into the underlying Pipenet implementation but unfortunately it seems that the only way to identify the anomaly is to find something exceeding a certain threshold and not just staying the same.

import numpy as np import pandas as pd from adtk.data import validate_series import matplotlib.pyplot as plt from adtk.visualization import plot from adtk.transformer import DoubleRollingAggregate from adtk.detector import ThresholdAD, QuantileAD from adtk.aggregator import AndAggregator

故障检测

from adtk.pipe import Pipenet

steps = { # 长窗口历史数据扫描，规避持续异常跳变 "more_step_roll_aggr": { "model": DoubleRollingAggregate( agg="max", window=(20, 1), diff="diff" ), "input": "original", }, # 确保递增是一直持续存在 "abs_more_step_roll_aggr": { "model": ThresholdAD( high=0 ), "input": "more_step_roll_aggr" },

# 短窗口数据扫描，相对变化率(斜率)判断
"one_step_change": {
    "model": DoubleRollingAggregate(
        agg="mean",
        window=(3, 1),
        diff="rel_diff"
    ),
    "input": "original",
},
"abs_one_step_change": {
    "model": ThresholdAD(high=0.2),
    "input": "one_step_change"
},

"base_level_detect1": {
    "model": ThresholdAD(high=100),
    "input": "original"
},

"positive_level_shift": {
    "model": AndAggregator(),
    "input": ["abs_more_step_roll_aggr", "abs_one_step_change", "base_level_detect"]
}

}

pipenet = Pipenet(steps) pipenet.plot_flowchart()

plt.show()

plt.savefig('net1.png')

Hello, I'd like to know if it was possible to retrieve the different parameters calculated by a pipeline? For example my pipeline is composed by a ClassicSeasonalDecomposition transformer and an InterQuartileAD detector. After fitting the model, I'd like to retrieve the pattern calculated and the value of abs_low_ and abs_high. It seems that we can do it using the two models separately but not when we use them through a pipeline.

Thank you in advance!

Hi the team, thanks for your great work, is it possible to return a list of anomaly scores instead of binary labels in the future version? As a researcher in anomaly detection, I'd like to say anomaly scores are much useful than the binary labels.

label: question

Hello,

I am a data science student using the PCA and LOF detectors for a project and I want to gain a deep understanding of how the algorithms decide whether an anomaly exists or not. Is there information somewhere that describes what is really going on underneath the hood? Thank you very much for any help!