Extremely long time-series data leads causes evaluation failure

[RobbyW551]

NOTE: ISSUES ARE NOT FOR CODE HELP - Ask for Help at https://stackoverflow.com

Your issue may already be reported!
Also, please search on the issue tracker before creating one.

• I'm submitting a ...
  • bug report

Issue Description

• When Issue Happens
• Steps To Reproduce
  1. Download Covid Death Daily Dataset from https://zenodo.org/record/4656009/files/covid_deaths_dataset.zip?download=1 as mentioned in the paper. This dataset
  2. Convert corresponding .tsf file into the same pandas data frame format as in https://github.com/automl/Auto-PyTorch/blob/master/examples/20_basics/example_time_series_forecasting.py.
  3. To prove that it is the length of the time series that matters, the first 30 time series data points are picked to form a shorter time series sequence, which did not cause any error.

Expected Behavior

Evaluate the performance normally of the initial designs on Covid Death Daily Dataset.

Current Behavior

An error "ValueError: Expected parameter df (Tensor of shape (32, 3, 1)) of distribution Chi2() to satisfy the constraint GreaterThan(lower_bound=0.0)" occurs.

Possible Solution

Maybe there's a default maximum length of time-series sequence I didn't notice.

Your Code

# will be added soon.

Error message

[INFO] [2022-08-07 11:07:57,681:Client-TAE] Starting to evaluate configuration 1
[DEBUG] [2022-08-07 11:07:57,685:Client-TAE] Search space updates for 2: <autoPyTorch.utils.hyperparameter_search_space_update.HyperparameterSearchSpaceUpdates object at 0x7ff3b2664f10>
[DEBUG] [2022-08-07 11:07:57,685:Client-pynisher] Restricting your function to 499 seconds wall time.
[DEBUG] [2022-08-07 11:07:57,685:Client-pynisher] Allowing a grace period of 0 seconds.
[DEBUG] [2022-08-07 11:07:57,685:Client-pynisher] Function called with argument: (), {'queue': <multiprocessing.queues.Queue object at 0x7ff3b2301bb0>, 'config': Configuration(values={
  'data_loader:backcast': False,
  'data_loader:batch_size': 32,
  'data_loader:num_batches_per_epoch': 50,
  'data_loader:sample_strategy': 'SeqUniform',
  'data_loader:transform_time_features': False,
  'data_loader:window_size': 2,
  'loss:DistributionLoss:aggregation': 'median',
  'loss:DistributionLoss:dist_cls': 'studentT',
  'loss:DistributionLoss:forecast_strategy': 'sample',
  'loss:DistributionLoss:num_samples': 100,
  'loss:__choice__': 'DistributionLoss',
  'lr_scheduler:ReduceLROnPlateau:factor': 0.5,
  'lr_scheduler:ReduceLROnPlateau:mode': 'max',
  'lr_scheduler:ReduceLROnPlateau:patience': 10,
  'lr_scheduler:__choice__': 'ReduceLROnPlateau',
  'network_backbone:__choice__': 'flat_encoder',
  'network_backbone:flat_encoder:MLPDecoder:has_local_layer': True,
  'network_backbone:flat_encoder:MLPDecoder:num_layers': 0,
  'network_backbone:flat_encoder:MLPDecoder:units_local_layer': 40,
  'network_backbone:flat_encoder:MLPEncoder:activation': 'relu',
  'network_backbone:flat_encoder:MLPEncoder:normalization': 'NoNorm',
  'network_backbone:flat_encoder:MLPEncoder:num_groups': 1,
  'network_backbone:flat_encoder:MLPEncoder:num_units_1': 40,
  'network_backbone:flat_encoder:MLPEncoder:use_dropout': False,
  'network_backbone:flat_encoder:__choice__': 'MLPEncoder',
  'network_embedding:__choice__': 'NoEmbedding',
  'network_init:XavierInit:bias_strategy': 'Normal',
  'network_init:__choice__': 'XavierInit',
  'optimizer:AdamOptimizer:beta1': 0.9,
  'optimizer:AdamOptimizer:beta2': 0.999,
  'optimizer:AdamOptimizer:lr': 0.001,
  'optimizer:AdamOptimizer:weight_decay': 1e-08,
  'optimizer:__choice__': 'AdamOptimizer',
  'target_scaler:scaling_mode': 'mean_abs',
  'trainer:__choice__': 'ForecastingStandardTrainer',
})
, 'backend': <autoPyTorch.automl_common.common.utils.backend.Backend object at 0x7ff3b24c1610>, 'metric': mean_MASE_forecasting, 'seed': 1, 'num_run': 2, 'output_y_hat_optimization': True, 'include': {}, 'exclude': {}, 'disable_file_output': [], 'instance': '{"task_id": "1fcff3ff-15fe-11ed-9173-1dfac5c47e62"}', 'init_params': {'instance': '{"task_id": "1fcff3ff-15fe-11ed-9173-1dfac5c47e62"}'}, 'budget': 5.555555555555555, 'budget_type': 'epochs', 'pipeline_config': {'device': 'cuda', 'budget_type': 'epochs', 'epochs': 50, 'runtime': 3600, 'torch_num_threads': 1, 'early_stopping': 20, 'use_tensorboard_logger': False, 'metrics_during_training': True, 'optimize_metric': 'mean_MASE_forecasting'}, 'logger_port': 61103, 'all_supported_metrics': True, 'search_space_updates': <autoPyTorch.utils.hyperparameter_search_space_update.HyperparameterSearchSpaceUpdates object at 0x7ff3b2664f10>}
[DEBUG] [2022-08-07 11:08:00,319:Client-pynisher] Redirecting output of the function to files. Access them via the stdout and stderr attributes of the wrapped function.
[DEBUG] [2022-08-07 11:08:00,320:Client-pynisher] call function
[DEBUG] [2022-08-07 11:08:00,674:Client-TimeSeriesForecastingTrainEvaluator(1)] Fit dictionary in Abstract evaluator: dataset_properties: {'input_shape': (94, 0), 'categorical_features': [], 'numerical_columns': [], 'feature_shapes': {}, 'n_prediction_steps': 3, 'numerical_features': [], 'output_shape': [3, 1], 'feature_names': (), 'categories': [], 'freq': '1Y', 'sequence_lengths_train': array([94]), 'is_small_preprocess': True, 'task_type': 'time_series_forecasting', 'time_feature_names': ('time_feature_Constant',), 'known_future_features': (), 'categorical_columns': [], 'static_features': (), 'output_type': 'continuous', 'issparse': False, 'sp': 1, 'time_feature_transform': [Constant()], 'uni_variant': True, 'static_features_shape': 0, 'future_feature_shapes': (3, 0), 'targets_have_missing_values': False, 'encoder_can_be_auto_regressive': True, 'features_have_missing_values': False}
additional_metrics: ['mean_MASE_forecasting', 'mean_MASE_forecasting', 'median_MASE_forecasting', 'mean_MAE_forecasting', 'median_MAE_forecasting', 'mean_MAPE_forecasting', 'median_MAPE_forecasting', 'mean_MSE_forecasting', 'median_MSE_forecasting']
X_train: None
y_train:                0
0   1.000000e-26
0   1.000000e-26
0   1.000000e-26
0   1.000000e-26
0   1.000000e-26
..           ...
0   4.200000e+01
0   4.300000e+01
0   4.700000e+01
0   5.000000e+01
0   5.700000e+01

[97 rows x 1 columns]
X_test: None
y_test: [[58.]
 [60.]
 [64.]]
backend: <autoPyTorch.automl_common.common.utils.backend.Backend object at 0x7f2c5a4bbca0>
logger_port: 61103
optimize_metric: mean_MASE_forecasting
device: cuda
budget_type: epochs
epochs: 5.555555555555555
torch_num_threads: 1
early_stopping: 20
use_tensorboard_logger: False
metrics_during_training: True
[DEBUG] [2022-08-07 11:08:00,675:Client-TimeSeriesForecastingTrainEvaluator(1)] Search space updates :<autoPyTorch.utils.hyperparameter_search_space_update.HyperparameterSearchSpaceUpdates object at 0x7f2c5a4d00a0>
[DEBUG] [2022-08-07 11:08:00,675:Client-TimeSeriesForecastingTrainEvaluator(1)] Search space updates :<autoPyTorch.utils.hyperparameter_search_space_update.HyperparameterSearchSpaceUpdates object at 0x7f2c5a4d00a0>
[INFO] [2022-08-07 11:08:00,675:Client-TimeSeriesForecastingTrainEvaluator(1)] Starting fit 0
[DEBUG] [2022-08-07 11:08:01,991:Client-TimeSeriesForecastingTrainEvaluator(1)] /home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/pipeline/components/setup/network_initializer/XavierInit.py:27: UserWarning:nn.init.xavier_normal is now deprecated in favor of nn.init.xavier_normal_.
[DEBUG] [2022-08-07 11:08:03,129:Client-pynisher] function returned properly: (None, 0)
[DEBUG] [2022-08-07 11:08:03,129:Client-pynisher] return value: (None, 0)
[DEBUG] [2022-08-07 11:08:03,700:Client-TAE] Finish function evaluation 2.
Status: StatusType.CRASHED, Cost: 2147483647.0, Runtime: 5.444417238235474,
Additional information:
traceback: Traceback (most recent call last):
  File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/evaluation/tae.py", line 61, in fit_predict_try_except_decorator
    ta(queue=queue, **kwargs)
  File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/evaluation/time_series_forecasting_train_evaluator.py", line 558, in forecasting_eval_train_function
    evaluator.fit_predict_and_loss()
  File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/evaluation/time_series_forecasting_train_evaluator.py", line 176, in fit_predict_and_loss
    y_train_pred, y_opt_pred, y_valid_pred, y_test_pred = self._fit_and_predict(pipeline, split_id,
  File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/evaluation/train_evaluator.py", line 364, in _fit_and_predict
    fit_and_suppress_warnings(self.logger, pipeline, X, y)
  File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/evaluation/abstract_evaluator.py", line 338, in fit_and_suppress_warnings
    pipeline.fit(X, y)
  File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/pipeline/base_pipeline.py", line 158, in fit
    self.fit_estimator(X, y, **fit_params)
  File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/pipeline/base_pipeline.py", line 177, in fit_estimator
    self._final_estimator.fit(X, y, **fit_params)
  File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/pipeline/components/training/trainer/__init__.py", line 211, in fit
    self._fit(
  File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/pipeline/components/training/trainer/__init__.py", line 310, in _fit
    train_loss, train_metrics = self.choice.train_epoch(
  File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/pipeline/components/training/trainer/forecasting_trainer/forecasting_base_trainer.py", line 106, in train_epoch
    loss, outputs = self.train_step(data, targets)
  File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/pipeline/components/training/trainer/forecasting_trainer/forecasting_base_trainer.py", line 206, in train_step
    outputs = self.model(past_targets=past_target,
  File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/pipeline/components/setup/network/forecasting_architecture.py", line 602, in forward
    output = self.head(decoder_output)
  File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/pipeline/components/setup/network_head/forecasting_network_head/distribution.py", line 100, in forward
    return self.dist_cls(*self.domain_map(*params_unbounded))
  File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/torch/distributions/studentT.py", line 50, in __init__
    self._chi2 = Chi2(self.df)
  File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/torch/distributions/chi2.py", line 22, in __init__
    super(Chi2, self).__init__(0.5 * df, 0.5, validate_args=validate_args)
  File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/torch/distributions/gamma.py", line 52, in __init__
    super(Gamma, self).__init__(batch_shape, validate_args=validate_args)
  File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/torch/distributions/distribution.py", line 55, in __init__
    raise ValueError(
ValueError: Expected parameter df (Tensor of shape (32, 3, 1)) of distribution Chi2() to satisfy the constraint GreaterThan(lower_bound=0.0), but found invalid values:
tensor([[[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]],

        [[nan],
         [nan],
         [nan]]], device='cuda:0', grad_fn=<MulBackward0>)

error: ValueError("Expected parameter df (Tensor of shape (32, 3, 1)) of distribution Chi2() to satisfy the constraint GreaterThan(lower_bound=0.0), but found invalid values:\ntensor([[[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]],\n\n        [[nan],\n         [nan],\n         [nan]]], device='cuda:0', grad_fn=<MulBackward0>)")

Further information are recorded in following log file:
nan-error.log

Your Local environment

• Operating System, version
  Ubuntu 20.04
• Python, version
  Python 3.8
• Outputs of pip freeze or conda list
  pytorch=1.12+cu116

Supplementary Information

Covid Death dataset is uni-variate dataset, to give more evidence about what's really happening behind this error, the following
are the print-out dataframe of y_train and y_test. (In order to form a complete control group following the format of example code, I further modify the data frequency from '1D' to '1Y' to be identical with Longley dataset just for debug purpose.)
nan-error-y-info.txt

[dengdifan]

Hi,
does this NAN errors happens for all the configurations or just this single configuration? Sometimes this is simply the result of an unsuited hyperparameter configuration that result in an unstable training.

[RobbyW551]

Hi!
I've just done several experiments, it shows that among the 9 initial configurations, the first 6 are crashed and the last 3 survived. The result is stable over repetitive runs. I don't know if this is normal.

The full log file and run history json file are as below:
AutoPyTorch_1c0e9c2e-1719-11ed-8c29-312016f08e28_1.log
runhistory.json.txt

[dengdifan]

The problem is a bit complicated; this dataset only contains integral values. Therefore, it is considered a 'multiclass' dataset and all its 0 values are transformed into a very small value. Thereby, the target values are not properly scaled.
As a temporary solution, you could let this judgment always return False: https://github.com/automl/Auto-PyTorch/blob/master/autoPyTorch/data/tabular_target_validator.py#L184. Alternatively, you could set the argument normalize_yas True within api.search

[RobbyW551]

Hi!
The error is gone after setting normalize_y as True, but as I'm trying to reproduce the time-series forecasting experiment result in the paper, and current implementation of auto-pytorch is not capable of time-series classification, I think it might not be the case, maybe there are connections I lose track of.
After several experiment, I found that the MASE loss value in log file sometimes appears to be negative, which is really confusing. The following log file records several such training sessions.
latest.log

a snapshot of one of the training sessions in the log file above.

[INFO] [2022-08-09 22:23:55,570:Client-TAE] Starting to evaluate configuration 1
[DEBUG] [2022-08-09 22:23:55,573:Client-TAE] Search space updates for 2: <autoPyTorch.utils.hyperparameter_search_space_update.HyperparameterSearchSpaceUpdates object at 0x7f4ab691ebe0>
[DEBUG] [2022-08-09 22:23:55,573:Client-pynisher] Restricting your function to 499 seconds wall time.
[DEBUG] [2022-08-09 22:23:55,573:Client-pynisher] Allowing a grace period of 0 seconds.
[DEBUG] [2022-08-09 22:23:55,573:Client-pynisher] Function called with argument: (), {'queue': <multiprocessing.queues.Queue object at 0x7f4ab6878e20>, 'config': Configuration(values={
  'data_loader:backcast': False,
  'data_loader:batch_size': 32,
  'data_loader:num_batches_per_epoch': 50,
  'data_loader:sample_strategy': 'SeqUniform',
  'data_loader:transform_time_features': False,
  'data_loader:window_size': 9,
  'loss:DistributionLoss:aggregation': 'median',
  'loss:DistributionLoss:dist_cls': 'studentT',
  'loss:DistributionLoss:forecast_strategy': 'sample',
  'loss:DistributionLoss:num_samples': 100,
  'loss:__choice__': 'DistributionLoss',
  'lr_scheduler:ReduceLROnPlateau:factor': 0.5,
  'lr_scheduler:ReduceLROnPlateau:mode': 'max',
  'lr_scheduler:ReduceLROnPlateau:patience': 10,
  'lr_scheduler:__choice__': 'ReduceLROnPlateau',
  'network_backbone:__choice__': 'flat_encoder',
  'network_backbone:flat_encoder:MLPDecoder:has_local_layer': True,
  'network_backbone:flat_encoder:MLPDecoder:num_layers': 0,
  'network_backbone:flat_encoder:MLPDecoder:units_local_layer': 40,
  'network_backbone:flat_encoder:MLPEncoder:activation': 'relu',
  'network_backbone:flat_encoder:MLPEncoder:normalization': 'NoNorm',
  'network_backbone:flat_encoder:MLPEncoder:num_groups': 1,
  'network_backbone:flat_encoder:MLPEncoder:num_units_1': 40,
  'network_backbone:flat_encoder:MLPEncoder:use_dropout': False,
  'network_backbone:flat_encoder:__choice__': 'MLPEncoder',
  'network_embedding:__choice__': 'NoEmbedding',
  'network_init:XavierInit:bias_strategy': 'Normal',
  'network_init:__choice__': 'XavierInit',
  'optimizer:AdamOptimizer:beta1': 0.9,
  'optimizer:AdamOptimizer:beta2': 0.999,
  'optimizer:AdamOptimizer:lr': 0.001,
  'optimizer:AdamOptimizer:weight_decay': 1e-08,
  'optimizer:__choice__': 'AdamOptimizer',
  'target_scaler:scaling_mode': 'mean_abs',
  'trainer:__choice__': 'ForecastingStandardTrainer',
})
, 'backend': <autoPyTorch.automl_common.common.utils.backend.Backend object at 0x7f4ab6766850>, 'metric': mean_MASE_forecasting, 'seed': 1, 'num_run': 2, 'output_y_hat_optimization': True, 'include': {}, 'exclude': {}, 'disable_file_output': [], 'instance': '{"task_id": "e2d24e1c-17ee-11ed-9a73-2f4d5d9d434e"}', 'init_params': {'instance': '{"task_id": "e2d24e1c-17ee-11ed-9a73-2f4d5d9d434e"}'}, 'budget': 5.555555555555555, 'budget_type': 'epochs', 'pipeline_config': {'device': 'cuda', 'budget_type': 'epochs', 'epochs': 50, 'runtime': 3600, 'torch_num_threads': 1, 'early_stopping': 20, 'use_tensorboard_logger': False, 'metrics_during_training': True, 'optimize_metric': 'mean_MASE_forecasting'}, 'logger_port': 47969, 'all_supported_metrics': True, 'search_space_updates': <autoPyTorch.utils.hyperparameter_search_space_update.HyperparameterSearchSpaceUpdates object at 0x7f4ab691ebe0>}
[DEBUG] [2022-08-09 22:23:58,146:Client-pynisher] Redirecting output of the function to files. Access them via the stdout and stderr attributes of the wrapped function.
[DEBUG] [2022-08-09 22:23:58,147:Client-pynisher] call function
[DEBUG] [2022-08-09 22:23:58,504:Client-TimeSeriesForecastingTrainEvaluator(1)] Fit dictionary in Abstract evaluator: dataset_properties: {'categorical_features': [], 'static_features': (), 'task_type': 'time_series_forecasting', 'numerical_features': [], 'categories': [], 'is_small_preprocess': True, 'known_future_features': (), 'categorical_columns': [], 'numerical_columns': [], 'freq': '1D', 'n_prediction_steps': 30, 'sequence_lengths_train': array([152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
       152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
       152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
       152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
       152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
       152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
       152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
       152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
       152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
       152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
       152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
       152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
       152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
       152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
       152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
       152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
       152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
       152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
       152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
       152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
       152, 152, 152, 152, 152, 152]), 'feature_names': (), 'input_shape': (152, 0), 'output_shape': [30, 1], 'time_feature_names': ('time_feature_DayOfWeek', 'time_feature_DayOfMonth', 'time_feature_DayOfYear'), 'feature_shapes': {}, 'output_type': 'continuous', 'issparse': False, 'sp': 7, 'time_feature_transform': [DayOfWeek(), DayOfMonth(), DayOfYear()], 'uni_variant': True, 'static_features_shape': 0, 'future_feature_shapes': (30, 0), 'targets_have_missing_values': False, 'encoder_can_be_auto_regressive': True, 'features_have_missing_values': False}
additional_metrics: ['mean_MASE_forecasting', 'mean_MASE_forecasting', 'median_MASE_forecasting', 'mean_MAE_forecasting', 'median_MAE_forecasting', 'mean_MAPE_forecasting', 'median_MAPE_forecasting', 'mean_MSE_forecasting', 'median_MSE_forecasting']
X_train: None
y_train:             0
0   -0.678603
0   -0.678603
0   -0.678603
0   -0.678603
0   -0.678603
..        ...
265  3.787003
265  3.974447
265  3.974447
265  4.161892
265  4.161892

[48412 rows x 1 columns]
X_test: None
y_test: [[ 2.84623863]
 [ 2.90844171]
 [ 2.94991043]
 ...
 [25.71804243]
 [27.40504549]
 [27.59249028]]
backend: <autoPyTorch.automl_common.common.utils.backend.Backend object at 0x7fe367e3fc70>
logger_port: 47969
optimize_metric: mean_MASE_forecasting
device: cuda
budget_type: epochs
epochs: 5.555555555555555
torch_num_threads: 1
early_stopping: 20
use_tensorboard_logger: False
metrics_during_training: True
[DEBUG] [2022-08-09 22:23:58,504:Client-TimeSeriesForecastingTrainEvaluator(1)] Search space updates :<autoPyTorch.utils.hyperparameter_search_space_update.HyperparameterSearchSpaceUpdates object at 0x7fe367e54070>
[DEBUG] [2022-08-09 22:23:58,504:Client-TimeSeriesForecastingTrainEvaluator(1)] Search space updates :<autoPyTorch.utils.hyperparameter_search_space_update.HyperparameterSearchSpaceUpdates object at 0x7fe367e54070>
[INFO] [2022-08-09 22:23:58,509:Client-TimeSeriesForecastingTrainEvaluator(1)] Starting fit 0
[DEBUG] [2022-08-09 22:23:59,786:Client-TimeSeriesForecastingTrainEvaluator(1)] /home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/pipeline/components/setup/network_initializer/XavierInit.py:27: UserWarning:nn.init.xavier_normal is now deprecated in favor of nn.init.xavier_normal_.
[DEBUG] [2022-08-09 22:24:06,842:Client-2_1660055039.7867444] 
========================================
		Epoch 1
========================================
	end_time: 1660055046.841151
	start_time: 1660055039.7924404
	test_loss: None
	test_metrics:
	train_loss: 0.5615030335262418
	train_metrics:
	val_loss: 0.7551549077033997
	val_metrics:
		mean_MAE_forecasting: 0.3713443875312805
		mean_MAPE_forecasting: 76028412166144.0
		mean_MASE_forecasting: 6.3204656801767305
		mean_MSE_forecasting: 0.5660822987556458
		median_MAE_forecasting: 0.12749794125556946
		median_MAPE_forecasting: 0.14768588542938232
		median_MASE_forecasting: 0.9045267487515163
		median_MSE_forecasting: 0.021999072283506393
========================================
[DEBUG] [2022-08-09 22:24:13,077:Client-2_1660055039.7867444] 
========================================
		Epoch 2
========================================
	end_time: 1660055053.075863
	start_time: 1660055046.84346
	test_loss: None
	test_metrics:
	train_loss: -0.21076665975153447
	train_metrics:
	val_loss: 0.37946489453315735
	val_metrics:
		mean_MAE_forecasting: 0.306547611951828
		mean_MAPE_forecasting: 34252896862208.0
		mean_MASE_forecasting: 5.483508303192071
		mean_MSE_forecasting: 0.47867074608802795
		median_MAE_forecasting: 0.06652390211820602
		median_MAPE_forecasting: 0.1471085548400879
		median_MASE_forecasting: 0.5327428760948727
		median_MSE_forecasting: 0.006378884892910719
========================================
[DEBUG] [2022-08-09 22:24:19,345:Client-2_1660055039.7867444] 
========================================
		Epoch 3
========================================
	end_time: 1660055059.3446581
	start_time: 1660055053.0779555
	test_loss: None
	test_metrics:
	train_loss: -0.4158153299242258
	train_metrics:
	val_loss: 0.11834084987640381
	val_metrics:
		mean_MAE_forecasting: 0.27670568227767944
		mean_MAPE_forecasting: 21054133108736.0
		mean_MASE_forecasting: 5.087560038876242
		mean_MSE_forecasting: 0.42972803115844727
		median_MAE_forecasting: 0.04717379808425903
		median_MAPE_forecasting: 0.1136779934167862
		median_MASE_forecasting: 0.4350006861389294
		median_MSE_forecasting: 0.0034421852324157953
========================================
[DEBUG] [2022-08-09 22:24:25,618:Client-2_1660055039.7867444] 
========================================
		Epoch 4
========================================
	end_time: 1660055065.6175072
	start_time: 1660055059.3467505
	test_loss: None
	test_metrics:
	train_loss: -0.5240315041341819
	train_metrics:
	val_loss: 0.09902344644069672
	val_metrics:
		mean_MAE_forecasting: 0.27489030361175537
		mean_MAPE_forecasting: 18999616208896.0
		mean_MASE_forecasting: 5.04518199855496
		mean_MSE_forecasting: 0.42932283878326416
		median_MAE_forecasting: 0.050067659467458725
		median_MAPE_forecasting: 0.11365534365177155
		median_MASE_forecasting: 0.442323123342876
		median_MSE_forecasting: 0.0035072178579866886
========================================
[DEBUG] [2022-08-09 22:24:32,030:Client-2_1660055039.7867444] 
========================================
		Epoch 5
========================================
	end_time: 1660055072.0294464
	start_time: 1660055065.6196034
	test_loss: None
	test_metrics:
	train_loss: -0.5758035796135664
	train_metrics:
	val_loss: 0.018720028921961784
	val_metrics:
		mean_MAE_forecasting: 0.2657284140586853
		mean_MAPE_forecasting: 14370088157184.0
		mean_MASE_forecasting: 4.9683659710981765
		mean_MSE_forecasting: 0.42335325479507446
		median_MAE_forecasting: 0.03455820679664612
		median_MAPE_forecasting: 0.11295084655284882
		median_MASE_forecasting: 0.3162790758898065
		median_MSE_forecasting: 0.0017664149636402726
========================================
[DEBUG] [2022-08-09 22:24:32,030:Client-2_1660055039.7867444]  Early stopped model 2 on epoch 5
[INFO] [2022-08-09 22:24:32,032:Client-2_1660055039.7867444] Finished training with 
========================================
		Epoch 5
========================================
	end_time: 1660055072.0294464
	start_time: 1660055065.6196034
	test_loss: None
	test_metrics:
	train_loss: -0.5758035796135664
	train_metrics:
	val_loss: 0.018720028921961784
	val_metrics:
		mean_MAE_forecasting: 0.2657284140586853
		mean_MAPE_forecasting: 14370088157184.0
		mean_MASE_forecasting: 4.9683659710981765
		mean_MSE_forecasting: 0.42335325479507446
		median_MAE_forecasting: 0.03455820679664612
		median_MAPE_forecasting: 0.11295084655284882
		median_MASE_forecasting: 0.3162790758898065
		median_MSE_forecasting: 0.0017664149636402726
========================================

[dengdifan]

if you want to reproduce the results from the paper, you could use the code from this branch: https://github.com/dengdifan/Auto-PyTorch/tree/ecml22_apt_ts (I did some modifications to make it compatible with the current workflow of AutoPyTorch).

The metric 'MASE' is only applied to evaluate the trained model as a loss that will be returned to SMAC. Depending on the head type, networks could be trained with different losses (for instance, this implementation is trained with negative log-likelihood loss (DistributionLoss)). Normally I would recommend not evaluating validation metrics at each iteration, as it might be too expensive for some auto-regressive networks to do inference at every iteration (for the same reason, the val_loss in the log file does not really correspond to the validation loss)

[RobbyW551]

Thanks for your reply. I will try to run the code you provided.

[ravinkohli]

I am closing this issue due to inactivity. Feel free to reopen if the issue persists.