Trainer cannot handle 1d tensor when return results from test_epoch_end

[zhiruiluo]

🐛 Bug

When trainer run_test() called, the results from test cannot properly handle a 1D tensor in the results dictionary.

Such error will happen:

/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py in run_test(self)
708 for k, v in result.items():
709 if isinstance(v, torch.Tensor):
--> 710 result[k] = v.cpu().item()
711
712 return eval_loop_results

ValueError: only one element tensors can be converted to Python scalars

Please reproduce using the BoringModel

To Reproduce

To reproduce with BoringModel, only need to replace the test_epoch_end.

def test_epoch_end(self, outputs) -> None:
    torch.stack([x["y"] for x in outputs]).mean()
    f1_score = torch.tensor([1,1,1,1])
    return {'f1_score': f1_score}

Expected behavior

def run_test(self):

        # remove the tensors from the eval results
        for i, result in enumerate(eval_loop_results):
            if isinstance(result, dict):
                for k, v in result.items():
                    if isinstance(v, torch.Tensor):
                        # should check if you can call .item()
                        result[k] = v.cpu().item()

Environment

• PyTorch Version (e.g., 1.0): 1.1.8

Additional context

[awaelchli]

Hi

You should see somewhere a warning:

UserWarning: The testing_epoch_end should not return anything as of 9.1. To log, use self.log(...) or self.write(...) directly in the LightningModule

Only scalar tensors are supported.

[zhiruiluo]

Hi

You should see somewhere a warning:

UserWarning: The testing_epoch_end should not return anything as of 9.1. To log, use self.log(...) or self.write(...) directly in the LightningModule

Only scalar tensors are supported.

I think returning from testing_epoch_end is a very useful functionality when you want to do any post-processing with the results.
Why is thie not recommended? Is there any potential bug to do so?

[awaelchli]

Because for example in multi gpu mode, if we would allow the user to return, then we're missing the information what to do with the data, how the data is collected and synced, or reduced or whatever.
The logging api offers reduction and sync, by specifying the custom arguments how to do so.
On the other hand, self.write offers a way to collect all results.
There will also be a prediction api in 1.2. #5752
cc @tchaton

[zhiruiluo]

Thank for the explaination. I will try the self.write. Could you please refer me to the page of self.write?

[awaelchli]

Sorry, it's not called "write", the warning seems to have the wrong name. There is currently write_prediction and write_prediction_dict
https://pytorch-lightning.readthedocs.io/en/latest/common/lightning_module.html#write-prediction