Lightning logger aggregation API deprecation

[edward-io]

Proposed refactoring or deprecation

Move and/or deprecate aggregation-related code to individual loggers

Motivation

We are auditing the Lightning components and APIs to assess opportunities for improvements:

• https://docs.google.com/document/d/1xHU7-iQSpp9KJTjI3As2EM0mfNHHr37WZYpDpwLkivA/edit#

• Review Lightning architecture & API #7740

Revisiting some of the API decisions regarding aggregations in Lightning Logger to simplify the interface and move logic specific to individual loggers away from the base class:

• Aggregation is currently being done by the step dimension. Users may want more control over complex aggregations, like aggregating metrics over a time window.
• It's not clear to the end-user whether they should use agg_and_log_metrics or log_metrics. In LoggerConnector, when log_metrics is called, we call agg_and_log_metrics.
  • Which method(s) should developers override when creating their custom logger?
• LightningLoggerBase.__init__ accepts two arguments, agg_key_funcs, and agg_default_func. They aren't being called in any sub-classed loggers within Lightning. They can be implementation details of the loggers instead.
  • Do we know if users are adding aggregation functionality using update_agg_funcs directly?

Pitch

• Simplify LightningLoggerBase.__init__ by removing agg_key_funcs and agg_default_func
• Move aggregation-related logger code out of base class to simplify the interface
• Provide clarity between what the Lightning trainer calls between log_metrics and agg_and_log_metrics. Proposal: the trainer should only call log_metrics

Additional context

Related issues:

#8991

#9004

---

If you enjoy Lightning, check out our other projects! ⚡

• Metrics: Machine learning metrics for distributed, scalable PyTorch applications.

• Flash: The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, finetuning and solving problems with deep learning

• Bolts: Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

• Lightning Transformers: Flexible interface for high performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.

[ananthsub]

Thanks for filing this @edward-io !

I believe this is very related to #8991 and #9004

I agree that we can simplify the base interface for logging metrics. At the very least I see:

• log_metrics -> can internally aggregate/buffer data before logging
• teardown -> flushes any buffered data and closes the experiment

Do we need a flush interface at all on the base logger in this case?

@awaelchli @tchaton @carmocca ?

[awaelchli]

Move aggregation-related logger code out of base class to simplify the interface

where would you move it?

Provide clarity between what the Lightning trainer calls between log_metrics and agg_and_log_metrics. Proposal: the trainer should only call log_metrics

sounds good as long, as we keep exactly the same logging behavior

Do we need a flush interface at all on the base logger in this case?

what we know as flush in the logger connector is just the save() function in the loggers.

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!