In pytorch_lightning 1.6.1 torch.backends.cudnn.benchmark gets overwritten with incorrect value #14149
Description
🐛 Bug
In pytorch_lightning 1.6.1, Trainer does not use the current value of torch.backends.cudnn.benchmark when the input parameter benchmark is not set (i.e., takes its default value (benchmark = None).
Important Note: This bug seems to be fixed in the latest version.
To Reproduce
The following colab notebook reproduces the bug: https://colab.research.google.com/drive/1ccysr3jjeoqRrlPJJNQB5xQiX7n16f1X?usp=sharing
Description: Before initializing the Trainer object if you have the following cuddnn backend variables set as:
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.enabled = True
torch.backends.cudnn.deterministic = False
and then you initialize the Trainer object without explicitly specifying the backend parameter, i.e.,:
trainer = pl.Trainer(accelerator="gpu", devices=1, logger=tb_logger, callbacks=callbacks) #
Lightning will change the value of torch.backends.cudnn.benchmark to True. This can be verified by printing the value of torch.backends.cudnn.benchmark while training.
Expected behavior
Pytorch lightning should not change torch.backends.cudnn.benchmark if no value was given to the Trainer parameter.
Environment
Details
* CUDA: - GPU: - Tesla T4 - available: True - version: 11.3 * Lightning: - pytorch-lightning: 1.6.1 - torch: 1.12.0+cu113 - torchaudio: 0.12.0+cu113 - torchmetrics: 0.9.3 - torchsummary: 1.5.1 - torchtext: 0.13.0 - torchvision: 0.13.0+cu113 * Packages: - absl-py: 1.2.0 - aiohttp: 3.8.1 - aiosignal: 1.2.0 - alabaster: 0.7.12 - albumentations: 1.2.1 - altair: 4.2.0 - appdirs: 1.4.4 - argon2-cffi: 21.3.0 - argon2-cffi-bindings: 21.2.0 - arviz: 0.12.1 - astor: 0.8.1 - astropy: 4.3.1 - astunparse: 1.6.3 - async-timeout: 4.0.2 - asynctest: 0.13.0 - atari-py: 0.2.9 - atomicwrites: 1.4.1 - attrs: 22.1.0 - audioread: 2.1.9 - autograd: 1.4 - babel: 2.10.3 - backcall: 0.2.0 - beautifulsoup4: 4.6.3 - bleach: 5.0.1 - blis: 0.7.8 - bokeh: 2.3.3 - branca: 0.5.0 - bs4: 0.0.1 - cachecontrol: 0.12.11 - cached-property: 1.5.2 - cachetools: 4.2.4 - catalogue: 2.0.8 - certifi: 2022.6.15 - cffi: 1.15.1 - cftime: 1.6.1 - chardet: 3.0.4 - charset-normalizer: 2.1.0 - click: 7.1.2 - clikit: 0.6.2 - cloudpickle: 1.3.0 - cmake: 3.22.6 - cmdstanpy: 1.0.4 - colorcet: 3.0.0 - colorlover: 0.3.0 - community: 1.0.0b1 - contextlib2: 0.5.5 - convertdate: 2.4.0 - crashtest: 0.3.1 - crcmod: 1.7 - cufflinks: 0.17.3 - cupy-cuda111: 9.4.0 - cvxopt: 1.3.0 - cvxpy: 1.2.1 - cycler: 0.11.0 - cymem: 2.0.6 - cython: 0.29.32 - daft: 0.0.4 - dask: 2.12.0 - datascience: 0.17.5 - debugpy: 1.0.0 - decorator: 4.4.2 - defusedxml: 0.7.1 - deprecat: 2.1.1 - descartes: 1.1.0 - dill: 0.3.5.1 - distributed: 1.25.3 - dlib: 19.24.0 - dm-tree: 0.1.7 - docutils: 0.17.1 - dopamine-rl: 1.0.5 - earthengine-api: 0.1.317 - easydict: 1.9 - ecos: 2.0.10 - editdistance: 0.5.3 - en-core-web-sm: 3.4.0 - entrypoints: 0.4 - ephem: 4.1.3 - et-xmlfile: 1.1.0 - etils: 0.6.0 - fa2: 0.3.5 - fastai: 2.7.7 - fastcore: 1.5.13 - fastdownload: 0.0.7 - fastdtw: 0.3.4 - fastjsonschema: 2.16.1 - fastprogress: 1.0.3 - fastrlock: 0.8 - feather-format: 0.4.1 - filelock: 3.7.1 - firebase-admin: 4.4.0 - fix-yahoo-finance: 0.0.22 - flask: 1.1.4 - flatbuffers: 2.0 - folium: 0.12.1.post1 - frozenlist: 1.3.0 - fsspec: 2022.7.1 - future: 0.16.0 - gast: 0.5.3 - gdal: 2.2.2 - gdown: 4.4.0 - gensim: 3.6.0 - geographiclib: 1.52 - geopy: 1.17.0 - gin-config: 0.5.0 - glob2: 0.7 - google: 2.0.3 - google-api-core: 1.31.6 - google-api-python-client: 1.12.11 - google-auth: 1.35.0 - google-auth-httplib2: 0.0.4 - google-auth-oauthlib: 0.4.6 - google-cloud-bigquery: 1.21.0 - google-cloud-bigquery-storage: 1.1.2 - google-cloud-core: 1.0.3 - google-cloud-datastore: 1.8.0 - google-cloud-firestore: 1.7.0 - google-cloud-language: 1.2.0 - google-cloud-storage: 1.18.1 - google-cloud-translate: 1.5.0 - google-colab: 1.0.0 - google-pasta: 0.2.0 - google-resumable-media: 0.4.1 - googleapis-common-protos: 1.56.4 - googledrivedownloader: 0.4 - graphviz: 0.10.1 - greenlet: 1.1.2 - grpcio: 1.47.0 - gspread: 3.4.2 - gspread-dataframe: 3.0.8 - gym: 0.17.3 - h5py: 3.1.0 - heapdict: 1.0.1 - hijri-converter: 2.2.4 - holidays: 0.14.2 - holoviews: 1.14.9 - html5lib: 1.0.1 - httpimport: 0.5.18 - httplib2: 0.17.4 - httplib2shim: 0.0.3 - httpstan: 4.6.1 - humanize: 0.5.1 - hyperopt: 0.1.2 - idna: 2.10 - imageio: 2.9.0 - imagesize: 1.4.1 - imbalanced-learn: 0.8.1 - imblearn: 0.0 - imgaug: 0.4.0 - importlib-metadata: 4.12.0 - importlib-resources: 5.9.0 - imutils: 0.5.4 - inflect: 2.1.0 - intel-openmp: 2022.1.0 - intervaltree: 2.1.0 - ipykernel: 4.10.1 - ipython: 5.5.0 - ipython-genutils: 0.2.0 - ipython-sql: 0.3.9 - ipywidgets: 7.7.1 - itsdangerous: 1.1.0 - jax: 0.3.14 - jaxlib: 0.3.14+cuda11.cudnn805 - jedi: 0.18.1 - jieba: 0.42.1 - jinja2: 2.11.3 - joblib: 1.1.0 - jpeg4py: 0.1.4 - jsonschema: 4.3.3 - jupyter: 1.0.0 - jupyter-client: 5.3.5 - jupyter-console: 5.2.0 - jupyter-core: 4.11.1 - jupyterlab-pygments: 0.2.2 - jupyterlab-widgets: 1.1.1 - kaggle: 1.5.12 - kapre: 0.3.7 - keras: 2.8.0 - keras-preprocessing: 1.1.2 - keras-vis: 0.4.1 - kiwisolver: 1.4.4 - korean-lunar-calendar: 0.2.1 - langcodes: 3.3.0 - libclang: 14.0.6 - librosa: 0.8.1 - lightgbm: 2.2.3 - llvmlite: 0.39.0 - lmdb: 0.99 - lunarcalendar: 0.0.9 - lxml: 4.9.1 - markdown: 3.4.1 - markupsafe: 2.0.1 - marshmallow: 3.17.0 - matplotlib: 3.2.2 - matplotlib-inline: 0.1.3 - matplotlib-venn: 0.11.7 - missingno: 0.5.1 - mistune: 0.8.4 - mizani: 0.6.0 - mkl: 2019.0 - mlxtend: 0.14.0 - more-itertools: 8.13.0 - moviepy: 0.2.3.5 - mpmath: 1.2.1 - msgpack: 1.0.4 - multidict: 6.0.2 - multitasking: 0.0.11 - murmurhash: 1.0.7 - music21: 5.5.0 - natsort: 5.5.0 - nbclient: 0.6.6 - nbconvert: 5.6.1 - nbformat: 5.4.0 - nest-asyncio: 1.5.5 - netcdf4: 1.6.0 - networkx: 2.6.3 - nibabel: 3.0.2 - nltk: 3.7 - notebook: 5.3.1 - numba: 0.56.0 - numexpr: 2.8.3 - numpy: 1.21.6 - oauth2client: 4.1.3 - oauthlib: 3.2.0 - okgrade: 0.4.3 - opencv-contrib-python: 4.6.0.66 - opencv-python: 4.6.0.66 - opencv-python-headless: 4.6.0.66 - openpyxl: 3.0.10 - opt-einsum: 3.3.0 - osqp: 0.6.2.post0 - packaging: 21.3 - palettable: 3.3.0 - pandas: 1.3.5 - pandas-datareader: 0.9.0 - pandas-gbq: 0.13.3 - pandas-profiling: 1.4.1 - pandocfilters: 1.5.0 - panel: 0.12.1 - param: 1.12.2 - parso: 0.8.3 - pastel: 0.2.1 - pathlib: 1.0.1 - pathy: 0.6.2 - patsy: 0.5.2 - pep517: 0.13.0 - pexpect: 4.8.0 - pickleshare: 0.7.5 - pillow: 7.1.2 - pip: 21.1.3 - pip-tools: 6.2.0 - pkgutil-resolve-name: 1.3.10 - plotly: 5.5.0 - plotnine: 0.6.0 - pluggy: 0.7.1 - pooch: 1.6.0 - portpicker: 1.3.9 - prefetch-generator: 1.0.1 - preshed: 3.0.6 - prettytable: 3.3.0 - progressbar2: 3.38.0 - prometheus-client: 0.14.1 - promise: 2.3 - prompt-toolkit: 1.0.18 - prophet: 1.1 - protobuf: 3.17.3 - psutil: 5.4.8 - psycopg2: 2.9.3 - ptyprocess: 0.7.0 - py: 1.11.0 - pyarrow: 6.0.1 - pyasn1: 0.4.8 - pyasn1-modules: 0.2.8 - pycocotools: 2.0.4 - pycparser: 2.21 - pyct: 0.4.8 - pydantic: 1.9.1 - pydata-google-auth: 1.4.0 - pydeprecate: 0.3.2 - pydot: 1.3.0 - pydot-ng: 2.0.0 - pydotplus: 2.0.2 - pydrive: 1.3.1 - pyemd: 0.5.1 - pyerfa: 2.0.0.1 - pyglet: 1.5.0 - pygments: 2.6.1 - pygobject: 3.26.1 - pylev: 1.4.0 - pymc3: 3.11.5 - pymeeus: 0.5.11 - pymongo: 4.2.0 - pymystem3: 0.2.0 - pyopengl: 3.1.6 - pyparsing: 3.0.9 - pyrsistent: 0.18.1 - pysimdjson: 3.2.0 - pysndfile: 1.3.8 - pysocks: 1.7.1 - pystan: 3.3.0 - pytest: 3.6.4 - python-apt: 0.0.0 - python-chess: 0.23.11 - python-dateutil: 2.8.2 - python-louvain: 0.16 - python-slugify: 6.1.2 - python-utils: 3.3.3 - pytorch-lightning: 1.6.1 - pytz: 2022.1 - pyviz-comms: 2.2.0 - pywavelets: 1.3.0 - pyyaml: 6.0 - pyzmq: 23.2.0 - qdldl: 0.1.5.post2 - qtconsole: 5.3.1 - qtpy: 2.1.0 - qudida: 0.0.4 - regex: 2022.6.2 - requests: 2.23.0 - requests-oauthlib: 1.3.1 - resampy: 0.3.1 - rpy2: 3.4.5 - rsa: 4.9 - scikit-image: 0.18.3 - scikit-learn: 1.0.2 - scipy: 1.7.3 - screen-resolution-extra: 0.0.0 - scs: 3.2.0 - seaborn: 0.11.2 - semver: 2.13.0 - send2trash: 1.8.0 - setuptools: 57.4.0 - setuptools-git: 1.2 - shapely: 1.8.2 - simplegeneric: 0.8.1 - six: 1.15.0 - sklearn: 0.0 - sklearn-pandas: 1.8.0 - smart-open: 5.2.1 - snowballstemmer: 2.2.0 - sortedcontainers: 2.4.0 - soundfile: 0.10.3.post1 - soupsieve: 2.3.2.post1 - spacy: 3.4.1 - spacy-legacy: 3.0.9 - spacy-loggers: 1.0.3 - sphinx: 1.8.6 - sphinxcontrib-serializinghtml: 1.1.5 - sphinxcontrib-websupport: 1.2.4 - sqlalchemy: 1.4.39 - sqlparse: 0.4.2 - srsly: 2.4.4 - statsmodels: 0.10.2 - sympy: 1.7.1 - tables: 3.7.0 - tabulate: 0.8.10 - tblib: 1.7.0 - tenacity: 8.0.1 - tensorboard: 2.9.1 - tensorboard-data-server: 0.6.1 - tensorboard-plugin-wit: 1.8.1 - tensorflow: 2.8.2+zzzcolab20220719082949 - tensorflow-datasets: 4.6.0 - tensorflow-estimator: 2.8.0 - tensorflow-gcs-config: 2.8.0 - tensorflow-hub: 0.12.0 - tensorflow-io-gcs-filesystem: 0.26.0 - tensorflow-metadata: 1.9.0 - tensorflow-probability: 0.16.0 - termcolor: 1.1.0 - terminado: 0.13.3 - testpath: 0.6.0 - text-unidecode: 1.3 - textblob: 0.15.3 - theano-pymc: 1.1.2 - thinc: 8.1.0 - threadpoolctl: 3.1.0 - tifffile: 2021.11.2 - tinycss2: 1.1.1 - toml: 0.10.2 - tomli: 2.0.1 - toolz: 0.12.0 - torch: 1.12.0+cu113 - torchaudio: 0.12.0+cu113 - torchmetrics: 0.9.3 - torchsummary: 1.5.1 - torchtext: 0.13.0 - torchvision: 0.13.0+cu113 - tornado: 5.1.1 - tqdm: 4.64.0 - traitlets: 5.1.1 - tweepy: 3.10.0 - typeguard: 2.7.1 - typer: 0.4.2 - typing-extensions: 4.1.1 - tzlocal: 1.5.1 - ujson: 5.4.0 - uritemplate: 3.0.1 - urllib3: 1.24.3 - vega-datasets: 0.9.0 - wasabi: 0.10.1 - wcwidth: 0.2.5 - webargs: 8.2.0 - webencodings: 0.5.1 - werkzeug: 1.0.1 - wheel: 0.37.1 - widgetsnbextension: 3.6.1 - wordcloud: 1.8.2.2 - wrapt: 1.14.1 - xarray: 0.20.2 - xarray-einstats: 0.2.2 - xgboost: 0.90 - xkit: 0.0.0 - xlrd: 1.1.0 - xlwt: 1.3.0 - yarl: 1.8.1 - yellowbrick: 1.4 - zict: 2.2.0 - zipp: 3.8.1 * System: - OS: Linux - architecture: - 64bit - - processor: x86_64 - python: 3.7.13 - version: #1 SMP Sun Apr 24 10:03:06 PDT 2022cc @awaelchli