Adding dnn based super resolution module.

modules/dnn_superres/CMakeLists.txt

@@ -0,0 +1,5 @@
+set(the_description "Super Resolution using CNNs")
+
+ocv_define_module(dnn_superres opencv_core opencv_imgproc opencv_dnn
+    OPTIONAL opencv_highgui opencv_imgcodecs opencv_datasets  # samples
+)

modules/dnn_superres/README.md

@@ -0,0 +1,93 @@
+# Super Resolution using Convolutional Neural Networks
+
+This module contains several learning-based algorithms for upscaling an image.
+
+## Usage
+
+Run the following command to build this module:
+
+```make
+cmake -DOPENCV_EXTRA_MODULES_PATH=<opencv_contrib>/modules -Dopencv_dnn_superres=ON <opencv_source_dir>
+```
+
+Refer to the tutorials to understand how to use this module.
+
+## Models
+
+There are four models which are trained.
+
+#### EDSR
+
+Trained models can be downloaded from [here](https://github.com/Saafke/EDSR_Tensorflow/tree/master/models).
+
+- Size of the model: ~38.5MB. This is a quantized version, so that it can be uploaded to GitHub. (Original was 150MB.)
+- This model was trained for 3 days with a batch size of 16
+- Link to implementation code: https://github.com/Saafke/EDSR_Tensorflow
+- x2, x3, x4 trained models available
+- Advantage: Highly accurate
+- Disadvantage: Slow and large filesize
+- Speed: < 3 sec for every scaling factor on 256x256 images on an Intel i7-9700K CPU.
+- Original paper: [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://arxiv.org/pdf/1707.02921.pdf) [1]
+
+#### ESPCN
+
+Trained models can be downloaded from [here](https://github.com/fannymonori/TF-ESPCN/tree/master/export).
+
+- Size of the model: ~100kb
+- This model was trained for ~100 iterations with a batch size of 32
+- Link to implementation code: https://github.com/fannymonori/TF-ESPCN
+- x2, x3, x4 trained models available
+- Advantage: It is tiny and fast, and still performs well.
+- Disadvantage: Perform worse visually than newer, more robust models.
+- Speed: < 0.01 sec for every scaling factor on 256x256 images on an Intel i7-9700K CPU.
+- Original paper: [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](<https://arxiv.org/abs/1609.05158>) [2]
+
+#### FSRCNN
+
+Trained models can be downloaded from [here](https://github.com/Saafke/FSRCNN_Tensorflow/tree/master/models).
+
+- Size of the model: ~40KB (~9kb for FSRCNN-small)
+- This model was trained for ~30 iterations with a batch size of 1
+- Link to implementation code: https://github.com/Saafke/FSRCNN_Tensorflow
+- Advantage: Fast, small and accurate
+- Disadvantage: Not state-of-the-art accuracy
+- Speed: < 0.01 sec for every scaling factor on 256x256 images on an Intel i7-9700K CPU.
+- Notes: FSRCNN-small has fewer parameters, thus less accurate but faster.
+- Original paper: [Accelerating the Super-Resolution Convolutional Neural Network](http://mmlab.ie.cuhk.edu.hk/projects/FSRCNN.html) [3]
+
+#### LapSRN
+
+Trained models can be downloaded from [here](https://github.com/fannymonori/TF-LapSRN/tree/master/export).
+
+- Size of the model: between 1-5Mb
+- This model was trained for ~50 iterations with a batch size of 32
+- Link to implementation code: https://github.com/fannymonori/TF-LAPSRN
+- x2, x4, x8 trained models available
+- Advantage: The model can do multi-scale super-resolution with one forward pass. It can now support 2x, 4x, 8x, and [2x, 4x] and [2x, 4x, 8x] super-resolution.
+- Disadvantage: It is slower than ESPCN and FSRCNN, and the accuracy is worse than EDSR.
+- Speed: < 0.1 sec for every scaling factor on 256x256 images on an Intel i7-9700K CPU.
+- Original paper: [Deep laplacian pyramid networks for fast and accurate super-resolution](<https://arxiv.org/abs/1710.01992>) [4]
+
+### Benchmarks
+
+Comparing different algorithms. Scale x4 on monarch.png.
+
+|               | Inference time in seconds (CPU)| PSNR | SSIM |
+| ------------- |:-------------------:| ---------:|--------:|
+| ESPCN            |0.01159   | 26.5471 | 0.88116 |
+| EDSR             |3.26758     |**29.2404**  |**0.92112**  |
+| FSRCNN           | 0.01298   | 26.5646 | 0.88064 |
+| LapSRN           |0.28257    |26.7330   |0.88622  |
+| Bicubic          |0.00031 |26.0635  |0.87537  |
+| Nearest neighbor |**0.00014** |23.5628  |0.81741  |
+| Lanczos          |0.00101  |25.9115  |0.87057  |
+
+### References
+[1] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee, **"Enhanced Deep Residual Networks for Single Image Super-Resolution"**, <i> 2nd NTIRE: New Trends in Image Restoration and Enhancement workshop and challenge on image super-resolution in conjunction with **CVPR 2017**. </i> [[PDF](http://openaccess.thecvf.com/content_cvpr_2017_workshops/w12/papers/Lim_Enhanced_Deep_Residual_CVPR_2017_paper.pdf)] [[arXiv](https://arxiv.org/abs/1707.02921)] [[Slide](https://cv.snu.ac.kr/research/EDSR/Presentation_v3(release).pptx)]
+
+[2] Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A., Bishop, R., Rueckert, D. and Wang, Z., **"Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network"**, <i>Proceedings of the IEEE conference on computer vision and pattern recognition</i> **CVPR 2016**. [[PDF](http://openaccess.thecvf.com/content_cvpr_2016/papers/Shi_Real-Time_Single_Image_CVPR_2016_paper.pdf)] [[arXiv](https://arxiv.org/abs/1609.05158)]
+
+[3] Chao Dong, Chen Change Loy, Xiaoou Tang. **"Accelerating the Super-Resolution Convolutional Neural Network"**, <i> in Proceedings of European Conference on Computer Vision </i>**ECCV 2016**. [[PDF](http://personal.ie.cuhk.edu.hk/~ccloy/files/eccv_2016_accelerating.pdf)]
+[[arXiv](https://arxiv.org/abs/1608.00367)] [[Project Page](http://mmlab.ie.cuhk.edu.hk/projects/FSRCNN.html)]
+
+[4] Lai, W. S., Huang, J. B., Ahuja, N., and Yang, M. H., **"Deep laplacian pyramid networks for fast and accurate super-resolution"**, <i> In Proceedings of the IEEE conference on computer vision and pattern recognition </i>**CVPR 2017**. [[PDF](http://openaccess.thecvf.com/content_cvpr_2017/papers/Lai_Deep_Laplacian_Pyramid_CVPR_2017_paper.pdf)] [[arXiv](https://arxiv.org/abs/1710.01992)] [[Project Page](http://vllab.ucmerced.edu/wlai24/LapSRN/)]

modules/dnn_superres/include/opencv2/dnn_superres.hpp

@@ -0,0 +1,159 @@
+// This file is part of OpenCV project.
+// It is subject to the license terms in the LICENSE file found in the top-level directory
+// of this distribution and at http://opencv.org/license.html.
+
+#ifndef _OPENCV_DNN_SUPERRES_DNNSUPERRESIMPL_HPP_
+#define _OPENCV_DNN_SUPERRES_DNNSUPERRESIMPL_HPP_
+
+#include <opencv2/highgui.hpp>
+#include <opencv2/imgproc.hpp>
+#include "opencv2/dnn.hpp"
+
+/** @defgroup dnn_superres DNN used for super resolution
+
+This module contains functionality for upscaling an image via convolutional neural networks.
+The following four models are implemented:
+
+- EDSR <https://arxiv.org/abs/1707.02921>
+- ESPCN <https://arxiv.org/abs/1609.05158>
+- FSRCNN <https://arxiv.org/abs/1608.00367>
+- LapSRN <https://arxiv.org/abs/1710.01992>
+
+*/
+
+namespace cv
+{
+namespace dnn_superres
+{
+    //! @addtogroup dnn_superres
+    //! @{
+
+    /** @brief A class to upscale images via convolutional neural networks.
+    The following four models are implemented:
+
+    - edsr
+    - espcn
+    - fsrcnn
+    - lapsrn
+     */
+    class CV_EXPORTS DnnSuperResImpl
+    {
+        private:
+
+            /** @brief Net which holds the desired neural network
+             */
+            dnn::Net net;
+
+            std::string alg; //algorithm
+
+            int sc; //scale factor
+
+            /// @private
+            static int layer_loaded;
+
+            void registerLayers();
+
+            void preprocess(const Mat inpImg, Mat &outpImg);
+
+            void reconstruct_YCrCb(const Mat inpImg, const Mat origImg, Mat &outpImg, int scale);
+
+            void reconstruct_YCrCb(const Mat inpImg, const Mat origImg, Mat &outpImg);
+
+            void preprocess_YCrCb(const Mat inpImg, Mat &outpImg);
+
+        public:
+
+            /** @brief Empty constructor
+             */
+            DnnSuperResImpl();
+
+            /** @brief Constructor which immediately sets the desired model
+            @param algo String containing one of the desired models:
+                - __edsr__
+                - __espcn__
+                - __fsrcnn__
+                - __lapsrn__
+            @param scale Integer specifying the upscale factor
+             */
+            DnnSuperResImpl(std::string algo, int scale);
+
+            /** @brief Read the model from the given path
+            @param path Path to the model file.
+            */
+            void readModel(std::string path);
+
+            /** @brief Read the model from the given path
+            @param weights Path to the model weights file.
+            @param definition Path to the model definition file.
+            */
+            void readModel(std::string weights, std::string definition);
+
+            /** @brief Set desired model
+            @param algo String containing one of the desired models:
+                - __edsr__
+                - __espcn__
+                - __fsrcnn__
+                - __lapsrn__
+            @param scale Integer specifying the upscale factor
+             */
+            void setModel(std::string algo, int scale);
+
+            /** @brief Upsample via neural network
+            @param img Image to upscale
+            @param img_new Destination upscaled image
+             */
+            void upsample(Mat img, Mat &img_new);
+
+            /** @brief Upsample via neural network of multiple outputs
+            @param img Image to upscale
+            @param imgs_new Destination upscaled images
+            @param scale_factors Scaling factors of the output nodes
+            @param node_names Names of the output nodes in the neural network
+            */
+            void upsampleMultioutput(Mat img, std::vector<Mat> &imgs_new, std::vector<int> scale_factors, std::vector<String> node_names);
+
+            /** @brief Upsamples videos via neural network and saves it into the given path.
+            @param inputPath Path to video to upscale
+            @param outputPath Destination upscaled video
+            */
+            void upsampleVideo(String inputPath, String outputPath);
+
+            /** @brief Returns the scale factor of the model:
+            @return Current scale factor.
+            */
+            int getScale();
+
+            /** @brief Returns the scale factor of the model:
+            @return Current algorithm.
+            */
+            std::string getAlgorithm();
+
+            private:
+            /** @brief Class for importing DepthToSpace layer from the ESPCN model
+            */
+            class DepthToSpace CV_FINAL : public cv::dnn::Layer
+            {
+                public:
+
+                /// @private
+                DepthToSpace(const cv::dnn::LayerParams &params);
+
+                /// @private
+                static cv::Ptr<cv::dnn::Layer> create(cv::dnn::LayerParams& params);
+
+                /// @private
+                virtual bool getMemoryShapes(const std::vector<std::vector<int> > &inputs,
+                                             const int,
+                                             std::vector<std::vector<int> > &outputs,
+                                             std::vector<std::vector<int> > &) const CV_OVERRIDE;
+
+                /// @private
+                virtual void forward(cv::InputArrayOfArrays inputs_arr,
+                                     cv::OutputArrayOfArrays outputs_arr,
+                                     cv::OutputArrayOfArrays) CV_OVERRIDE;
+            };
+    };
+    //! @}
+}
+}
+#endif

modules/dnn_superres/include/opencv2/dnn_superres_quality.hpp

@@ -0,0 +1,116 @@
+// This file is part of OpenCV project.
+// It is subject to the license terms in the LICENSE file found in the top-level directory
+// of this distribution and at http://opencv.org/license.html.
+
+#ifndef OPENCV_CONTRIB_DNN_SUPERRES_QUALITY_HPP
+#define OPENCV_CONTRIB_DNN_SUPERRES_QUALITY_HPP
+
+#include <opencv2/highgui.hpp>
+#include <opencv2/imgproc.hpp>
+#include <cstdarg>
+#include <sstream>
+#include <ctime>
+
+#ifdef _WIN32
+#include <Windows.h>
+#endif
+
+#include <opencv2/dnn_superres.hpp>
+#include "opencv2/dnn.hpp"
+
+namespace cv
+{
+    namespace dnn_superres
+    {
+
+        //! @addtogroup dnn_superres
+        //! @{
+
+        /** @brief A class benchmarking the dnn based super resolution algorithms.
+         */
+        class CV_EXPORTS DnnSuperResQuality
+        {
+            private:
+
+                static int fontFace;
+
+                static double fontScale;
+
+                static cv::Scalar fontColor;
+
+            public:
+
+                /** @brief Sets the font face used for drawing measures on images drawn by showBenchmark().
+                @param fontface font face
+                */
+                static void setFontFace(int fontface);
+
+                /** @brief Sets the font scale used for drawing measures on images drawn by showBenchmark().
+                @param fontscale font scale
+                */
+                static void setFontScale(double fontscale);
+
+                /** @brief Sets the font color used for drawing measures on images drawn by showBenchmark().
+                @param fontcolor font color
+                */
+                static void setFontColor(cv::Scalar fontcolor);
+
+                /** @brief Returns the PSNR of two given image.
+                @param img Upscaled image
+                @param orig Original image
+                @return PSNR value.
+                */
+                static double psnr(Mat img, Mat orig);
+
+                /** @brief Returns the SSIM of two given image.
+                @param img Upscaled image
+                @param orig Original image
+                @return SSIM value.
+                */
+                static double ssim(Mat img, Mat orig);
+
+                /** @brief Gives a benchmark for the given super resolution algorithm. It compares it to
+                * bicubic, nearest neighbor, and lanczos interpolation methods.
+                @param sr DnnSuperRes object
+                @param img Image to upscale
+                @param showImg Displays the images if set to true
+                */
+                static void benchmark(DnnSuperResImpl sr, Mat img, bool showImg = false);
+
+                /** @brief Gives a benchmark for the given super resolution algorithm. It compares it to
+                 * bicubic, nearest neighbor, and lanczos interpolation methods.
+                @param sr DnnSuperRes object
+                @param img Image to upscale
+                @param psnrValues Output container of the PSNR values
+                @param ssimValues Output container of the SSIM values
+                @param perfValues Output container of the Performance values
+                @param showImg Displays the images if set to true
+                @param showOutput Writes benchmarking to standard output if set to true
+                */
+                static void benchmark(DnnSuperResImpl sr, Mat img,
+                                        std::vector<double>& psnrValues,
+                                        std::vector<double>& ssimValues,
+                                        std::vector<double>& perfValues,
+                                        bool showImg = false,
+                                        bool showOutput = false);
+
+                /** @brief Displays benchmarking for given images.
+                @param orig Original image
+                @param images Upscaled images
+                @param title Title to window
+                @param imageSize Display size of images
+                @param imageTitles Titles of images values to display (optional)
+                @param psnrValues PSNR values to display (optional)
+                @param ssimValues SSIM values to display (optional)
+                @param perfValues Speed values to display (optional)
+                */
+                static void showBenchmark(Mat orig, std::vector<Mat> images, std::string title, Size imageSize,
+                                        const std::vector<String> imageTitles = std::vector<String>(),
+                                        const std::vector<double> psnrValues = std::vector<double>(),
+                                        const std::vector<double> ssimValues = std::vector<double>(),
+                                        const std::vector<double> perfValues = std::vector<double>());
+        };
+        //! @}
+    }
+}
+#endif //OPENCV_CONTRIB_DNN_SUPERRES_QUALITY_HPP