cv2.cudacodec.createVideoReader error when input is a video stream

[wqh17101]

System information (version)

• OpenCV => 4.5.2
• Operating System / Platform => linux
• Compiler => gcc

Detailed description

I built opencv_contrib_python by myself.
Here is the build information

Extra dependencies:          m pthread cudart_static dl rt nppc nppial nppicc nppidei nppif nppig nppim nppist nppisu nppitc npps cublas cudnn cufft -L/usr/local/cuda/lib64 -L/lib64
    3rdparty dependencies:

  OpenCV modules:
    To be built:                 calib3d core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudastereo cudawarping cudev dnn features2d flann highgui imgcodecs imgproc ml objdetect photo python3 stitching video videoio ximgproc
    Disabled:                    aruco bgsegm bioinspired ccalib datasets dnn_objdetect dnn_superres dpm face fuzzy hfs img_hash intensity_transform line_descriptor mcc optflow phase_unwrapping plot quality rapid reg rgbd saliency shape stereo structured_light superres surface_matching text tracking videostab wechat_qrcode world xfeatures2d xobjdetect xphoto
    Disabled by dependency:      cudaoptflow
    Unavailable:                 alphamat cnn_3dobj cvv freetype gapi hdf java julia matlab ovis python2 sfm ts viz
    Applications:                -
    Documentation:               NO
    Non-free algorithms:         NO

  GUI: 
    GTK+:                        NO

  Media I/O: 
    ZLib:                        /usr/local/lib/libz.so (ver 1.2.12)
    JPEG:                        /usr/local/lib/libjpeg.so (ver 90)
    WEBP:                        build (ver encoder: 0x020f)
    PNG:                         /usr/local/lib/libpng.so (ver 1.6.37)
    TIFF:                        build (ver 42 - 4.2.0)
    JPEG 2000:                   build (ver 2.4.0)
    HDR:                         YES
    SUNRASTER:                   YES
    PXM:                         YES
    PFM:                         YES

  Video I/O:
    DC1394:                      NO
    FFMPEG:                      YES
      avcodec:                   YES (58.91.100)
      avformat:                  YES (58.45.100)
      avutil:                    YES (56.51.100)
      swscale:                   YES (5.7.100)
      avresample:                NO
    GStreamer:                   NO
    v4l/v4l2:                    YES (linux/videodev2.h)

  Parallel framework:            pthreads

  Trace:                         YES (with Intel ITT)

  Other third-party libraries:
    VA:                          NO
    Lapack:                      NO
    Eigen:                       NO
    Custom HAL:                  NO
    Protobuf:                    build (3.5.1)

  NVIDIA CUDA:                   YES (ver 11.0, CUFFT CUBLAS NVCUVID)
    NVIDIA GPU arch:             35 37 50 52 60 61 70 75 80
    NVIDIA PTX archs:

  cuDNN:                         YES (ver 8.1.1)

I want to use cuda to accelerate video decoding, but only found it worked on files.
I want to use it on video stream.

Steps to reproduce

import cv2
url="rtsp://admin:admin@192.168.2.46/media/video1"

cpu:

cv2.VideoCapture(url).read() # worked

gpu:

cv2.cudacodec.createVideoReader(url).next_frame()

get

opencv_contrib/modules/cudacodec/src/video_reader.cpp:128: error: (-210:Unsupported format or combination of formats) Unsupported video source in function 'nextFrame'

but if the url is a video file
both cpu and gpu worked.

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues,
  forum.opencv.org, Stack Overflow, etc and have not found any solution
• I updated to the latest OpenCV version and the issue is still there
• There is reproducer code and related data files: videos, images, onnx, etc

[wqh17101]

Also i try 4.5.5 ,failed again @cudawarped
here is the debug msg

[OPENCV:FFMPEG:40] Starting connection attempt to xxx port 8554
[OPENCV:FFMPEG:40] Successfully connected to xxx port 8554
[OPENCV:FFMPEG:40] SDP:
v=0
o=- 0 0 IN IP4 127.0.0.1
s=Stream
c=IN IP4 0.0.0.0
t=0 0
m=video 0 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1; sprop-parameter-sets=Z2QAH6zZQFAFuwEQAAADABAAAAMDIPGDGWA=,aOvjyyLA; profile-level-id=64001F
a=control:trackID=0

[OPENCV:FFMPEG:40] setting jitter buffer size to 0
[OPENCV:FFMPEG:40] Reinit context to 1280x720, pix_fmt: yuv420p

Also get

cv2.error: OpenCV(4.5.5) modules/cudacodec/src/video_reader.cpp:140: error: (-210:Unsupported format or combination of formats) Unsupported video source in function 'internalGrab'

[cudawarped]

@wqh17101

I don't have any issues streaming from h264[5] RTSP sources. My guess would be the codec isn't supported or it has something to do with the way the parameter set is included.

Is it possible to give me a sample of the video using

ffmpeg -i "rtsp://admin:admin@192.168.2.46/media/video1" -c copy sample.264

[wqh17101]

i push the stream by

 ffmpeg -re -stream_loop -1 -i local_test.mp4 -vcodec libx264 -f flv -f rtsp rtsp://admin:admin@192.168.2.46/stream/

and pull by

ffmpeg -hwaccel cuda -c:v h264_cuvid -hwaccel_output_format cuda -i rtsp://admin:admin@192.168.2.46/stream/ -c:v h264_nvenc -b:v 5M output.mp4 -y

@cudawarped

[cudawarped]

Which verson of ffmpeg are you using, I cannot send and recieve a stream using those exact commands?

Have you tried streaming from other rtsp sources, live555, vlc etc.?

[wqh17101]

ffmpeg 4.3.2
live555, vlc i do not have these source.

[cudawarped]

I am not sure why I cannot push an rtsp stream using your command

ffmpeg -re -stream_loop -1 -i local_test.mp4 -vcodec libx264 -f flv -f rtsp rtsp://admin:admin@192.168.2.46/stream/

so lets try this another way. Can you use VideoCapture to get the SPS PPS info and the header of the first raw frame so I can see what is being passed to the decoder.

import cv2 as cv
import numpy as np
np.set_printoptions(formatter={'int':hex})
url="rtsp://admin:admin@192.168.2.46/media/video1"
cap = cv.VideoCapture(url)
cap.set(cv.CAP_PROP_FORMAT,-1)
if(cap.get(cv.CAP_PROP_FORMAT) !=-1): print('Unable to activate raw bitstream reads')
iSpsPps = int(cap.get(cv.CAP_PROP_CODEC_EXTRADATA_INDEX))
ret, spsPps = cap.retrieve(flag=iSpsPps)
if(not ret): print('Unable to retrieve parapeter sets')
ret, encodedFrame = cap.read()
if(not ret): print('Unable to retrieve encoded frame')
print(f'SPS and PPS\n {spsPps[0]}')
print(f'Start of encoded frame\n {encodedFrame[0,:100]}')

The above should output something similar to

SPS and PPS
[0x0 0x0 0x0 0x1 0x67 0x64 0x0 0x32 0xac 0xd2 0x0 0xa2 0x3 0xd7 0xe5 0x84
0x0 0x0 0xf 0xa4 0x0 0x3 0xe 0x8 0x10 0x0 0x0 0x0 0x1 0x68 0xea 0x8f 0x2c]
Start of encoded frame
[0x0 0x0 0x0 0x1 0x67 0x64 0x0 0x32 0xac 0xd2 0x0 0xa2 0x3 0xd7 0xe5 0x84
0x0 0x0 0xf 0xa4 0x0 0x3 0xe 0x8 0x10 0x0 0x0 0x0 0x1 0x68 0xea 0x8f 0x2c
0x0 0x0 0x0 0x1 0x65 0xb8 0x40 0x2f 0x6a 0xc6 0xa6 0x3 0xff 0x84 0x9a
0x51 0x6d 0x55 0x55 0x67 0xe9 0xd 0x52 0x14 0xed 0x33 0x3c 0xa7 0x88 0x70
0x2c 0xd0 0x6 0x29 0x8e 0xe0 0x0 0xf 0xec 0xe4 0x21 0x5e 0x1e 0xc6 0xd5
0x63 0x76 0x48 0xb7 0xb9 0x7e 0x0 0x18 0x51 0xe7 0x2 0x3f 0x1b 0x7c 0x8
0x21 0xa8 0x50 0xa0 0xe9 0x61 0x1b]

[wqh17101]

Here

[h264 @ 0x55c913ea2440] Missing reference picture, default is 0
[h264 @ 0x55c913ea2440] co located POCs unavailable
SPS and PPS
 [0x0 0x0 0x0 0x1 0x67 0x64 0x0 0x1f 0xac 0xd9 0x40 0x50 0x5 0xbb 0x1 0x10
 0x0 0x0 0x3 0x0 0x10 0x0 0x0 0x3 0x3 0x20 0xf1 0x83 0x19 0x60 0x0 0x0 0x0
 0x1 0x68 0xeb 0xe3 0xcb 0x22 0xc0]
Start of encoded frame
 [0x0 0x0 0x0 0x1 0x1 0x9e 0x3a 0x6a 0x46 0x7f 0x3b 0xaa 0x41 0xb4 0x3c
 0x36 0xc1 0xf3 0xf2 0xf2 0x51 0x81 0x7 0x74 0xa0 0x5a 0xfe 0x80 0x31 0xa
 0x41 0xc 0x8 0x55 0x61 0xac 0xc 0xf0 0x7a 0xe9 0x84 0xca 0x8c 0x73 0xc2
 0xa9 0xb3 0xc1 0x1 0x29 0xca 0x4 0x21 0xec 0xa5 0x2d 0xc 0x90 0xc7 0xfc
 0x6a 0xa1 0x5f 0x4b 0xdd 0x62 0x39 0x3e 0x55 0x60 0x7f 0xe4 0xf1 0x87
 0x26 0xc 0x1f 0xb8 0x7f 0x73 0xa4 0x47 0xdd 0x3a 0x85 0x26 0x4d 0x94 0x7
 0x8b 0xd0 0x4e 0xbf 0x3d 0xa0 0xe2 0x8f 0x24 0x66 0x2d]

Another problem caused core dump when i called createVideoWriter while createVideoReader will not core dump:

>>> cv2.cudacodec.createVideoWriter()
terminate called after throwing an instance of 'cv::Exception'
  what():  OpenCV(4.5.5)opencv-python/opencv/modules/core/include/opencv2/core/private.cuda.hpp:112: error: (-213:The function/feature is not implemented) The called functionality is disabled for current build or platform in function 'throw_no_cuda'

[cudawarped]

Thank you. Since your first NAL unit is non IDR I am wondering if the maximum number of unparsed packets is being exceeded. If this was the case I would also expect to see a message similar to the below

[h264 @ 0000027fcbea33c0] co located POCs unavailable

That said if its easy try setting maxUnparsedPackets = 100; and re-building

opencv_contrib/modules/cudacodec/src/video_parser.cpp

Line 86 in bbe0448

constexpr int maxUnparsedPackets = 20;

Could you also upload and send me the link to local_test.mp4 file and retrieve and write the hex of all frames from ret, encodedFrame = cap.read() to a file and send it to me?

Another problem caused core dump when i called createVideoWriter while createVideoReader will not core dump:

cudacodec::VideoWriter() has not worked for a few years now. If you need to write the output this can be done without encoding it once the python bindings have been fixed.

[wqh17101]

Could you also upload and send me the link to local_test.mp4 file and retrieve and write the hex of all frames from ret, encodedFrame = cap.read() to a file and send it to me?

I do not know what you need, maybe give me a script.

If its easy try setting maxUnparsedPackets = 100; and re-building

I will try.

[wqh17101]

Also my rtsp server is following here . https://github.com/aler9/rtsp-simple-server. Maybe you can setup and test.

[cudawarped]

I do not know what you need, maybe give me a script.

Try the following then upload out.264 somewhere and send me the link. Then I can run the same bitstream through the decoder at my end and find out what the issue is.

import cv2 as cv
import numpy as np
import sys
np.set_printoptions(formatter={'int':hex})
cap = cv.VideoCapture(url)
cap.set(cv.CAP_PROP_FORMAT,-1)
if(cap.get(cv.CAP_PROP_FORMAT) !=-1): print('Unable to activate raw bitstream reads')
iSpsPps = int(cap.get(cv.CAP_PROP_CODEC_EXTRADATA_INDEX))
ret, spsPps = cap.retrieve(flag=iSpsPps)
if(not ret): print('Unable to retrieve parapeter sets')
f = open("out.264", "wb")
spsPps.tofile(f)
for i in range(100):
    ret, encodedFrame = cap.read()
    if(not ret): 
        print('Unable to retrieve encoded frame')
        break
    encodedFrame.tofile(f)

[wqh17101]

out.zip
here

The video is opencv/doc/js_tutorials/js_assets/cup.mp4 which you can get easily.

[wqh17101]

@cudawarped

If its easy try setting maxUnparsedPackets = 100; and re-building

It works ! WOW Amazing.

>>> cap.nextFrame()
(True, <cuda_GpuMat 0x7f06c5a84190>)
>>> cap.nextFrame()
(True, <cuda_GpuMat 0x7f06c5a841b0>)
>>> cap.nextFrame()
(True, <cuda_GpuMat 0x7f06c5a841d0>)
>>> cap.nextFrame()
(True, <cuda_GpuMat 0x7f06c5a84170>)
>>> cap.nextFrame()
(True, <cuda_GpuMat 0x7f06c5a84190>)
>>> cap.nextFrame()
(True, <cuda_GpuMat 0x7f06c5a841b0>)
>>> cap.nextFrame()
(True, <cuda_GpuMat 0x7f06c5a841d0>)

[wqh17101]

So Does this mean that this var maxUnparsedPackets is not set normally?

And another question: If i have a ffmpeg built with cuda, is there anyway to use it such as using VideoCapture with some parameter like ffmepg cmd line.