#1 I would love to run it at full res, but the CPU is already at 92% just at the mid res of : 2304x1296, and I am not getting that 56.03 fps when MP4 / JPGE still is involved. I get the 1.5 jpeg stills per second, and the video is like 12.5 fps. Video isn't too bad because it's autofocus clear and it has a better that cd quality soundtrack ![:)]( "Smile")
a) I noticed the ArduCam PINSIGHT, it has onboard compression accelerator on a 12mp camera. Encoding: H.264, H.265, MJPEG – 4K/30FPS, 1080P/60FPS. I fell out of love with it because no low light visibility, I didn't like the 81 deg FOV,. Also as you mentioned no control over frames. I don't know if I want to spend $99 just to find out more things about it that would make it less than useful. During some of those demo videos, seemed the objects it was detecting needed to be at least 25% of the video frame. Not good. A lot of this stuff just ends up in my junk box, the "Island of the Misfit Toys".
https://www.arducam.com/arducam-pinsight/
b) I know that FFMPEG has CUDA support, but that means you would have to put on a big honking power grubbing heat inducing NVIDIA GPU. I was wondering if FFMPEG has any support for the TPU's/CoProcessors like the Google Coral TPU, or the Hailo 8 TPU. There's got to be some sort of bolt on video compression accelerator out there otherwise Raspberry PI severely missed the needs of industry. Yeah you can do all that AI jumping around, pose detection, detecting oranges on the countertop, but if you cant compress it to a 4-12k video stream @ 30 fps what good is having a 12k chip? It's a contemptible bottleneck, like having a V12 engine on a model T frame. Might be good for time-elapsed photography, but not for real world security camera usage where you want to be able to review footage for any second of the day.
c) I will look into what you speak of here. I don't get why you can just do:
stream = encodeMP4ToFile()
frame = stream.getCurrentAlreadyCompressedJPEGFrame()
I mean I got to believe there would be a faster way to get a current compressed frame from what is being written to the MP4 file than getting back a raw stream of image data, then calling the compressor on it, then saving it out to a file. Seems there would be a big boost if there were some enhancements to picamera2. Someone could figure it out.
In reality I would like:
encodeMP4ToFile(compressed_frame_callback,compressed_audio_callback)
So whenever a new compressed frame is made, your callback gets it! Same thing with audio, where it's feeding you chunk after chunk so that if you want to do audio processing in real time or play back live sound on a web page, you can do it !
d) I had first started out with that MSE detecting stuff on the lores frames. That would be good in a classroom, but not outside. To much going on with light and clouds, and wind. You really need something that shows you what it's detecting that moving so that you can either adjust the camera to point away from clouds in the sky, or setup exclusion polygons. So I went with this mechanism: https://hackmd.io/@lKuOpplzSUWLhLim2Z7ZJw/BkGj66aW3 It was fast enough on the high res frames and I don't have to scale the detection rectangles up to full size. Also the low res frames were some sort of composite image with different channels of color isolated in greyscale images. So it wasn't simply a shrunken down image frame. Then my detection rectangles be very hard to scale.
4) Didn't know about the YUV420 format, I might look into it to see what it helps/hinders.
5) Glad there are some other options that might give me some more speed for JPEG encoding. I will look into this. I have a webpage that will poll for the PI's current frame. I would like the camera to save the JPG as well as some server to get it so we know exactly what happened to the camera when it went down provided the image was sent, or enough of them....and the SSD card is still in tact when I get to examine the camera. As mentioned before, I'm trying to determine if there is motion on the camera by reading frames and getting detection rectangles. Throwing out stuff that's in bad areas or too small. If there is a detection, I want to send the 10 second MP4 file to a server as well as the current frame where motion was detected. This is for a backup purpose and for AI / Neural Net / YOLO3-8 object classification back on the server side.
Thanks for all the help!
a) I noticed the ArduCam PINSIGHT, it has onboard compression accelerator on a 12mp camera. Encoding: H.264, H.265, MJPEG – 4K/30FPS, 1080P/60FPS. I fell out of love with it because no low light visibility, I didn't like the 81 deg FOV,. Also as you mentioned no control over frames. I don't know if I want to spend $99 just to find out more things about it that would make it less than useful. During some of those demo videos, seemed the objects it was detecting needed to be at least 25% of the video frame. Not good. A lot of this stuff just ends up in my junk box, the "Island of the Misfit Toys".
https://www.arducam.com/arducam-pinsight/
b) I know that FFMPEG has CUDA support, but that means you would have to put on a big honking power grubbing heat inducing NVIDIA GPU. I was wondering if FFMPEG has any support for the TPU's/CoProcessors like the Google Coral TPU, or the Hailo 8 TPU. There's got to be some sort of bolt on video compression accelerator out there otherwise Raspberry PI severely missed the needs of industry. Yeah you can do all that AI jumping around, pose detection, detecting oranges on the countertop, but if you cant compress it to a 4-12k video stream @ 30 fps what good is having a 12k chip? It's a contemptible bottleneck, like having a V12 engine on a model T frame. Might be good for time-elapsed photography, but not for real world security camera usage where you want to be able to review footage for any second of the day.
c) I will look into what you speak of here. I don't get why you can just do:
stream = encodeMP4ToFile()
frame = stream.getCurrentAlreadyCompressedJPEGFrame()
I mean I got to believe there would be a faster way to get a current compressed frame from what is being written to the MP4 file than getting back a raw stream of image data, then calling the compressor on it, then saving it out to a file. Seems there would be a big boost if there were some enhancements to picamera2. Someone could figure it out.
In reality I would like:
encodeMP4ToFile(compressed_frame_callback,compressed_audio_callback)
So whenever a new compressed frame is made, your callback gets it! Same thing with audio, where it's feeding you chunk after chunk so that if you want to do audio processing in real time or play back live sound on a web page, you can do it !
d) I had first started out with that MSE detecting stuff on the lores frames. That would be good in a classroom, but not outside. To much going on with light and clouds, and wind. You really need something that shows you what it's detecting that moving so that you can either adjust the camera to point away from clouds in the sky, or setup exclusion polygons. So I went with this mechanism: https://hackmd.io/@lKuOpplzSUWLhLim2Z7ZJw/BkGj66aW3 It was fast enough on the high res frames and I don't have to scale the detection rectangles up to full size. Also the low res frames were some sort of composite image with different channels of color isolated in greyscale images. So it wasn't simply a shrunken down image frame. Then my detection rectangles be very hard to scale.
4) Didn't know about the YUV420 format, I might look into it to see what it helps/hinders.
5) Glad there are some other options that might give me some more speed for JPEG encoding. I will look into this. I have a webpage that will poll for the PI's current frame. I would like the camera to save the JPG as well as some server to get it so we know exactly what happened to the camera when it went down provided the image was sent, or enough of them....and the SSD card is still in tact when I get to examine the camera. As mentioned before, I'm trying to determine if there is motion on the camera by reading frames and getting detection rectangles. Throwing out stuff that's in bad areas or too small. If there is a detection, I want to send the 10 second MP4 file to a server as well as the current frame where motion was detected. This is for a backup purpose and for AI / Neural Net / YOLO3-8 object classification back on the server side.
Statistics: Posted by UltimateCodeWarrior — Tue Dec 17, 2024 7:34 pm