feat: SG-44605: Optimize the loading and scrubbing of S3/Streaming media in RV

[deltag0]

feat: Optimize the loading and scrubbing of S3/Streaming media in RV

Summarize your change.

• Updated FFMPEG 8 to 9
• Added local caching for streamed media using the FFMPEG shared: protocol
• Added background threads to open FFMPEG contexts for MovieReader clones since FFMPEG even if all the media is cached with FFFMPEG, it will try to establish a connection, which adds latency (even if everything was cached). If we do this process at the launch of RV, the latency is removed
• Added background threads to start caching media over the internet in the background to optimize scrubbing further
• Added a lookahead thread that reads and caches ahead of the playhead which, only 1 can be active, and if there is a lookahead thread in the worker queue, no new workers are started.
• Disabled slow random access for streamed media

Describe the reason for the change.

Previously, when media hosted on shotgrid was opened in OpenRV, to view a new file, there had to be a request made over the network, which was adding 1-2 seconds of latency every time a user jumped to a new file. This made scrubbing virtually impossible because of the delay. To improve this problem, the idea was to cache the media's raw bytes locally, so that RV can decode the bytes locally instead of having to establish a new HTTP connection for a new file. This would work, but would require us to manually jump to the new file for the caching to start. So, background threads to download every file were spawned.
After testing, on my machine, I found that 4 threads for background fetching provided the best performance.
Soon after, it was found that even if all media was cached before a user streams it on RV, scrubbing would still be painful because there were FFMPEG reader clones which had to establish their own connection, so they still added latency. To fix this, the connections were established at load time, with background threads, when the media is opened (after pre-loading).

The lookahead thread helps with scrubbing when not everything is cached, however it currently still reads ahead if all the media is cached. When testing without it, scrubbing was still laggy for the first half minute until the background threads finished caching everything, but with it enabled, scrubbing worked almost immediately.

Describe what you have tested and on which operating system.

• Tested on a Mac
• Viewed improvements that if files are cached locally, scrubbing is instant

Add a list of changes, and note any that might need special attention during the review.

• Verify update from FFMPEG 8 to 9

Notes:

• Currently no limit to the size of the raw media cache

• Background threads will still run even if all media is cached (although they will just read from disk)

• Verified that warming reader clones was still necessary after background threads and lookahead thread, the performance degrades without it

• manual latency test:

time ffmpeg -loglevel error \
  -headers $'Cookie: _session_id=fa0d43df120581721917cd2d820484e8\r\n' \
  -i 'https://sg-create-prod.shotgunstudio.com/file_serve/version/14232/mp4?20097' \
  -frames:v 1 \
  -f null -
ffmpeg -loglevel error -headers  -i  -frames:v 1 -f null -  0.19s user 0.06s system 21% cpu 1.159 total
rel (.venv) termev@ADSKJ7HQLQCG6D rv %
Goiing through S3 directly only saves about 0.1ms

• Tested blocking new workers while the lookahead thread was active, but observed no noticeable performance improvement

If possible, provide screenshots.

[eloisebrosseau]

Why is job passed as a copy here? I believe it should be a const reference instead like you did for download and downloadWindow

[deltag0]

We pass job from the schedulerLoop which is local to the function

[deltag0]

The object in the queue is destroyed when we pop it

[eloisebrosseau]

I appreciate that you are trying to clean up some of the dead code in the IPCore directory, but I feel like it is unrelated to the purpose of this PR, since the removals are made on files that you are not touching otherwise. You could create a second PR for that cleanup if you want, and remove it from this one, to help us review only what is actually important for the feature to work. Same thing for MovieFFMpegReader::scan(). Have you checked why it was kept with just a comment instead of being removed? If it's not related to your changes, I would leave it alone unless you are 100% sure this can be safely removed

[eloisebrosseau]

Is this still being used in this file? If not, you should remove it. Also, you should put back <boost/thread/mutex.hpp> with the other boost header files where it was before

[deltag0]

fixed