Add Packet.from_tensor to build packets from raw data

[oxkitsune]

What

Packets could previously only be obtained from a Demuxer.

This adds Packet.from_bytes and Packet.from_tensor to construct a Packet directly from raw compressed data (e.g. from a network stream), with caller-provided pts, duration, and is_key_frame.

pts and duration are in the consuming decoder's stream time base. dts is set to pts, so this is only correct for streams without B-frame reordering.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/meta-pytorch/torchcodec/1585

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[meta-cla]

Hi @oxkitsune!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

[NicolasHug]

Hi @oxkitsune , thanks for the PR and for your interest in the these new APIs! They are still very WIP and subject to change (and not public!) so I'd love to learn more about your use for it?

---

On the PR itself, I'm open to it and to support that feature in general, but I won't merge it now because I plan to continue working (and heavily changing) these APIs in the near future. I also suspect we'll be supporting a file-like input in the Demuxer, which should address the network stream use-case.

[oxkitsune]

Hi @NicolasHug, these new APIs are awesome!
My usecase is to streaming in packets (typically one or two GOPs) in over the network and decoding them to tensors for training.

I also suspect we'll be supporting a file-like input in the Demuxer, which should address the network stream use-case.

This sounds great! Looking forward to it. If there's anything I can help out with, let me know!

[NicolasHug]

Thanks! Can you share more on why the VideoDecoder (which supports file-like inputs already) isn't addressing your use-case? Is it because your streams don't have a duration metadata, which the VideoDecoder expects (it's not truly a stream decoder by that definition)?

[meta-cla]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

[oxkitsune]

You're right that missing duration metadata is one of the blockers (exact mode also fails on the rewind seek after its scan). We have PTS/duration, so we can work around it by wrapping each packet window (keyframe -> target) in an in-memory MP4 and using the file-like input.

Can you share more on why the VideoDecoder (which supports file-like inputs already) isn't addressing your use-case?

Yea! Our use case is a bit different than the usual one I guess 😄

---

Some context on our setup: Rerun stores video as raw codec packets (Annex B / OBUs) in a database, one packet per row keyed by timestamp, with keyframe flags in a separate column. There is no file.
Our training dataloader samples random timestamps across many recordings, and for each sampled frame the server hands us exactly the packets from the preceding keyframe up to the target. So our ideal input would be a list of raw packets + the PTS/duration metadata which we can reconstruct.

With the current file-like input API we need to wrap the window of packets (keyframe to target frame) in an in-memory MP4 container in order to feed it to a VideoDecoder instance. This also means we cannot re-use a decoder across samples: consecutive targets often land in the same GOP, but since samples arrive one at a time we can't know the full window up front, each sample means muxing a fresh container, constructing a fresh VideoDecoder, and re-decoding from the keyframe.

---

So what I'd love from a packet-level API is the ability to feed raw packets (with keyframe/PTS info) into a decoder object, ask for decoded frames as tensors, and ideally be able to keep feeding packets to an existing instance.