01 [Industry] OpenAI’s Safety Stack for Sora 2 Reveals How Hard Real-Time Video Moderation Actually Is
Content safety for video generation was once assumed to be a solved problem—simply applying image classifiers frame-by-frame. However, Sora 2 and its social creation platform demand a more robust architectural solution. Video generated at scale, by real users, in real time, breaks every assumption that image-era safety tooling was built on.
The safety stack Sora ships with operates at multiple layers simultaneously. Prompt classifiers intercept harmful requests before generation begins. Watermarking using C2PA (Coalition for Content Provenance and Authenticity) metadata embeds provenance at the file level, so Sora-generated video carries a verifiable origin signal regardless of where it travels. A separate video-level classifier runs on outputs post-generation, catching what prompt filtering misses, such as stylistic jailbreaks or indirect requests that produce harmful content through plausible deniability. The social platform layer adds another dimension: user-facing reporting, human review queues, and account-level enforcement that treats the creation surface as distinct from the model itself.
A significant limitation is the system’s coverage under adversarial pressure. Developers evaluate every deployed safety system against the distribution of attempts it was designed to catch. The Sora platform is a new surface, with a new user population and incentive structures that haven’t been stress-tested at scale. Watermarking survives casual sharing but degrades under video re-compression, format conversion, and screen recording, which are the exact workflows bad actors use. C2PA metadata is also opt-in to verify, meaning downstream platforms must actively check for it.
Key takeaways:
- Multi-layer defense (prompt filter → generation → output classifier → platform enforcement) reflects the structural reality that no single intervention point catches everything in a video generation pipeline.
- Deploying a social creation platform forces safety infrastructure to operate at social-network scale and speed, a qualitatively different threat model from the API-call scale of a research preview.
- Teams building on top of video generation APIs should not treat upstream safety layers as sufficient; output classifiers and provenance checks at the application layer remain the practitioner’s responsibility.
Source: Creating with Sora Safely