Sony has filed a patent describing an AI system that predicts the next frame of a game before the graphics hardware has finished rendering it. It is meant to reduce stuttering while avoiding the extra latency that some frame-generation methods cause.
The filing describes a system that uses the most recent completed frame, and optionally recent controller input, to generate a predicted frame. This frame is shown while the hardware renders the next one. Some frame-generation techniques need two completed frames to create an image between them, and waiting for the second frame adds latency. Sony’s system avoids this by predicting from a frame that is already available instead of interpolating between two.
This difference could help games deliver frames more consistently when rendering slows down. The filing includes no performance measurements, however, so the effects on latency and visual quality have not been shown.
The architecture pairs a jointly trained encoder and decoder. The encoder maps a rendered frame into a low-dimensional embedding, a compressed numerical representation of the image, and the decoder reconstructs a predicted frame from that embedding.
During gameplay, the system operates alongside the graphics pipeline. While the hardware renders the next frame, the AI processes the previous one and sends a prediction to the display buffer. The generated image can then appear before the next rendered frame is ready.
The model can also use motion vectors, which describe how parts of an image move, and recent controller input. These give it more information than the previous frame by itself. Motion vectors show the direction objects are heading, and button presses show the player’s latest actions.
Maybe you would like other interesting articles?