# How to capture frames in a MacOS MTKView for recording to a video

I have created a 3D model of a local outdoor performance space and I have an app that uses Metal on MacOS 10.14.2 to display the model. I want to create an animation by flying the camera around the scene, while recording each frame (actually every other frame for 30 fps). I know how to do the animated fly-around and I know how to create a video frame-by-frame with AVFoundation. The step for which I can find no information is how I can capture each frame.

I have a completion handler so I know when the gpu has finished each command buffer.

But what is the best way to get the image in CPU space?

I thought perhaps I could do this by attaching a second texture to colorAttachments[1] but this has resulted in some odd behavior where the original scene that used to fill my MTKView window now occupies just the upper left quadrant of the window.

What I was trying to do is write the same color to both colorAttachments[0] (my screen) and to colorAttachments[1]. In my shader I defined:

struct FragmentOut {
float4 color0 [[ color(0) ]];
float4 color1 [[ color(1) ]];
};


fragment FragmentOut grove_fragment_function(
VertexOut vIn [[ stage_in ]],
constant Uniforms &uniforms [[buffer(1)]]) {
....
float4 color = diffuseColor + ambientColor + specularColor;
out.color0 = color;
out.color1 = color;
return out;
}


My hope was that I could then use something like:

offScreenTextureBuffer?.getBytes(buffer, bytesPerRow: 4 * w, from: MTLRegionMake2D(0, 0, w, h), mipmapLevel: 0)


to transfer the image data to a local buffer. This doesn't seem to work, plus I have the unexpected display behavior noted above.

I am configuring the offScreenTextureBuffer thus:

let pixelFormat = MTLPixelFormat.bgra8Unorm_srgb
var offScreenBufferDescriptor =  MTLTextureDescriptor.texture2DDescriptor(pixelFormat: pixelFormat,                                                                             width: 1000,                                                                             height: 600,                                                                            mipmapped: false)
offScreenTextureBuffer = device.makeTexture(descriptor: offScreenBufferDescriptor)


I also tried to use the presentation layer from my MTKView. I can get the texture from the presentation layer but when I try to use the getBytes() func on the returned texture I get an error because the buffer is configured as "framebufferonly". A presentation layer that can't be read doesn't seem particularly useful.

Any suggestions on how to capture each frame would be appreciated.

• Do you need to render to screen and generate the movie at the same time? If you can do a separate pass for the movie, you could draw to an offscreen and call getBytes() on the offscreen once it's finished before moving on to the next frame, maybe? – user1118321 Feb 4 '19 at 4:48
• @user1118321 I have been trying to do that (render just to an offscreen buffer) but have been going around in circles with how to set the Descriptor. Apparently I don't fully understand how to use the colorAttachments. I get an error if I try to attach a buffer without the usage set to include .rendertarget. But if I do that then I get an error when trying to use getBytes. Is there any sample code on the web I could study? – Richard Stover Feb 5 '19 at 4:19
• Definitely would like to see a code sample – ClearRange Mar 9 '19 at 0:52

I've seen no answer to my question. But after lots of experimentation and further study of Apple documentation for clues I have figured out how to capture GPU generated frames.

First, I gave up on the attempt to have the shader write to the screen and an offscreen buffer in the same render pass. The odd (and undocumented as far as I can tell) behavior of MTKView in which the onscreen view occupies only a portion of the original area of the screen. I think this a bug in MTKView. It seems to act as if each attachment should appear somewhere on the screen. MTKView is not checking to see if the attachments are all on the screen.

Since MTKView calls the draw() function 60 times per second I decided to alternate. One draw() goes to the screen and the following draw() goes to my off screen buffer. This gives me a 30 Hz frame rate for the captured video frames, which is what I wanted anyway. I can watch the screen and see what will appear in the video.

For the video capture draw() calls I now define the MTLTextureDescriptor in this way:

let offScreenPixelFormat = MTLPixelFormat.rgba32Float   // 32Float is convenient for AVFoundation video recording
let offScreenBufferDescriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: offScreenPixelFormat,
width: offW, height: offH, mipmapped: false)
offScreenBufferDescriptor.storageMode = .managed
offScreenBufferDescriptor.resourceOptions = [.storageModeManaged]
offScreenBufferDescriptor.textureType = MTLTextureType.type2D
offScreenTextureBuffer = device.makeTexture(descriptor:    offScreenBufferDescriptor)


It turns out this still did not work. I got nothing back from the GPU. That is, I got nothing back until I added this code to the end of the commandBuffer:

  let blitCommandEncoder = commandBuffer.makeBlitCommandEncoder()
blitCommandEncoder?.synchronize(resource: offScreenTextureBuffer!)
blitCommandEncoder?.endEncoding()


This causes the GPU memory version of the offScreenTextureBuffer to be copied back to the CPU memory version. I figured this out by looking at the Apple documentation. Like all Apple documentation, it doesn't tell you why you might want to synchronize. But you definitely do need to do this. I have seen several online code examples where this is NOT done and they claim to get data back from the GPU. I tried one example code without the synchronize command and nothing was returned. So I assume the requirement to synchronize is a new feature and the online code is just out of date. It would be nice if the Apple documentation made it clear this is now a requirement.

With these changes I can retrieve GPU frames. I have created a short video using GPU-generated frames.

If anyone is interested in seeing a complete code sample let me know.

• Going through metal and trying to understand this bit. I am interested in seeing complete sample code – Rohan Sanap Dec 31 '19 at 4:03
• It has been a few months since I looked at this code. I'll review it and get back to you in a few days. – Richard Stover Jan 1 at 4:02