MTLTexture getBytes returning blank image

Question

Inside of my application I am trying to convert a MTLTexture into a CGImage or CIimage for use in other parts of the app. I am having a difficult time doing so because none of the example code I find seems to work properly.

For some reason something breaks on each and every one of them. Here are my attempts and why they dont work. If you could give me ideas on why that would be appreciated.

All of my textures are in brga8Unorm format.

Method 1 is to copy the pixels into an unsafe mutable pointer and use that to directly create a CGTexture. It creates a textures of all 0s

func toImage() -> NSImage? {
    let texture = self
    let width = texture.width
    let height = texture.height
    let bytesPerRow = width * 4
    let data = UnsafeMutableRawPointer.allocate(bytes: bytesPerRow * height, alignedTo: 4)
    defer {
        data.deallocate(bytes: bytesPerRow * height, alignedTo: 4)
    }

    let region = MTLRegionMake2D(0, 0, width, height)
    texture.getBytes(data, bytesPerRow: bytesPerRow, from: region, mipmapLevel: 0)
    var buffer = vImage_Buffer(data: data, height: UInt(height), width: UInt(width), rowBytes: bytesPerRow)

    let map: [UInt8] = [2, 1, 0, 3]
    vImagePermuteChannels_ARGB8888(&buffer, &buffer, map, 0)

    guard let colorSpace = CGColorSpace(name: CGColorSpace.genericRGBLinear) else { return nil }
    guard let context = CGContext(data: data, width: width, height: height, bitsPerComponent: 8, bytesPerRow: bytesPerRow,
            space: colorSpace, bitmapInfo: CGImageAlphaInfo.noneSkipLast.rawValue) else { return nil }
    guard let cgImage = context.makeImage() else { return nil }

    return NSImage(cgImage: cgImage, size: NSSize(width: width, height: height))
}

Method 2 tries to use CImage to do the job however CImage reports Cannot render image (with an input Metal texture) using a metal context. and returns nil

extension MTLTexture {
    func toImage() -> CGImage? {
        let context = CIContext()
        let texture = self
        let cImg = CIImage(mtlTexture: texture, options: nil)!
        let cgImg = context.createCGImage(cImg, from: cImg.extent)!
        return cgImg
    }
}

This last method doesnt even try to convert the image it is just trying to determine if the image is blank (all 0s) or not however even on fully green or even textured images it says they are blank.

func anythingHere(_ texture: MTLTexture) -> Bool {
    let width = texture.width
    let height = texture.height
    let bytesPerRow = width * 4

    let data = UnsafeMutableRawPointer.allocate(bytes: bytesPerRow * height, alignedTo: 4)
    defer {
        data.deallocate(bytes: bytesPerRow * height, alignedTo: 4)
    }

    let region = MTLRegionMake2D(0, 0, width, height)
    texture.getBytes(data, bytesPerRow: bytesPerRow, from: region, mipmapLevel: 0)
    var bind = data.assumingMemoryBound(to: UInt8.self)

    var sum:UInt8 = 0;
    for i in 0..<width*height {
        sum += bind.pointee
        bind = bind.advanced(by: 1)
    }
    return sum != 0
}

Clearly I am doing something fundamentally wrong as none of these methods are actually producing results. What am I doing wrong? What is the right way to do this?

Answer 1

maybe you forget to call MTLBlitCommandEncoder.synchronizeTexture to sync VRAM to system memory.

please check: https://forums.developer.apple.com/thread/30488

and document: https://developer.apple.com/documentation/metal/mtlblitcommandencoder/1400757-synchronize

Answer 2

This is more a comment than an answer, but since I'm new here with no reputation, I can't comment.

I was trying to use your method 2 to read a metal texture (having hit problems with your other methods, too, as well as half a dozen other methods from other threads). I used a .waitUntilCompleted method, so figured my problem didn't have to deal with flushing. My shader is a toy example:

kernel void compute_main(
                     texture2d<float, access::read> inTexture [[texture(0)]],
                     texture2d<float, access::write> outTexture [[texture(1)]],  // this doesn't work
                     //texture2d<float, access::read_write> outTexture [[texture(1)]],  // this does
                     uint2 gid [[thread_position_in_grid]])
{
    float4 inColor = inTexture.read(gid);
    float4 outColor (inColor.g, inColor.r, inColor.b, inColor.a);
    outTexture.write(outColor, gid);
}

I initially was getting this error:

validateComputeFunctionArguments:857: failed assertion `Compute Function(ciKernelMain): reads texture ([1]) whose usage (0x02) doesn't specify MTLTextureUsageShaderRead (0x01)'

I realize it isn't the same error you were getting, but maybe something has changed in metal/swift/xcode. I'm running Xcode 9.4.1

After changing the output texture to read_write (the commented line in the code above), and setting up the texture descriptor appropriately,

//textureDescriptor.usage = MTLTextureUsage.shaderWrite // not this
textureDescriptor.usage = [.shaderWrite, .shaderRead] // this works

it worked. Seems the CI library for this uses metal (I'm guessing from the error) and it reads the texture I thought I only needed to write to. Not obvious until you run it and get the error. I'm not sure if our situations were different or what, but in mine, at least I got an error message that was useful.