Running ARCore in landscape

[Bersaelor]

I was wondering whether someone already figured out how to run the ARCore sample in Landscape.

I was roughly trying:

        let orientation = UIDeviceOrientation(statusBarOrientation: UIApplication.shared.statusBarOrientation)

        // Set the scene camera's transform to the projection matrix for this frame.
        sceneCamera.projectionTransform = SCNMatrix4.init(
            frame.projectionMatrix(
                forViewportSize: previewLayer?.bounds.size ?? .zero,
                presentationOrientation: orientation,
                mirrored: false,
                zNear: 0.05,
                zFar: 100)
        )

but it's not really working for landscape.

Best I got was the right orientation of scene and video-image, with camera-image and 3D content mis-scaled (in landscape). In portrait the same code is fine.

[Bersaelor]

My changes so far boil down to:

    cameraImageLayer.contentsGravity = .resizeAspectFill

and resizing it as part of viewWillTransition:

    public override func viewWillTransition(to size: CGSize, with coordinator: UIViewControllerTransitionCoordinator) {
        super.viewWillTransition(to: size, with: coordinator)
        
        coordinator.animate(alongsideTransition: { (_) in
            
        }) { (_) in
            self.cameraImageLayer.frame = CGRect(origin: .zero, size: size)
        }
    }

and then with a helper method:

extension UIDeviceOrientation {
    init(statusBarOrientation: UIInterfaceOrientation, mirrored: Bool) {
        switch statusBarOrientation {
        case .portrait:
            self = .portrait
        case .portraitUpsideDown:
            self = .portraitUpsideDown
        case .landscapeLeft:
            self = mirrored ? .landscapeRight : .landscapeLeft
        case .landscapeRight:
            self = mirrored ? .landscapeLeft : .landscapeRight
        default:
            self = .unknown
        }
    }
}

I can do

        // Set the scene camera's transform to the projection matrix for this frame.
        sceneCamera.projectionTransform = SCNMatrix4.init(
            frame.projectionMatrix(
                forViewportSize: sceneView.bounds.size,
                presentationOrientation: UIDeviceOrientation(statusBarOrientation: UIApplication.shared.statusBarOrientation, mirrored: true),
                mirrored: false,
                zNear: 0.05,
                zFar: 100)
        )
        
        // Update the camera image layer's transform to the display transform for this frame.
        cameraImageLayer.contents = frame.capturedImage as CVPixelBuffer
        cameraImageLayer.setAffineTransform(
            frame.displayTransform(
                forViewportSize: sceneView.bounds.size,
                presentationOrientation: UIDeviceOrientation(statusBarOrientation: UIApplication.shared.statusBarOrientation, mirrored: false),
                mirrored: true
            )
        )

This works in portrait, and in LandscapeLeft/Right the 3D mask and the video match the right directions, just not the scale.
Did anyone test the frame.displayTransform method in Landscape?

(Here's a link to the full modified example)

Example video:

[Bersaelor]

Update: I was able to hotfix the ARCore frame.projectionMatrix() method by modifying the resulting Matrix when in landscape:

        var pM = SCNMatrix4.init(
            frame.projectionMatrix(
                forViewportSize: viewPortSize,
                presentationOrientation: orientation,
                mirrored: false,
                zNear: 0.05,
                zFar: 100)
        )
        
        if orientation != .portrait && orientation != .portraitUpsideDown {
            let factor: Float = 2.101334 / 1.1820006
            pM.m11 *= factor
            pM.m22 *= factor
        }

        sceneCamera.projectionTransform = pM

I had played around with the projection matrix, and noticed that when in landscape the factors that x&y are multiplied by were off by about 1.8. When I looked at the matrix in portrait, the value m12 was 2.101334 and in landscape m11 was 1.1820006. Since portrait was working and landscape didn't, I just scaled x and y so that it's the same scale as in portrait.

This works on iPhone X (375x812), iPad 11" (834 x 1194) and iPhone 6S (375 x 667).

[deshan]

Update: I was able to hotfix the ARCore frame.projectionMatrix() method by modifying the resulting Matrix when in landscape:

        var pM = SCNMatrix4.init(
            frame.projectionMatrix(
                forViewportSize: viewPortSize,
                presentationOrientation: orientation,
                mirrored: false,
                zNear: 0.05,
                zFar: 100)
        )
        
        if orientation != .portrait && orientation != .portraitUpsideDown {
            let factor: Float = 2.101334 / 1.1820006
            pM.m11 *= factor
            pM.m22 *= factor
        }

        sceneCamera.projectionTransform = pM

I had played around with the projection matrix, and noticed that when in landscape the factors that x&y are multiplied by were off by about 1.8. When I looked at the matrix in portrait, the value m12 was 2.101334 and in landscape m11 was 1.1820006. Since portrait was working and landscape didn't, I just scaled x and y so that it's the same scale as in portrait.

This works on iPhone X (375x812), iPad 11" (834 x 1194) and iPhone 6S (375 x 667).

This works for me too. Thanks for sharing your code.

But why this let factor: Float = 2.101334 / 1.1820006 ? Shouldn't this calculate internally?

Tested on ipad pro 10.5.

[Bersaelor]

But why this let factor: Float = 2.101334 / 1.1820006 ? Shouldn't this calculate internally?

@deshan That is a question for the ARCore developers, I have no why this factor is necessary.
If you don't scale the x-y components though, the resulting 3D objects will not match the camera in landscape (and be too small).

The fact that I needed this factor to make the math come out seems to show that there is a bug in the ARCore code when in landscape.

[Cloov]

@Bersaelor is it possible that this factor value (2.101334 / 1.1820006) is only suitable for a particular aspect ratio?

I ask because I'm rendering this pipeline to video, and in cases where the video preset is not 9:16 aspect ratio, the projection is scaled wrongly. I could probably just add more magic numbers/calculations and scale again from that ratio to the ratio I'm using, but that's yet another assumption in place - the assumption that 9:16 is what this factor value is based upon.

:)

Update: I just reversed the two numbers used, and 1.1820006:2.101334 is essentially 9:16!

[Bersaelor]

Yup, I am quite certain that is the case, I mean if orientation != .portrait && orientation != .portraitUpsideDown means any kind of landcape orientation.
The issue is that given the input

        var pM = SCNMatrix4.init(
            frame.projectionMatrix(
                forViewportSize: viewPortSize,
                presentationOrientation: orientation,

is already accepting the screen size and orientation, one shouldn't have to manually mess with the results anymore as it should be working correctly for each orientation.