# NRSDK Coordinate Systems

This document describes the coordinate systems of the Nreal Glass used in the NRSDK for Unity. It also describes the corresponding interfaces for getting extrinsics between the glass components, camera image data, and camera intrinsics, as well as conversion to other definition of coordinate systems.Note that this document is applicable to the NRSDK for Unity only, and does not apply to other types of NRSDK.

In the NRSDK for Unity, in terms of coordinate system's definition and the corresponding extrinsics, we use definition of the Unity coordinate system (left handed).

- The Nreal glass consists of the following key components

- 1.2 x Grayscale Cameras
- 2.2 x Display Cameras
- 3.Head / IMU
- 4.RGB Camera

- The placement of the above components and their corresponding coordinate systems, as defined in NRSDK for Unity, are as follows

- The global coordinate frame of the tracking system is as follows

- The following Interface returns the 6dof head pose with respect to the global frame, as defined above.

// Get the pose of device in unity world coordinate.

// "NRCameraRig" transform in Unity.

Pose headpose = NRFrame.HeadPose;

- The following Interface returns the 6dof extrinsics, as a transformation matrix, of a Device's coordinate frame expressed in the Head coordinate frame.

public class NRFrame

{

public static Pose GetDevicePoseFromHead(NativeDevice device);

}

- For example, given a vector's coordinate$P_{d}$in the Device's coordinate frame, and using the extrinsic transformation matrix${^{h}}{T}{_{d}}$obtained as above, we can compute the vector's coordinate$P_{h}$in the Head coordinate frame, by$P_{h}={^{h}}{T}{_{d}}*P_{d}$

The following example code gets the extrinsic transformation of RGB Camera in Head, and transforms a point's coordinate from the RGB camera frame to the Head frame.

// Get Pose RGBCamera From Head

Pose camPos = NRFrame.GetDevicePoseFromHead(NativeDevice.RGB_CAMERA);

// Translate Pose to Matrix4x4.

Matrix4x4 Head_T_cam = Matrix4x4.TRS(camPos.position, camPos.rotation, Vector3.one);

// Transform a vector from camera space to head space

Vector3 pInCam = new Vector3(1, 0, 0);

Vector3 pInHead = Head_T_cam.MultiplyPoint(pInCam);

For computer vision algorithm developers, it is often convenient to handle quantities expressed in the OpenCV coordinate system (right handed). Hereafter, we describe how to convert the aforementioned Unity coordinate systems and their corresponding extrinsics to the OpenCV convention. We also describe the definitions and interfaces for image data and camera intrinsics.

- In the OpenCV convention, the Nreal Glass components and their corresponding coordinate systems are as follows

- The definition difference between Unity and OpenCV coordinate systems for a camera is as follows

- Note that only the y-axis needs to be negated between these two conventions. Therefore, given an extrinsic transformation defined under the Unity coordinate systems, we can obtain the equivalent transformation defined under the OpenCV, by using the following utility function

public static Matrix4x4 UnityToCVMatrix(Matrix4x4 unityA_T_unityB)

{

Matrix4x4 cv_T_unity = Matrix4x4.Scale(new Vector3(1, -1, 1));

Matrix4x4 cvA_T_cvB = cv_T_unity * unityA_T_unityB * cv_T_unity.inverse;

return cvA_T_cvB;

}

The following example code first gets the extrinsic transformation of RGB Camera in Head, under the Unity coordinate systems as described earlier, and then converts it to the OpenCV coordinate systems by using the above utility function.

// Get Pose RGBCamera From Head

Pose camInHead = NRFrame.GetDevicePoseFromHead(NativeDevice.RGB_CAMERA);

// Translate Pose to Matrix4x4.

Matrix4x4 unityHead_T_unitycam =

Matrix4x4.TRS(camInHead.position, camInHead.rotation, Vector3.one);

// Convert from Unity to OpenCV

Matrix4x4 cvHead_T_cvcam = UnityToCVMatrix(unityHead_T_unitycam);

The following example code shows how to get the extrinsic transformation between the two Grayscale cameras and convert it to the OpenCV coordinate systems.

// Get Extrinsic Left Grayscale Camera From Head

Pose lCamPos = NRFrame.GetDevicePoseFromHead(NativeDevice.LEFT_GRAYSCALE_CAMERA);

Matrix4x4 Head_T_Lcam = Matrix4x4.TRS(lCamPos.position, lCamPos.rotation, Vector3.one);

// Get Extrinsic Right Grayscale Camera From Head

Pose rCamPos = NRFrame.GetDevicePoseFromHead(NativeDevice.RIGHT_GRAYSCALE_CAMERA);

Matrix4x4 Head_T_Rcam = Matrix4x4.TRS(rCamPos.position, rCamPos.rotation, Vector3.one);

// Calculate Extrinsic Right Camera From Left

Matrix4x4 unityLcam_T_unityRcam = Head_T_Lcam.inverse * Head_T_Rcam;

// Convert Unity Extrinsic to CV

Matrix4x4 cvLcam_T_cvRcam = UnityToCVMatrix(unityLcam_T_unityRcam);

// Transform a vector from right camera to left camera

Vector3 pInRCam = new Vector3(1, 0, 0);

Vector3 pInLCam = cvLcam_T_cvRcam.MultiplyPoint(pInRCam);

- The definition of the image pixel coordinates and the camera intrinsics in the NRSDK follows the OpenCV convention.
- The image data is stored row-wise in memory as follows

- The camera intrinsic matrix$K$is composed of the focal lengths$f_x$and$f_y$, and the principal point$c_x$and$c_y$, expressed in pixel units.

$K=
\begin{bmatrix}
f_x&0&c_x\\
0&f_y&c_y\\
0&0&1
\end{bmatrix}$

- The distortion parameters contain radial coefficients$k_1, k_2,k_3,k_4,k_5$and tangential coefficients$p_1,p_2$. The order of NRDistortionParams is$(k_1,k_2,p_1,p_2,k_3,k_4,k_5)$.

- Raw image data can be obtained through NRRGBCamTexture or NRGrayCameraTexture for the RGBCamera or GrayCamera, respectively.

In the current version of NRSDK, one can use Texture2D to get the raw image data. The following example code uses GetRawTextureData to get raw data by accessing Texture2D from NRRGBCamTexture. The output raw data array stores the image pixel data row-wise as described above.

// Here are parts of the example code for using RGB camera, you can find the

// complete code in the CameraCaptureController file of NRSDK.

public class CameraCaptureController : MonoBehaviour

{

// Save the reference for Texture2D from NRRGBCamTexture.

Texture2D mTex2d;

// The instance of NRRGBCamTexture.

NRRGBCamTexture mCamTex;

void Start()

{

// Create an instance of NRRGBCamTexture

mCamTex = new NRRGBCamTexture();

// Get Texture2D target and save it.

mTex2d = mCamTex.GetTexture();

mCamTex.Play();

}

void LateUpdate()

{

// Get raw data from Texture2D per frame.

byte[] rawData = mTex2d.GetRawTextureData();

}

}

- The interfaces for getting camera intrinsics, distortion parameters, and resolution are as follows

public class NRFrame

{

// Get the intrinsic matrix of device.

public static NativeMat3f GetDeviceIntrinsicMatrix(NativeDevice device);

// Get the distortion coefficients of device.

public static NRDistortionParams GetDeviceDistortion(NativeDevice device);

// Get the resolution of device.

public static NativeResolution GetDeviceResolution(NativeDevice device);

}

The following example code gets the RGB camera's intrinsic matrix and distortion parameters as described above.

// Get the rgb camera's intrinsic matrix

NativeMat3f mat = NRFrame.GetDeviceIntrinsicMatrix(NativeDevice.RGB_CAMERA);

// Get the rgb camera's distortion coeffcients

NRDistortionParams distort = NRFrame.GetDeviceDistortion(NativeDevice.RGB_CAMERA);

Last modified 5mo ago