Table of Contents

Evaluation Protocol

Interested researchers should first start their readings with the following paper:

F. Pomerleau, F. Colas, R. Siegwart, and S. Magnenat, Comparing ICP variants on real-world data sets, Autonomous Robots, Online First, Feb. 2013. Publisher Link, bibtex

This page presents only summary of the evaluation protocol with the required files for the evaluation.

Download

Here is the list of evaluation files and 3D scans for each environment:

Environment Download
3D Scans Protocol Validation
ETH data eth_protocol.csv eth_validation.csv
Apartment data apartment_protocol.csv apartment_validation.csv
Stairs data stairs_protocol.csv stairs_validation.csv
Wood Summer data wood_protocol.csv wood_validation.csv
Gazebo Winter data gazebo_protocol.csv gazebo_validation.csv
Mountain Plain data plain_protocol.csv plain_validation.csv

How to Use the Evaluation Files

The protocol header has the following elements:

reference_name reading_name iT00 iT01 iT02 … iT33
Hokuyo_0.csv Hokuyo_1.csv 1.0 0.0 0.0 … 1.0
Hokuyo_0.csv Hokuyo_1.csv 0.98 -0.14 0.009 … 1.0
… … … … … … …

File names in the column reading_name and reference_name correspond to point cloud file name express in their own local coordinates. So files named Hokuyo_<i>.csv can be expressed as follow:

P_{i} = \begin{bmatrix} x_0 \quad & x_1 \quad & x_2 \quad & x_3 \quad & ... \quad & x_k \quad \\ y_0 \quad & y_1 \quad & y_2 \quad & y_3 \quad & ... \quad & y_k \quad \\ z_0 \quad & z_1 \quad & z_2 \quad & z_3 \quad & ... \quad & z_k \quad \\ 1 \quad & 1 \quad & 1 \quad & 1 \quad & ... \quad & 1 \quad \\ \end{bmatrix} , where $k+1$ is the number of lines in the file.

The validation header has the following elements:

overlap_ratio perturbation_type gT00 gT01 gT02 … gT33
0.308 easyPoses 1.0 0.0 0.0 … 1.0
0.858 mediumPoses 0.98 -0.14 0.009 … 1.0
… … … … … … …

The columns named iTxx are the initial transformation matrices $\check{T}_{i_{ref} \leftarrow j_{read}}$ that need to be applied to the reading point cloud before registration. The same pair of scans is used multiple time with different initialization to evaluate the robustness of a registration solution against prior transformation. The columns named gTxx are the transformations matrices of the ground truth $T_{i_{ref} \leftarrow j_{read}}$.

A transformation matrix can be deserialized using the columns T00 to T33 as follow:

T = \begin{bmatrix} \text{T00} \quad & \text{T01} \quad & \text{T02} \quad & \text{T03} \quad \\ \text{T10} \quad & \text{T11} \quad & \text{T12} \quad & \text{T13} \quad \\ \text{T20} \quad & \text{T21} \quad & \text{T22} \quad & \text{T23} \quad \\ \text{T30} \quad & \text{T31} \quad & \text{T32} \quad & \text{T33} \quad \\ \end{bmatrix}

Protocol

Using the first line of the Evaluation file presented above as an example, the computation should be as follow:

  1. Load the reading point cloud $P_{1}$ from the file Hokuyo_1.csv, which is expressed in coordinate frame 1.
  2. Load the reference point cloud $Q_{0}$ from the file Hokuyo_0.csv, which is expressed in coordinate frame 0.
  3. Load the prior transformation $\check{T}_{0 \leftarrow 1}$ (i.e., initial guess) from the protocol file (columns iT00 to iT33).
  4. Apply your registration algorithm that should estimate the transformation to express the point cloud reading in reference:

    $$\hat{T}_{0 \leftarrow 1} = registration(Q_{0}, P_{1}, \check{T}_{0 \leftarrow 1}),$$

    where $Q_{0}$ is the reference point cloud, $P_{1}$ is the reading point cloud, and $\check{T}_{0 \leftarrow 1}$ is the prior transformation of $P_{1}$.

  5. Load the ground truth transformation $T_{0 \leftarrow 1}$ from the validation file (columns gT00 to gT33).
  6. Compute the difference between the estimation and the ground truth transformation $T_{0 \leftarrow 1}$:

    $$ \Delta T = \begin{bmatrix} r_{00} \quad & r_{01} \quad & r_{02} \quad & \Delta x \quad \\ r_{10} \quad & r_{11} \quad & r_{12} \quad & \Delta y \quad \\ r_{20} \quad & r_{21} \quad & r_{22} \quad & \Delta z \quad \\ 0 \quad & 0 \quad & 0 \quad & 1 \quad \\ \end{bmatrix} = \hat{T}_{0 \leftarrow init} * T_{0 \leftarrow 1}^{-1}$$

  7. Compute the translation error:

    $$e_{trans} = \sqrt{\Delta x^2 + \Delta y^2 + \Delta z^2}$$

  8. Compute the rotational error. For that, we use the angular component of the Euler axis representation:

    $$ e_{rot} = \arccos \Bigg( \frac{1}{2} (r_{00} + r_{11} + r_{22} - 1)\Bigg) = \arccos \Bigg( \frac{trace(\Delta T)}{2} - 1\Bigg)$$

  9. Repeat for all lines in the Protocol File.

How to Write your Results

The result file should be a comma-separated value (csv) file with a comma ', ' as separator and a header. Here is an example of the expected format:

time T00 T01 T02 … T33
6.15 1.0 0.0 0.0 … 1.0
2.78 0.98 -0.14 0.009 … 1.0
… … … … … …

The field time is expressed in second and should include all preprocessing steps required by the solution except the loading time from disk to memory. The columns Txx follow the same serialization format defined in the prior section. The 4×4 transformation matrix is the result of the registration process $\hat{T}_{0 \leftarrow 1}$. Then, there should be one result per test which means that the number of lines in a Result file should be the same as in the Evaluation file. We suggest to name the file with the environment name as in this example for a test run on apartment: apartment_chen91.csv.

You can send your results to François Pomerleau so we can publish them on this web site.