===== Evaluation Protocol ===== Interested researchers should first start their readings with the following paper: F. Pomerleau, F. Colas, R. Siegwart, and S. Magnenat, //Comparing ICP variants on real-world data sets//, **Autonomous Robots**, Online First, Feb. 2013. [[http://link.springer.com/article/10.1007/s10514-013-9327-2|Publisher Link]], [[bibtex::comparing_icp|bibtex]] This page presents only summary of the evaluation protocol with the required files for the evaluation. ==== Download ==== Here is the list of evaluation files and 3D scans for each environment: ^ Environment ^ ^ Download ^^^ ^ ::: ^:::^ 3D Scans ^ Protocol ^ Validation ^ | ETH | | [[laserregistration:eth_hauptgebaude:home#point_clouds_in_base_frame|data]] | [[http://robotics.ethz.ch/~asl-datasets/evaluations/protocols/eth_protocol.csv|eth_protocol.csv]] | [[http://robotics.ethz.ch/~asl-datasets/evaluations/validation/eth_validation.csv|eth_validation.csv]] | | Apartment |:::| [[laserregistration:apartment:home#point_clouds_in_base_frame|data]] | [[http://robotics.ethz.ch/~asl-datasets/evaluations/protocols/apartment_protocol.csv|apartment_protocol.csv]] | [[http://robotics.ethz.ch/~asl-datasets/evaluations/validation/apartment_validation.csv|apartment_validation.csv]] | | Stairs |:::| [[laserregistration:stairs:home#point_clouds_in_base_frame|data]] | [[http://robotics.ethz.ch/~asl-datasets/evaluations/protocols/stairs_protocol.csv|stairs_protocol.csv]] | [[http://robotics.ethz.ch/~asl-datasets/evaluations/validation/stairs_validation.csv|stairs_validation.csv]] | | Wood Summer |:::| [[laserregistration:wood_summer:home#point_clouds_in_base_frame|data]] | [[http://robotics.ethz.ch/~asl-datasets/evaluations/protocols/wood_protocol.csv|wood_protocol.csv]] | [[http://robotics.ethz.ch/~asl-datasets/evaluations/validation/wood_validation.csv|wood_validation.csv]] | | Gazebo Winter |:::| [[laserregistration:gazebo_winter:home#point_clouds_in_base_frame|data]] | [[http://robotics.ethz.ch/~asl-datasets/evaluations/protocols/gazebo_protocol.csv|gazebo_protocol.csv]] | [[http://robotics.ethz.ch/~asl-datasets/evaluations/validation/gazebo_validation.csv|gazebo_validation.csv]] | | Mountain Plain |:::| [[laserregistration:mountain_plain:home#point_clouds_in_base_frame|data]] | [[http://robotics.ethz.ch/~asl-datasets/evaluations/protocols/plain_protocol.csv|plain_protocol.csv]] | [[http://robotics.ethz.ch/~asl-datasets/evaluations/validation/plain_validation.csv|plain_validation.csv]] | ==== How to Use the Evaluation Files ==== The **protocol** header has the following elements: ^ reference_name ^ reading_name ^ iT00 ^ iT01 ^ iT02 ^ ... ^ iT33 ^ |Hokuyo_0.csv |Hokuyo_1.csv | 1.0 | 0.0 | 0.0 | ... | 1.0 | |Hokuyo_0.csv |Hokuyo_1.csv | 0.98 | -0.14 | 0.009| ... | 1.0 | | ... | ... | ... | ... | ... | ... | ... | File names in the column //reading_name// and //reference_name// correspond to point cloud file name express in their own local coordinates. So files named Hokuyo_.csv can be expressed as follow: P_{i} = \begin{bmatrix} x_0 \quad & x_1 \quad & x_2 \quad & x_3 \quad & ... \quad & x_k \quad \\ y_0 \quad & y_1 \quad & y_2 \quad & y_3 \quad & ... \quad & y_k \quad \\ z_0 \quad & z_1 \quad & z_2 \quad & z_3 \quad & ... \quad & z_k \quad \\ 1 \quad & 1 \quad & 1 \quad & 1 \quad & ... \quad & 1 \quad \\ \end{bmatrix} , where $k+1$ is the number of lines in the file. The **validation** header has the following elements: ^ overlap_ratio ^ perturbation_type ^ gT00 ^ gT01 ^ gT02 ^ ... ^ gT33 ^ |0.308 |easyPoses | 1.0 | 0.0 | 0.0 | ... | 1.0 | |0.858 |mediumPoses | 0.98 | -0.14 | 0.009| ... | 1.0 | | ... | ... | ... | ... | ... | ... | ... | The columns named ''iTxx'' are the initial transformation matrices $\check{T}_{i_{ref} \leftarrow j_{read}}$ that need to be applied to the //reading// point cloud **before** registration. The same pair of scans is used multiple time with different initialization to evaluate the robustness of a registration solution against prior transformation. The columns named ''gTxx'' are the transformations matrices of the ground truth $T_{i_{ref} \leftarrow j_{read}}$. A transformation matrix can be deserialized using the columns ''T00'' to ''T33'' as follow: T = \begin{bmatrix} \text{T00} \quad & \text{T01} \quad & \text{T02} \quad & \text{T03} \quad \\ \text{T10} \quad & \text{T11} \quad & \text{T12} \quad & \text{T13} \quad \\ \text{T20} \quad & \text{T21} \quad & \text{T22} \quad & \text{T23} \quad \\ \text{T30} \quad & \text{T31} \quad & \text{T32} \quad & \text{T33} \quad \\ \end{bmatrix} ==== Protocol ==== Using the first line of the Evaluation file presented above as an example, the computation should be as follow: - Load the reading point cloud $P_{1}$ from the file Hokuyo_1.csv, which is expressed in coordinate frame 1. - Load the reference point cloud $Q_{0}$ from the file Hokuyo_0.csv, which is expressed in coordinate frame 0. - Load the prior transformation $\check{T}_{0 \leftarrow 1}$ (i.e., initial guess) from the **protocol file** (columns ''iT00'' to ''iT33''). - Apply your registration algorithm that should estimate the transformation to express the point cloud //reading// in //reference//: $$\hat{T}_{0 \leftarrow 1} = registration(Q_{0}, P_{1}, \check{T}_{0 \leftarrow 1}),$$ where $Q_{0}$ is the reference point cloud, $P_{1}$ is the reading point cloud, and $\check{T}_{0 \leftarrow 1}$ is the prior transformation of $P_{1}$. - Load the ground truth transformation $T_{0 \leftarrow 1}$ from the **validation file** (columns ''gT00'' to ''gT33''). - Compute the difference between the estimation and the ground truth transformation $T_{0 \leftarrow 1}$: $$ \Delta T = \begin{bmatrix} r_{00} \quad & r_{01} \quad & r_{02} \quad & \Delta x \quad \\ r_{10} \quad & r_{11} \quad & r_{12} \quad & \Delta y \quad \\ r_{20} \quad & r_{21} \quad & r_{22} \quad & \Delta z \quad \\ 0 \quad & 0 \quad & 0 \quad & 1 \quad \\ \end{bmatrix} = \hat{T}_{0 \leftarrow init} * T_{0 \leftarrow 1}^{-1}$$ - Compute the translation error: $$e_{trans} = \sqrt{\Delta x^2 + \Delta y^2 + \Delta z^2}$$ - Compute the rotational error. For that, we use the angular component of the Euler axis representation: $$ e_{rot} = \arccos \Bigg( \frac{1}{2} (r_{00} + r_{11} + r_{22} - 1)\Bigg) = \arccos \Bigg( \frac{trace(\Delta T)}{2} - 1\Bigg)$$ - Repeat for all lines in the Protocol File. ==== How to Write your Results ==== The result file should be a comma-separated value (csv) file with a comma '' ', ' '' as separator and a header. Here is an example of the expected format: ^ time ^ T00 ^ T01 ^ T02 ^ ... ^ T33 ^ |6.15 | 1.0 | 0.0 | 0.0 | ... | 1.0 | |2.78 | 0.98 | -0.14 | 0.009| ... | 1.0 | | ... | ... | ... | ... | ... | ... | The field //time// is expressed in second and should include all preprocessing steps required by the solution except the loading time from disk to memory. The columns //Txx// follow the same serialization format defined in the prior section. The 4x4 transformation matrix is the result of the registration process $\hat{T}_{0 \leftarrow 1}$. Then, there should be one result per test which means that the number of lines in a Result file should be the same as in the Evaluation file. We suggest to name the file with the environment name as in this example for a test run on apartment: [[http://robotics.ethz.ch/~asl-datasets/evaluations/results/chen91/apartment_chen91.csv|apartment_chen91.csv]]. You can send your results to [[http://www.asl.ethz.ch/people/pomerlef | François Pomerleau]] so we can publish them on this web site. ===== Related Links ===== * [[laserregistration:evaluations:home| Evaluation Results]] * [[laserregistration:laserregistration | Data set descriptions]] * [[hardware:tiltinglaser | Recording sensor description]]