Use of texture (including color) to improve the detection of some objects

[albertodesouza]

The texture of the coke.obj model used in the Tutorials does not appear during training when I turn on LINEMOD_VIZ_IMG... Relevant code for visualization (linemod_train.cpp):

#if LINEMOD_VIZ_IMG
// Display the rendered image
if (*visualize_)
{
  cv::namedWindow("Rendering");
  if (!image.empty()) {
  cv::imshow("Rendering", image);
  cv::waitKey(1);
  }
}
#endif

I would like to use texture (including color) to improve detection of some objects...

Alberto.

[albertodesouza]

In line 212 of linemod_train.cpp, addTemplate() might return -1. This turns the indexes of the poses of the templates saved in the database out of sync with the id of the detected objects. This typically shows up in rviz as not properly oriented object detections. To correct the problem I suggest changing this peace of code:

detector_->addTemplate(sources, "object1", mask);


// Also store the pose of each template
Rs_->push_back(cv::Mat(R));
Ts_->push_back(cv::Mat(T));
distances_->push_back(distance);
Ks_->push_back(cv::Mat(K));

to that one:

int template_in = detector_->addTemplate(sources, "object1", mask);
if (template_in == -1)
{
  // Delete the status
  for (size_t j = 0; j < status.str().size(); ++j)
    std::cout << '\b';
  continue;
}

// Also store the pose of each template
Rs_->push_back(cv::Mat(R));
Ts_->push_back(cv::Mat(T));
distances_->push_back(distance);
Ks_->push_back(cv::Mat(K));

Alberto.

[bmagyar]

Hi Alberto,

regarding the title of this issue, I'd direct you to the paper of LINEMOD.
http://campar.cs.tum.edu/pub/hinterstoisser2011linemod/hinterstoisser2011linemod.pdf

I would rather add a post-filtering step after the linemod module in a pipeline fashion (in the end that is what ORK is for) that gets rid of wrong detections. If you prepare such a system, I'm sure many would apprieciate using it/ reading about it on the mailing list.

Cheers,

[albertodesouza]

Hi Bence,

On Wed, Jan 28, 2015 at 7:22 AM, Bence Magyar [email protected]
wrote:

Hi Alberto,

regarding the title of this issue, I'd direct you to the paper of LINEMOD.

http://campar.cs.tum.edu/pub/hinterstoisser2011linemod/hinterstoisser2011linemod.pdf

I have read the paper (quite a good one). If one examines it, one can see
that the authors used color images (see first paragraph of page 4). Linemod
is good for textureless objects, but it uses color texture (in OpenGL
sense) to improve performance on color textured objects.

The current implementation of OpenCV linemod (used by ORK linemod) uses
color gradients, as described in the paper (see
http://docs.opencv.org/ref/master/d7/d7a/linemod_8hpp.html, class
cv::linemod::ColorGradient
http://docs.opencv.org/ref/master/dc/d64/classcv_1_1linemod_1_1ColorGradient.html).
Actualy, ORK linemod calls OpenCV linemod passing color images as parameter
(see
https://github.com/wg-perception/linemod/blob/master/src/linemod_detect.cpp).
They just do not have color due to some bug in the OpenGL redering code.

I would rather add a post-filtering step after the linemod module in a
pipeline fashion (in the end that is what ORK is for) that gets rid of
wrong detections.

The use of colors in the center of the object in a post-processing step
might increase precision, but note that the color gradients in the borders
(which are used by linemod) are very important for color objects in any
case.

Even in the case of a post-linemod color processing pipeline step, a color
(OpenGL texture) model is required and, in the current version of the code,
this is not working. The code for texture processing is there, it is just
not working.

If you prepare such a system, I'm sure many would apprieciate using it/
reading about it on the mailing list.

I would like to see the benefits of using color texture, as proposed in the
original paper, before going for a post-processing pipeline step. The
performance of the code as it is now is significantly inferior than the
results obtained by the authors and, I believe, one of the reasons is the
lack of color texture processing. So, any help on making it work will be
greatly appreciated.

I am in for an extra pipeline step for reducing false positives as well,
but we will need OpenGL texture processing for that...

Cheers,

Alberto.

Cheers,

—
Reply to this email directly or view it on GitHub
#10 (comment)
.

[bmagyar]

Hi,
Now I understand you. For the first read it wasn't clear to me in which context you used 'texture'. Now I see you meant texture in an OpeGL model sense, and color in the context of linemod.

[albertodesouza]

Hi,

On Fri, Jan 30, 2015 at 3:41 AM, Bence Magyar [email protected]
wrote:

Hi,
Now I understand you. For the first read it wasn't clear to me in which
context you used 'texture'. Now I see you meant texture in an OpeGL model
sense, and color in the context of linemod.

Great! Did you ever see the object model in color during linemod training?
The code for texturing is there, it is just not working... The coke.obj
file loads OK if you start the training command in the same directory of
coke.obj.

Alberto.

—
Reply to this email directly or view it on GitHub
#10 (comment)
.

[gypsytearss]

The solution we found involved solving two problems: (1) enabling texture in OpenGL and (2) fixing the recursive call to Model::recursive_render. Here's the quick solution we used:

Model::recursive_render(const struct aiScene *sc, const aiNode* nd, const int j) const
{
  int i;
  unsigned int n = 0, t;
  aiMatrix4x4 m = nd->mTransformation;

  // update transform
  aiTransposeMatrix4(&m);
  glPushMatrix();
  glMultMatrixf((float*) &m);

  // draw all meshes assigned to this node
  for (; n < nd->mNumMeshes; ++n)
  {
    glEnable(GL_TEXTURE_2D);
    const struct aiMesh* mesh = sc->mMeshes[nd->mMeshes[n]];

    if (n < texturesAndPaths.size())
      glBindTexture(GL_TEXTURE_2D, texturesAndPaths[j].hTexture);

    apply_material(sc->mMaterials[mesh->mMaterialIndex]);

    if (mesh->mNormals == NULL)
      glDisable (GL_LIGHTING);
    else
      glEnable(GL_LIGHTING);

    if (mesh->mColors[0] != NULL)
      glEnable (GL_COLOR_MATERIAL);
    else
      glDisable(GL_COLOR_MATERIAL);

    for (t = 0; t < mesh->mNumFaces; ++t)
    {
      const struct aiFace* face = &mesh->mFaces[t];
      GLenum face_mode;

      switch (face->mNumIndices)
      {
        case 1:
          face_mode = GL_POINTS;
          break;
        case 2:
          face_mode = GL_LINES;
          break;
        case 3:
          face_mode = GL_TRIANGLES;
          break;
        default:
          face_mode = GL_POLYGON;
          break;
      }

      glBegin(face_mode);

      for (i = 0; i < face->mNumIndices; i++)
      {
        int index = face->mIndices[i];
        if (mesh->mColors[0] != NULL)
          Color4f(&mesh->mColors[0][index]);
        if (mesh->mNormals != NULL)
          glNormal3fv(&mesh->mNormals[index].x);
        if (mesh->HasTextureCoords(0))
          glTexCoord2f(mesh->mTextureCoords[0][index].x, mesh->mTextureCoords[0][index].y);
        glVertex3fv(&mesh->mVertices[index].x);
      }

      glEnd();
    }
  }

  // draw all children
  for (n = 0; n < nd->mNumChildren; ++n)
    recursive_render(sc, nd->mChildren[n], n);

  glPopMatrix();
}

But note that with any outside calls to the recursive_render function (e.g. in Model::Draw), you'll have to specify a third parameter (typically 0).

Colin

[vrabaud]

That looks good. So that we check better the differences, can you please make a proper pull request ? Thx

[bhack]

See also @mtamburrano linergb branches at https://github.com/mtamburrano/lineRGB