Converting RGB8 buffer into ArrayFire compatible layout

[IvanChepurnyi]

I was trying to implement GPU accelerated image resize function by using ArrayFire with the code like this:

use image::{open, GenericImageView, DynamicImage};
use arrayfire;
use std::env::args;
use arrayfire::*;

fn main() {
    let file = args().nth(1).unwrap();
    let target = args().nth(2).unwrap();

    let original_image =  open(file.clone()).unwrap();

    let array_fire_image: Array<u8> = Array::new(
        original_image.as_rgb8().unwrap().as_ref(),
        Dim4::new(
            &[original_image.height().into(), original_image.width().into(), 3, 1]
        )
    );

    let resized_image = resize(
        &array_fire_image,
        (original_image.height() / 2).into(),
        (original_image.width() / 2).into(),
        InterpType::NEAREST
    );

    let mut new_image:DynamicImage = DynamicImage::new_rgb8(
        (original_image.width() / 2).into(),
        (original_image.height() / 2).into()
    );

    resized_image.host( &mut new_image.as_mut_rgb8().unwrap());

    new_image.save( target ).unwrap();
}

Although the compute run successfully the result is far from the expected.
It appears that colors and pixel layout are different in ArrayFire image processing functions and require the prior transformation of RGB8 byte layout into something different.

The problem is that the layout is not documented and reading C++ source code for the library makes it hard for noobs like me to make any sense of it. Any help or direction would be greatly appreciated.

[9prady9]

ArrayFire's built-in imageio functions automatically convert input color pixels, triplets(RGB) or quartets(RGBA), to a non-interleaved data i.e. for example, a bunch of RGB(s) will get converted to three sets of R(s) followed by G(s) and then by B(s). Another way to think about this is, a matrix of RGB triplets is converted to three matrices of R, G, B stacked on top of each other. This is done to extract maximum performance and it suits the GPU memory accesses work.

Since, you are using an external crate that is loading/saving the images for you. When you pass the data to Array::new you need to take care of this interleaved format conversion. There is simple solution to this problem however. It will add two extra steps to your above code.

   let array_fire_image: Array<u8> = // ... 

    let reordered_img = reorder_v2(&array_fire_image, 2, 1, Some(vec![0, 3]));

    let resized_image = // ...

    let out_img = reorder_v2(&resized_image, 2, 1, Some(vec![0, 3]));

    let mut new_image:DynamicImage = // ... 

    out_img.host( &mut new_image.as_mut_rgb8().unwrap());

[IvanChepurnyi]

Thanks for the explanation @9prady9. The reorder seems like it worked, but the internal structures of reordered array not the same as load_image from imageio. Also reorder function documentation is a bit cryptic as there is no indication what are behaviours behind each argument.

Here is a code I used to test order of the array elements:

use image::{open, GenericImageView, DynamicImage};
use arrayfire;
use std::env::args;
use arrayfire::*;

fn main() {
    let file = args().nth(1).unwrap();

    let original_image =  open(file.clone()).unwrap();

    let array_fire_image: Array<u8> = Array::new(
        original_image.as_rgb8().unwrap().as_ref(),
        Dim4::new(
            &[original_image.height().into(), original_image.width().into(), 3, 1]
        )
    );

    let fire_image: Array<u8> = load_image(file.clone(), true);

    let reordered_image = reorder_v2(
        &array_fire_image,
        2,
        1,
        Some(vec![0, 3])
    );

    af_print!("Original RGB Array", array_fire_image);
    af_print!("Reordered RGB Array", reordered_image);
    af_print!("Loaded image via imageio api", fire_image);
}

The code output for this image

Original RGB Array
[5 5 3 1]
       255          0          0        255          0 
         0        255          0          0        255 
         0          0        255          0          0 
       255          0          0        255          0 
         0        255          0          0        255 

         0          0        255          0          0 
         0          0          0        255        255 
       255        255          0          0          0 
         0          0        255          0          0 
         0          0          0        255        255 

         0        255          0          0        255 
         0          0        255          0          0 
       255          0          0        255          0 
         0        255          0          0        255 
         0          0        255          0          0 


Reordered RGB Array
[3 5 5 1]
       255          0          0        255          0 
         0          0        255          0          0 
         0        255          0          0        255 

         0        255          0          0        255 
         0          0          0        255        255 
         0          0        255          0          0 

         0          0        255          0          0 
       255        255          0          0          0 
       255          0          0        255          0 

       255          0          0        255          0 
         0          0        255          0          0 
         0        255          0          0        255 

         0        255          0          0        255 
         0          0          0        255        255 
         0          0        255          0          0 


Loaded image via imageio api
[5 5 3 1]
       255        255        255        255        255 
       255        255        255        255        255 
         0          0          0          0          0 
         0          0          0          0          0 
         0          0          0          0          0 

         0          0          0          0          0 
         0          0          0          0          0 
         0          0          0          0          0 
       255        255        255        255        255 
       255        255        255        255        255 

         0          0          0          0          0 
         0          0          0          0          0 
       255        255        255        255        255 
         0          0          0          0          0 
         0          0          0          0          0

[9prady9]

yes, the documentation could use improvements. reorder basically means reshaping the array. Couple of basic examples are available in upstream project. I have updated the docs of reorder in rust wrapper to have an example. I will add a better explanation soon.

Coming to the short example. My mistake, I explained it but I had a hidden minor change in Array::new in the code stub I shared earlier. Given below is full modified example.

    let original_image =  open(file.clone()).unwrap();

    let array_fire_image: Array<u8> = Array::new(
        original_image.as_rgb8().unwrap().as_ref(),
        Dim4::new(
            &[3, original_image.width().into(), original_image.height().into(), 1]
        )
    );
    let reordered_img = reorder_v2(&array_fire_image, 2, 1, Some(vec![0, 3]));

    let resized_image = resize(
        &reordered_img,
        (original_image.width() / 2).into(),
        (original_image.height() / 2).into(),
        InterpType::NEAREST
    );
    let out_img = reorder_v2(&resized_image, 2, 1, Some(vec![0, 3]));

    let mut new_image:DynamicImage = DynamicImage::new_rgb8(
        (original_image.width() / 2).into(),
        (original_image.height() / 2).into()
    );

    out_img.host( &mut new_image.as_mut_rgb8().unwrap());

Note:

1. Array::new has 3 as the first axis length because here pixel is made of RGB values
2. You don't need to swap the width and height axes like you had earlier. Number of pixels along a scan line is the length of second axis, followed by number of scan lines along third axis. So, the dimension of array_fire_image is [3 width height 1]. Thus a reorder is required to convert this interleaved format to [width height 3 1]

[9prady9]

#217

[IvanChepurnyi]

This example works like a charm! Thank you very much for your help.