How to copy Ort::Value in c++?

[seungtaek94]

Hi.

I want to copy Ort::Value to other new Ort::Value like below.

But I dont know how to do that right way.

Ort::Value tensor = Ort::Value::CreateTensor(...);

Ort::Value new_tensor = tensort;

[RyanUnderhill]

There's no current way to copy an Ort::Value, but I'm curious why you'd want to?

The next best thing if it's a tensor would be to just create a new one with the data from the original one.

[seungtaek94]

Actually, I use two stage model that like below;

Stage1-1: Input1 -> model1 -> model1_output1
Stage1-2: Input2 -> model1 -> model1_output2
Stage2: (model1_output1, model1_output2) -> model 2 -> mode2_output

If I use model1_output1, model1_output2 as input for model2 diretly like below, it cause error.

std::vector<Ort::Value> model1_output1 =  model1->Run(...)
std::vector<Ort::Value> model1_output2 =  model1->Run(...)

std::vector<Ort::Value> model2_input;
model2_input.emplace_back(model1_output1);
model2_input.emplace_back(model1_output2);

std::vector<Ort::Value> model2_output = m_session->Run(Ort::RunOptions{ nullptr },
		m_vecInputNames.data(),
		model2_input.data(),
		m_vecInputNames.size(),
		m_vecOutputNames.data(),
		m_vecOutputNames.size());

So I get tensor data using tensor.GetTensorMutableData<float>() from model1_out1 and model1_out2 then create new tensor using the data, and it works.

I attached the code below that exactly same with above code block except model2_input.emplace_back(CopyTensor(model1_output1, memoryInfo)); and model2_input.emplace_back(CopyTensor(model1_output2, memoryInfo));

Ort::Value CopyTensor(Ort::Value& tensor, Ort::MemoryInfo & memoryInfo)
{
	return Ort::Value::CreateTensor<float>(memoryInfo,
		tensor.GetTensorMutableData<float>(),
		tensor.GetTensorTypeAndShapeInfo().GetElementCount(),
		tensor.GetTensorTypeAndShapeInfo().GetShape().data(),
		tensor.GetTensorTypeAndShapeInfo().GetShape().size());
}

std::vector<Ort::Value> model1_output1 =  model1->Run(...)
std::vector<Ort::Value> model1_output2 =  model1->Run(...)

std::vector<Ort::Value> model2_input;
model2_input.emplace_back(CopyTensor(model1_output1, memoryInfo));
model2_input.emplace_back(CopyTensor(model1_output2, memoryInfo));

std::vector<Ort::Value> model2_output = model2->Run(Ort::RunOptions{ nullptr },
		m_vecInputNames.data(),
		model2_input.data(),
		m_vecInputNames.size(),
		m_vecOutputNames.data(),
		m_vecOutputNames.size());

Is there any example for 2-stage model? Or any advice would be appreciated.

[RyanUnderhill]

Looks like my github email rule didn't handle Comments properly, I'll see replies sooner now.

I thought a little about it and the main issue is that Ort::Value is like std::unique_ptr, it has ownership over the tensor.

So I thought of a few other things you could try doing.

Instead of 'CopyTensor' you could do this:
model2_input.emplace_back(std::move(model1_output1[0]))

Or you could also use std::swap if you already created the std::vectorOrt::Value model2_input using 'nullptr' as the construction value (this creates trivially empty Ort::Value objects).

Then you'd need to move/swap it back to the input if you wanted to use it in a loop. Does that help any?

[seungtaek94]

Thanks for reply.

I think it will help for me :)

[vincentme]

There's no current way to copy an Ort::Value, but I'm curious why you'd want to?

The next best thing if it's a tensor would be to just create a new one with the data from the original one.

Hi, I would like to know how to "create a new one with the data from the original one"? Is this like the CopyTensor function defined above?

My situation is a bit different here. I have a processing tree that each node is a onnx model. The user can request to compute output from any of the nodes. Some node has a cache attribute, which requires the computed output from this node to be cached. In the next time, any descendants of the node is required, the computation can be start here. So I also need to some the outputTensors (std::vectorOrt::Value) in a map.

Thanks!

[LHcroix]

I had the same issue with the sam2 onnx model, it was solved using your function, Thanks for sharing