ORTOptimizer for the model type table-transformer

[zachmayer]

Feature request

I'd like to add support for the new microsoft table transformers to ORTOptimizer

Motivation

These table transformers are object detection models. They have a backbone that's a CNN, so at the very least I think we could optimize the CNN part of these models?

Your contribution

Here is my sample script:

import os
from pathlib import Path

import torch
from optimum.onnxruntime import ORTModel, ORTOptimizer
from optimum.onnxruntime.configuration import OptimizationConfig
from transformers import AutoImageProcessor, TableTransformerForObjectDetection

# Define the input model and the output directory
model_name = "microsoft/table-transformer-detection"
onnx_path = Path("~/Downloads/onnx_model/").expanduser()
onnx_model_name = "table_transformer.onnx"

# Load the Processor and save its config
processor = AutoImageProcessor.from_pretrained(model_name)
processor.save_pretrained(onnx_path)

# Load the model and save its config
model = TableTransformerForObjectDetection.from_pretrained(
    "microsoft/table-transformer-detection"
)
model.eval()
model.config.save_pretrained(onnx_path)

# Convert the model to onnx
torch.onnx.export(
    model,
    torch.rand((1, 3, 800, 600)),
    os.path.join(onnx_path, onnx_model_name),
    export_params=True,
    do_constant_folding=True,
    opset_version=11,
    input_names=["pixel_values"],
    output_names=["max_probability"],
    dynamic_axes={
        "pixel_values": {0: "batch_size", 2: "height", 3: "width"},
        "max_probability": {0: "batch_size"},
    },
)

# Optimize the onnx model
onnx_model = ORTModel.from_pretrained(onnx_path, file_name=onnx_model_name)
optimizer = ORTOptimizer.from_pretrained(onnx_model)
optimizer.optimize(
    save_dir=onnx_path,
    optimization_config=OptimizationConfig(optimization_level=99),
)

Which raises NotImplementedError: Tried to use ORTOptimizer for the model type table-transformer, but it is not available yet. Please open an issue or submit a PR at https://github.com/huggingface/optimum.

I'm more than happy to make PR, if someone could share some example PRs I could use as a template or give me some guidance as to how to start.

[mht-sharma]

Hi @zachmayer thanks for interest in adding the table-transformer model in ORTOptimizer.

Here are the steps to follow:

• Register the model in NormalizedConfigManager
• Register the model in ORTConfigManager.
• Also add a test to confirm that it works.

Check more detail in the very good first issue here #351.

Let me know if you require additional details 🙂

[zachmayer]

I made a PR here for segformers. once that's merged I'll add table transformers!

#1820

[zachmayer]

segformers PR merged!

[zachmayer]

@mht-sharma — I don't see a modeling ORT for table-transformers (like there was with segformers). Does this mean I'll need to make one?

https://github.com/huggingface/optimum/blob/main/optimum/onnxruntime/modeling_ort.py

[zachmayer]

I took a shot at this. What do you think? #1900