[Performance]Why is loading an ONNX model taking so long?

[MoniqueSciortino]

Describe the issue

I am using Visual Studio 2022 and onnxruntime downloaded from Manage NuGet Packages. My onnx file of size 915MB (random forest with 500 trees) is taking ages to load using the following code.

To reproduce

#include
#include <onnxruntime_cxx_api.h>

using namespace std;

int main() {
const std::string model_s = "C:/Users/moniq/Downloads/MgarrRFModel.onnx";
std::basic_string<ORTCHAR_T> model = std::basic_string<ORTCHAR_T>(model_s.begin(), model_s.end());

// onnxruntime setup
Ort::Env env(ORT_LOGGING_LEVEL_WARNING, "ONNXModelLoader");
Ort::SessionOptions session_options;
session_options.SetIntraOpNumThreads(4);
Ort::Session session = Ort::Session(env, model.c_str(), session_options);
std::cout << "number of model input:" << session.GetInputCount() << std::endl;
std::cout << "number of model Output:" << session.GetOutputCount() << std::endl;

return 0;

}

Urgency

Urgent

Platform

Windows

OS Version

24h2

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

latest on

ONNX Runtime API

C++

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

Model File

No response

Is this a quantized model?

Yes

[yuslepukhin]

Without seeing a model, it is hard to say. However, if your model has lots of functions, it may take time to inline them. Optimizations may take time.
My suggestion is to pre-optimize your model with either ONNX optimizer or ONNXRuntime optimizer and save it for repeated runs, your loading time will decrease dramatically.

[MoniqueSciortino]

Thanks for your suggestion. Can you provide help on how to optimize it please?

[yuslepukhin]

You try this in C++ (although in Python it is quicker)

[MoniqueSciortino]

Thank you, I'll try it out

[MoniqueSciortino]

Is there any way I can share the .onnx file with you? With optimization, it is still taking quite a long time (almost two hours - more than without optimization).

[yuslepukhin]

You can zip it up and put on a cloud provider of your choice.