Remove skip building graph check for quantization use case

Remove skip building graph check for quantization use case #4919