Add batched_run to Engine

[mgoin]

Refactors Pipeline.split_engine_inputs and Pipeline.join_engine_outputs into standalone functions in utils/data.py so that Engine can also use them as batch_run (as an alternative to run)

>>> import deepsparse
>>> engine = deepsparse.Engine("zoo:cv/classification/mobilenet_v1-1.0/pytorch/sparseml/imagenet/base-none")
>>> import numpy as np
>>> inp = [np.zeros((4,3,224,224), dtype=np.float32)]
>>> inp[0].shape
(4, 3, 224, 224)

>>> engine.run(inp)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nm/drive1/home/mgoin/code/deepsparse/src/deepsparse/engine.py", line 503, in run
    self._validate_inputs(inp)
  File "/nm/drive1/home/mgoin/code/deepsparse/src/deepsparse/engine.py", line 688, in _validate_inputs
    raise ValueError(
ValueError: array batch size of 4 must match the batch size the model was instantiated with 1

>>> engine.batched_run(inp)
2023-07-11 15:40:43 deepsparse.engine WARNING  Using batched_run with an Engine of batch_size=1 isn't recommended for optimal performance.
[array([[-1.346764  ,  0.7402935 , -0.6495765 , ..., -1.8527336 ,
        -0.29401016,  0.99657345],
       [-1.346764  ,  0.7402935 , -0.6495765 , ..., -1.8527336 ,
        -0.29401016,  0.99657345],
       [-1.346764  ,  0.7402935 , -0.6495765 , ..., -1.8527336 ,
        -0.29401016,  0.99657345],
       [-1.346764  ,  0.7402935 , -0.6495765 , ..., -1.8527336 ,
        -0.29401016,  0.99657345]], dtype=float32), array([[1.1318562e-04, 9.1240724e-04, 2.2728769e-04, ..., 6.8241949e-05,
        3.2433696e-04, 1.1789345e-03],
       [1.1318562e-04, 9.1240724e-04, 2.2728769e-04, ..., 6.8241949e-05,
        3.2433696e-04, 1.1789345e-03],
       [1.1318562e-04, 9.1240724e-04, 2.2728769e-04, ..., 6.8241949e-05,
        3.2433696e-04, 1.1789345e-03],
       [1.1318562e-04, 9.1240724e-04, 2.2728769e-04, ..., 6.8241949e-05,
        3.2433696e-04, 1.1789345e-03]], dtype=float32)]

[bfineran]

FYI - we have some similar functionality in the pipeline (fails if input batch size isn't divisible by compiled batch size):

deepsparse/src/deepsparse/pipeline.py

Line 259 in 58f250c

batches = self.split_engine_inputs(engine_inputs, self._batch_size)

could look into migrating

[mgoin]

Great callout @bfineran , I refactored those functions with my padding functionality into utils/data.py so both Engine and Pipeline can use thme

[bfineran]

see comment. LGTM- nice tests