Runtime(模型推理Runtime)

fastdeploy.Runtime

class fastdeploy.Runtime(runtime_option)

FastDeploy Runtime object.

Initialize a FastDeploy Runtime object.

Parameters

runtime_option – (fastdeploy.RuntimeOption)Options for FastDeploy Runtime

bind_input_tensor(name, fdtensor)

Bind FDTensor by name, no copy and share input memory

Parameters

• name – (str)The name of input data.

• fdtensor – (fastdeploy.FDTensor)The input FDTensor.

bind_output_tensor(name, fdtensor)

Bind FDTensor by name, no copy and share output memory

Parameters

• name – (str)The name of output data.

• fdtensor – (fastdeploy.FDTensor)The output FDTensor.

compile(warm_datas)

[Only for Poros backend] compile with prewarm data for poros

Parameters

data – (list[str : numpy.ndarray])The prewarm data list

:return TorchScript Model

forward(*inputs)

[Only for Poros backend] Inference with input data for poros

Parameters

data – (list[str : numpy.ndarray])The input data list

:return list of numpy.ndarray

get_input_info(index)

Get input information of the loaded model.

Parameters

index – (int)Index of the input

:return fastdeploy.TensorInfo

get_output_info(index)

Get output information of the loaded model.

Parameters

index – (int)Index of the output

:return fastdeploy.TensorInfo

get_output_tensor(name)

Get output FDTensor by name, no copy and share backend output memory

Parameters

name – (str)The name of output data.

:return fastdeploy.FDTensor

get_profile_time()

Get profile time of Runtime after the profile process is done.

infer(data)

Inference with input data.

Parameters

data – (dict[str : numpy.ndarray])The input data dict, key value must keep same with the loaded model

:return list of numpy.ndarray

num_inputs()

Get number of inputs of the loaded model.

num_outputs()

Get number of outputs of the loaded model.

zero_copy_infer()

No params inference the model.

the input and output data need to pass through the bind_input_tensor and get_output_tensor interfaces.