aidevelopementtoolkit.torch_utils.AbstractValidator

AbstractValidationStep

Bases: ABC

This abstract class serves as a blueprint for creating specific validation step classes that define how a model is evaluated on a single batch of data. Encapsulating this logic outside of the validator allows the same AbstractValidator subclass to be reused simply by swapping the validation step implementation that is passed to its constructor.

Parameters:
  • loss_fn (Module) –

    The loss function used in the validation process.

  • device (str or device) –

    The device (e.g., 'cpu' or 'cuda:0') on which the model and data will be loaded for validation.

Source code in aidevelopementtoolkit/torch_utils/AbstractValidator.py
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
class AbstractValidationStep(ABC):
    """This abstract class serves as a blueprint for creating specific validation 
    step classes that define how a model is evaluated on a single batch of data. 
    Encapsulating this logic outside of the validator allows the same 
    `AbstractValidator` subclass to be reused simply by swapping the validation 
    step implementation that is passed to its constructor.

    Parameters
    ----------
    loss_fn : nn.Module
        The loss function used in the validation process.

    device : str or torch.device
        The device (e.g., 'cpu' or 'cuda:0') on which the model and data will 
        be loaded for validation.
    """

    def __init__(
            self, 
            loss_fn: nn.Module, 
            device: Union[str, torch.device],
        ) -> None:

        self.loss_fn = loss_fn
        self.device = torch.device(device) if isinstance(device, str) else device


    @abstractmethod
    def __call__(
            self, 
            model: nn.Module, 
            batch: Tuple[torch.Tensor, ...], 
            **kwargs: Dict[str, Any],
        ) -> Any:
        """This abstract method defines the validation step for a 
        single batch of data. It must be implemented in subclasses to specify 
        how the model is evaluated based on the input batch.

        Parameters
        ----------
        model : nn.Module
            The model to be validated.

        batch : Tuple[torch.Tensor, ...]
            A tuple containing the input data and eventual other tensors (e.g., labels) 
            for a single batch.

        **kwargs : Dict[str, Any]
            Additional keyword arguments that subclasses can use to receive extra 
            information needed for a specific validation step implementation.

        Returns
        -------
        Any
            The output of the validation step, which can be any type depending on the 
            specific implementation in the subclass.
        """
        logger.error("Subclasses must implement this method.", NotImplementedError)

__call__(model: nn.Module, batch: Tuple[torch.Tensor, ...], **kwargs: Dict[str, Any]) -> Any abstractmethod

This abstract method defines the validation step for a single batch of data. It must be implemented in subclasses to specify how the model is evaluated based on the input batch.

Parameters:
  • model (Module) –

    The model to be validated.

  • batch (Tuple[Tensor, ...]) –

    A tuple containing the input data and eventual other tensors (e.g., labels) for a single batch.

  • **kwargs (Dict[str, Any], default: {} ) –

    Additional keyword arguments that subclasses can use to receive extra information needed for a specific validation step implementation.

Returns:
  • Any –

    The output of the validation step, which can be any type depending on the specific implementation in the subclass.

Source code in aidevelopementtoolkit/torch_utils/AbstractValidator.py
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
@abstractmethod
def __call__(
        self, 
        model: nn.Module, 
        batch: Tuple[torch.Tensor, ...], 
        **kwargs: Dict[str, Any],
    ) -> Any:
    """This abstract method defines the validation step for a 
    single batch of data. It must be implemented in subclasses to specify 
    how the model is evaluated based on the input batch.

    Parameters
    ----------
    model : nn.Module
        The model to be validated.

    batch : Tuple[torch.Tensor, ...]
        A tuple containing the input data and eventual other tensors (e.g., labels) 
        for a single batch.

    **kwargs : Dict[str, Any]
        Additional keyword arguments that subclasses can use to receive extra 
        information needed for a specific validation step implementation.

    Returns
    -------
    Any
        The output of the validation step, which can be any type depending on the 
        specific implementation in the subclass.
    """
    logger.error("Subclasses must implement this method.", NotImplementedError)

AbstractValidator

Bases: ABC

This abstract class serves as a blueprint for creating specific validator classes that handle the validation process of machine learning models using PyTorch. It defines the essential components and methods that any concrete validator class must implement, namely the overall validation loop, while the per-batch validation logic is delegated to an AbstractValidationStep instance passed at construction time. This allows the same validator class to be reused simply by changing the validation step implementation.

Notes

This can be used also in distributed validation scenarios, but the implementation of the validate method should take into account the distributed setting. We suggest to check the example :doc:examples/mnist_training.py for a reference implementation of a validator that can be used in distributed validation.

Parameters:
  • validation_step (AbstractValidationStep) –

    The validation step used to evaluate the model's performance on a single batch of data.

  • device (str or device) –

    The device (e.g., 'cpu' or 'cuda:0') on which the model and data will be loaded for validation.

Source code in aidevelopementtoolkit/torch_utils/AbstractValidator.py
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
class AbstractValidator(ABC):
    """This abstract class serves as a blueprint for creating specific validator classes
    that handle the validation process of machine learning models using PyTorch.
    It defines the essential components and methods that any concrete validator class 
    must implement, namely the overall validation loop, while the per-batch validation 
    logic is delegated to an `AbstractValidationStep` instance passed at construction 
    time. This allows the same validator class to be reused simply by changing the 
    validation step implementation.

    Notes
    -----
    This can be used also in distributed validation scenarios, but the implementation 
    of the `validate` method should take into account the distributed setting.
    We suggest to check the example :doc:`examples/mnist_training.py` for a reference 
    implementation of a validator that can be used in distributed validation.

    Parameters
    ----------
    validation_step : AbstractValidationStep
        The validation step used to evaluate the model's performance on a single batch of data.

    device : str or torch.device
        The device (e.g., 'cpu' or 'cuda:0') on which the model and data will 
        be loaded for validation.
    """

    def __init__(
            self, 
            validation_step: AbstractValidationStep,
            device: Union[str, torch.device],
        ) -> None:

        self.validation_step = validation_step
        self.device = torch.device(device) if isinstance(device, str) else device


    @abstractmethod
    def validate(
            self, 
            model: nn.Module, 
            data_loader: torch.utils.data.DataLoader, 
            current_epoch: int,
            **kwargs: Any,
        ) -> Any:
        """Validate the model on the given data loader.

        Parameters
        ----------
        model : nn.Module
            The model to be validated.

        data_loader : torch.utils.data.DataLoader
            The data loader providing the validation data.

        current_epoch : int
            The current epoch number, which can be used for logging or checkpointing purposes.

        **kwargs : Dict[str, Any]
            Additional keyword arguments that subclasses can use to receive extra 
            information needed for a specific validation loop implementation.

        Returns
        -------
        Any
            The result of the validation, which can be any type depending on the 
            specific implementation in the subclass.
        """
        logger.error("Subclasses must implement this method.", NotImplementedError)

validate(model: nn.Module, data_loader: torch.utils.data.DataLoader, current_epoch: int, **kwargs: Any) -> Any abstractmethod

Validate the model on the given data loader.

Parameters:
  • model (Module) –

    The model to be validated.

  • data_loader (DataLoader) –

    The data loader providing the validation data.

  • current_epoch (int) –

    The current epoch number, which can be used for logging or checkpointing purposes.

  • **kwargs (Dict[str, Any], default: {} ) –

    Additional keyword arguments that subclasses can use to receive extra information needed for a specific validation loop implementation.

Returns:
  • Any –

    The result of the validation, which can be any type depending on the specific implementation in the subclass.

Source code in aidevelopementtoolkit/torch_utils/AbstractValidator.py
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
@abstractmethod
def validate(
        self, 
        model: nn.Module, 
        data_loader: torch.utils.data.DataLoader, 
        current_epoch: int,
        **kwargs: Any,
    ) -> Any:
    """Validate the model on the given data loader.

    Parameters
    ----------
    model : nn.Module
        The model to be validated.

    data_loader : torch.utils.data.DataLoader
        The data loader providing the validation data.

    current_epoch : int
        The current epoch number, which can be used for logging or checkpointing purposes.

    **kwargs : Dict[str, Any]
        Additional keyword arguments that subclasses can use to receive extra 
        information needed for a specific validation loop implementation.

    Returns
    -------
    Any
        The result of the validation, which can be any type depending on the 
        specific implementation in the subclass.
    """
    logger.error("Subclasses must implement this method.", NotImplementedError)