HDDM_A and HDDM_W with DriftRetrainingClassifier and progressive_val_score

[ngabrek]

Hi there,

I'm currently evaluating different concept drift detection methods in combination with the GaussianNB() on different datasets (e.g. all Insects, Elec2, and Phishing). However, the HDDM_A and HDDM_W methods are the only detectors, which do not detect any concept drift (an example of my code is attached). Any other detection methods from this package detect concept drift and therefor adapt the Naive Bayes model when using the DriftRetrainingClassifier.

Because I also want to evaluate additional concept drift detection methods, not available within the River package, I'm adapting the concept drift detection methods from the Frouros package to be compatible with River (you can find the adapted function from the HDDMA below), especially with the progressive_val_score(). When using the HDDMA and HDDMW detectors from Frouros, the classifier is adapted and concept drifts are detected. Even after comparing their implementation, I don't understand why the HDDMA and HDDMW in Frouros and River achieve different results, since their default settings are the same. And those are the only two detectors, with which I faced the problem. With ADWIN or DDM for example the results were the same or at least similar.

Does anyone know, why the HDDM_A and HDDM_W do not detect any concept drift in this evaluation set-up?

Thank you,

Nadine

Adapted functions of the HDDMA in Frouros to be compatible with River

def init( # noqa: D107
self,
config: Optional[HDDMAConfig] = None,
callbacks: Optional[
Union[BaseCallbackStreaming, List[BaseCallbackStreaming]]
] = None,
) -> None:
super().init(
config=config,
callbacks=callbacks,
)
self.additional_vars = {
"test_type": (
HoeffdingTwoSidedTest(
alpha_d=self.config.alpha_d, # type: ignore
alpha_w=self.config.alpha_w, # type: ignore
)
if self.config.two_sided_test # type: ignore
else HoeffdingOneSidedTest(
alpha_d=self.config.alpha_d, # type: ignore
alpha_w=self.config.alpha_w, # type: ignore
)
),
**self.additional_vars, # type: ignore
}
self._set_additional_vars_callback()
#This part is added to be compatible with River
self.warning_detected = False
self.drift_detected = False

def _update(self, value: Union[int, float], **kwargs) -> None:
    #added, since otherwise the detector from Frouros only detect the first concept drift 
    if self.drift:
        self.reset()
        
    self.num_instances += 1

    self.test_type.z.update(value=value)
    self.test_type.set_initial_cut_mean()

    epsilon_z = self.test_type.hoeffding_error_bound(
        num_values=self.test_type.z.num_values
    )
    self.test_type.update_cut_point(epsilon_z=epsilon_z)

    if self.num_instances >= self.config.min_num_instances:
        drift_flag, warning_flag = (
            self.test_type.check_cases()
            if self.test_type.z.num_values != self.test_type.x.num_values
            else (False, False)
        )
        if drift_flag:
            # Out-of-Control
            self.drift = True
            self.warning = False
            self.test_type.reset()
        else:
            if warning_flag:
                # Warning
                self.warning = True
            else:
                # In-Control
                self.warning = False
            self.drift = False
    else:
        self.drift, self.warning = False, False
    #added 
    self.warning_detected = self.warning
    self.drift_detected = self.drift

#added 
def reset(self) -> None:
    super().reset()

[smastelini]

Hi @ngabrek, that indeed sounds strange. I will check as soon as possible.

[smastelini]

@ngabrek, an update in this issue. I think I tracked the issue down to the reset strategies. I am using the MOA implementation as reference. Currently, my available time is scarse but I intend to submit a PR with a fix in the next couple days. I just need to perform some additional tests.

[smastelini]

I created #1318 to address the problem.

[ngabrek]

@smastelini thank you! The fix solved the problem.