[v2] Getting password prompt for task and tests

[sam-hey]

Branch: v2.0.0

Steps to Reproduce:

import mteb
from mteb import MTEB

model = mteb.get_model("intfloat/multilingual-e5-small")
tasks = mteb.get_tasks(tasks=["NFCorpus"], languages=["eng"])

eval_splits = ["test"]

evaluation = MTEB(tasks=tasks)

evaluation.run(
    model,
    eval_splits=eval_splits,
    corpus_chunk_size=500,
)

Error:

INFO:sentence_transformers.SentenceTransformer:Use pytorch device_name: mps
INFO:sentence_transformers.SentenceTransformer:Load pretrained SentenceTransformer: intfloat/multilingual-e5-small
INFO:mteb.models.sentence_transformer_wrapper:Model prompts will be overwritten with {'query': 'query: ', 'passage': 'passage: '}
INFO:mteb.evaluation.MTEB:

## Evaluating 1 tasks:
────────────────────────────────────────────────────────────────────── Selected tasks  ───────────────────────────────────────────────────────────────────────
Retrieval
    - NFCorpus, s2p


INFO:mteb.evaluation.MTEB:

********************** Evaluating NFCorpus **********************
INFO:mteb.abstasks.dataloaders:Loading Corpus...
INFO:mteb.abstasks.dataloaders:Loaded 3633 TEST Documents.
INFO:mteb.abstasks.dataloaders:Doc Example: {'id': 'MED-10', 'title': 'Statin Use and Breast Cancer Survival: A Nationwide Cohort Study from Finland', 'text': 'Recent studies have suggested that statins, an established drug group in the prevention of cardiovascular mortality, could delay or prevent breast cancer recurrence but the effect on disease-specific mortality remains unclear. We evaluated risk of breast cancer death among statin users in a population-based cohort of breast cancer patients. The study cohort included all newly diagnosed breast cancer patients in Finland during 1995–2003 (31,236 cases), identified from the Finnish Cancer Registry. Information on statin use before and after the diagnosis was obtained from a national prescription database. We used the Cox proportional hazards regression method to estimate mortality among statin users with statin use as time-dependent variable. A total of 4,151 participants had used statins. During the median follow-up of 3.25 years after the diagnosis (range 0.08–9.0 years) 6,011 participants died, of which 3,619 (60.2%) was due to breast cancer. After adjustment for age, tumor characteristics, and treatment selection, both post-diagnostic and pre-diagnostic statin use were associated with lowered risk of breast cancer death (HR 0.46, 95% CI 0.38–0.55 and HR 0.54, 95% CI 0.44–0.67, respectively). The risk decrease by post-diagnostic statin use was likely affected by healthy adherer bias; that is, the greater likelihood of dying cancer patients to discontinue statin use as the association was not clearly dose-dependent and observed already at low-dose/short-term use. The dose- and time-dependence of the survival benefit among pre-diagnostic statin users suggests a possible causal effect that should be evaluated further in a clinical trial testing statins’ effect on survival in breast cancer patients.'}
INFO:mteb.abstasks.dataloaders:Loading Queries...
INFO:mteb.abstasks.dataloaders:Loaded 323 TEST Queries.
INFO:mteb.abstasks.dataloaders:Query Example: {'id': 'PLAIN-2', 'text': 'Do Cholesterol Statin Drugs Cause Breast Cancer?'}
WARNING:mteb.evaluation.MTEB:Evaluating multiple MTEB runs simultaniously will produce incorrect CO₂ results
Password:

Same error for:make test

make test 
--- 🧪 Running tests ---
pytest -n auto --durations=5
==================================================================== test session starts =====================================================================
platform darwin -- Python 3.12.7, pytest-8.3.4, pluggy-1.5.0
rootdir: /Users/A200009373/Documents/Coding/mteb
configfile: pyproject.toml
plugins: cov-6.0.0, typeguard-4.4.1, anyio-4.7.0, xdist-3.6.1
10 workers [2476 items]   
...................................................................................................................................................... [  6%]
...................................................................................................................................................... [ 12%]
...................................................................................................................................................... [ 18%]
...................................................................................................................................................... [ 24%]
...................................................................................................................................................... [ 30%]
...................................................................................................................................................... [ 36%]
...................................................................................................................................................... [ 42%]
...................................................................................................................................................... [ 48%]
............................Password:..............................................

[sam-hey]

Disabling co2_tracker by setting it to False resolves this specific issue for me. However, the tests continue to fail due to the default behavior of co2_tracker. I strongly suspect the root cause lies in the EmissionsTracker implementation.

[sam-hey]

I believe this issue is critical: mlco2/codecarbon#736. It could become a significant blocker for releasing v2 in the near future, as it affects all Mac users.
mlco2/codecarbon#731

cc @KennethEnevoldsen @isaac-chung @Samoed

[KennethEnevoldsen]

I can't reproduce this:

git checkout v2.0.0
make install

and then running the above command. Is it possible that you need to allow it in system settings?

Regardless it is probably a thing we could consider disabling by default. Would love to know what the opinion is here.

[sam-hey]

@KennethEnevoldsen Currently, this issue affects MAC users with M1 chips or newer CPUs. Reverting to a non-default option could be a good solution if we prefer not to wait for the implementation of the non-sudo version.

To give sudo rights without having to enter a password each time, you can modify the sudoers file with the following command:

sudo visudo
Then add the following line at the end of the file:

username ALL = (root) NOPASSWD: /usr/bin/powermetrics
If you do not want to give sudo rights to your user, then CodeCarbon will fall back to constant mode to measure CPU energy consumption.

[KennethEnevoldsen]

I am using an M2 mac and I don't have the line:

username ALL = (root) NOPASSWD: /usr/bin/powermetrics

Though I might have aded another workaround somewhere, not sure where though.

[isaac-chung]

I'd prefer to stick with our decision to run Code carbon by default (see discussion here).

I have an M1 chip and can try reproducing the issue later.

[isaac-chung]

Actually, I have an M2 pro chip, and am able to see password coming up and failed tests by running make test on branch v2.0.0:

--- 🧪 Running tests ---
pytest -n auto --durations=5
============================================================= test session starts =============================================================
platform darwin -- Python 3.11.11, pytest-8.3.4, pluggy-1.5.0
rootdir: /Users/isaac.chung/work/mteb
configfile: pyproject.toml
plugins: cov-6.0.0, anyio-4.8.0, xdist-3.6.1
12 workers [2477 items]   
....................................................................................................................................... [  5%]
....................................................................................................................................... [ 10%]
....................................................................................................................................... [ 16%]
....................................................................................................................................... [ 21%]
....................................................................................................................................... [ 27%]
....................................................................................................................................... [ 32%]
....................................................................................................................................... [ 38%]
....................................................................................................................................... [ 43%]
....................................................................................................................................... [ 49%]
.....F........F................................F..Password:FFFFFFFFFFFFFFF...F.FF..........................................F.................... [ 54%]
................F....F..F.............................................................................................................. [ 59%]
....................................................................................................................................... [ 65%]
................................s.sss...ss.ssssssss.sssss.sss.......................................................sssssssssssssssssss [ 70%]
sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 76%]
sssssssssssssssssssssssssssssss.ssssssssssss.sssssss......................ssssss.................................ssss.................. [ 81%]
...........sssssss.........ss................ssssss...................sssssssssssssssssssssss..s......F................................ [ 87%]
.........................................F.F.F....

[KennethEnevoldsen]

Hmm not sure why I don't get the password, but seems like we either need a fix for it or to disable co2 tracking by default.