khoj plugin in obsidian ends with error ensure khoj backend is running and khoj url is pointing to it in plugin settings

[arisugawaa]

khoj plugin in obsidian ends with error ensure khoj backend is running and khoj url is pointing to it in plugin settings

i have followed the steps stated in the reddit post and in the repo demo:
executed pip install khoj-assistant && khoj --no-gui in terminal
and installed and turned on khoj obsidian plugin
error appears when i try to use the khoj search feature

What do you see in your terminal when your start the khoj backend with khoj --no-gui -vv
C:\Python310\lib\site-packages\src\utils\state.py:34: UserWarning: Failed to initialize NumPy: module compiled against API version 0x10 but this version of numpy is 0xf (Triggered internally at ..\torch\csrc\utils\tensor_numpy.cpp:77.)
device = torch.device("cpu")
INFO: 2023-02-12 16:07:54,679: src | Starting Khoj...
WARNING: 2023-02-12 16:07:58,204: src.configure | Khoj is not configured.
Configure it via khoj GUI, plugins or by editing C:\Users\jonat.khoj\khoj.yml.
INFO: Started server process [22640]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)

What is your current operating system?
Windows 10

What is your current python version with python -V, current pip version pip -V and khoj version with khoj -V?
Python 3.10.9
pip 23.0 from C:\Python310\lib\site-packages\pip (python 3.10)
C:\Python310\lib\site-packages\src\utils\state.py:34: UserWarning: Failed to initialize NumPy: module compiled against API version 0x10 but this version of numpy is 0xf (Triggered internally at ..\torch\csrc\utils\tensor_numpy.cpp:77.)
device = torch.device("cpu")
usage: khoj [-h] [--config-file CONFIG_FILE] [--no-gui] [--regenerate] [--verbose] [--host HOST] [--port PORT]
[--socket SOCKET] [--version]
khoj: error: unrecognized arguments: version with khoj

my khoj url in obsidian is http://localhost:8000

[debanjum]

Hi @arisugawaa, thanks for sharing details in a github discussion

C:\Python310\lib\site-packages\src\utils\state.py:34: UserWarning: Failed to initialize NumPy: module compiled against API version 0x10 but this version of numpy is 0xf (Triggered internally at ..\torch\csrc\utils\tensor_numpy.cpp:77.)

This is a problem as the correct numpy package version required by khoj is not loaded
To resolve this, I've removed the sub-dependency requirements in the latest pre-release version of khoj at c156b3e
To use the latest pre-release version of khoj run the following command in your windows cmd prompt:

pip install --upgrade --pre khoj-assistant
khoj --no-gui -vv

[arisugawaa]

it seems i get an error when trying to run the "khoj --no-gui -vv" command
backend issue was solved, but could not search. tried to disable and enable the plugin in obsidian, then the backend error returned.

log:
C:\Users\jonat>khoj --no-gui -vv
C:\Python310\lib\site-packages\src\utils\state.py:34: UserWarning: Failed to initialize NumPy: module compiled against API version 0x10 but this version of numpy is 0xf (Triggered internally at ..\torch\csrc\utils\tensor_numpy.cpp:77.)
device = torch.device("cpu")
INFO: 2023-02-13 03:53:12,398: src | Starting Khoj...
Traceback (most recent call last):
File "C:\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Python310\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Python310\Scripts\khoj.exe_main.py", line 7, in
File "C:\Python310\lib\site-packages\src\main.py", line 80, in run
configure_server(args, required=False)
File "C:\Python310\lib\site-packages\src\configure.py", line 42, in configure_server
state.model = configure_search(state.model, state.config, args.regenerate)
File "C:\Python310\lib\site-packages\src\configure.py", line 78, in configure_search
model.markdown_search = text_search.setup(
File "C:\Python310\lib\site-packages\src\search_type\text_search.py", line 150, in setup
previous_entries = extract_entries(config.compressed_jsonl) if config.compressed_jsonl.exists() and not regenerate else None
File "C:\Python310\lib\site-packages\src\search_type\text_search.py", line 58, in extract_entries
return list(map(Entry.from_dict, load_jsonl(jsonl_file)))
File "C:\Python310\lib\site-packages\src\utils\jsonl.py", line 27, in load_jsonl
for line in jsonl_file:
File "C:\Python310\lib\codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x95 in position 1691: invalid start byte

[debanjum]

UserWarning: Failed to initialize NumPy: module compiled against API version 0x10 but this version of numpy is 0xf

You still seem to be hitting this issue. This could be because another existing package depends on an older numpy version.

1. Can you try run force upgrade numpy with pip install --upgrade numpy and see what it returns?
2. If above steps fails because of conflicts. You'll need to run khoj from a virtualenv to avoid the conflict. See pipx for details
   1. Uninstall khoj with pip uninstall khoj-assistant
   2. Install pipx with python -m pip install --user pipx
   3. Install khoj backend in a isolated virtual environment (without dependency conflicts) run:
      pipx install --pip-args "'--pre'" khoj-assistant
   4. Start khoj backend with khoj --no-gui -vv
   5. Re-install/re-enable khoj obsidian plugin

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x95 in position 1691: invalid start byte

I pushed a (potential) fix for this issue just now in 11517ba.
To verify that it fixes this error of yours, can you follow these steps:

1. Delete the .khoj folder in your home folder (on windows at C:\Users\<your-windows-username>\.khoj)
2. Upgrade to the latest khoj with pip install --upgrade --pre khoj-assistant
3. Restart the khoj backend with khoj --no-gui -vv
4. Re-install/re-enable your Khoj obsidian plugin

[arisugawaa]

Can you try run force upgrade numpy with pip install --upgrade numpy and see what it returns?

C:\Users\jonat>pip install --upgrade numpy
Requirement already satisfied: numpy in c:\python310\lib\site-packages (1.22.4)
Collecting numpy
Downloading numpy-1.24.2-cp310-cp310-win_amd64.whl (14.8 MB)
---------------------------------------- 14.8/14.8 MB 14.5 MB/s eta 0:00:00
Installing collected packages: numpy
Attempting uninstall: numpy
Found existing installation: numpy 1.22.4
Uninstalling numpy-1.22.4:
Successfully uninstalled numpy-1.22.4
Successfully installed numpy-1.24.2

at this point it still did not work, remained the same.

after uninstalling khoj and installing pipx with the commands given,
when trying to execute

but when following the pipx installation instructions, i could not open the pipx.exe. (it would open and immediately close)
and all pipx commands would return as

C:\Users\jonat>pipx ensurepath
'pipx' is not recognized as an internal or external command,
operable program or batch file.

or

C:\Users\jonat>pipx install --pip-args "'--pre'" khoj-assistant
'pipx' is not recognized as an internal or external command,
operable program or batch file.

as for the

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x95 in position 1691: invalid start byte

deleted the .khoj file with any issue

when entering (second time, did not copy down first time)

pip install --upgrade --pre khoj-assistant

C:\Users\jonat>pip install --upgrade --pre khoj-assistant
Requirement already satisfied: khoj-assistant in c:\python310\lib\site-packages (0.3.0a1676245372)
Requirement already satisfied: defusedxml==0.7.1 in c:\python310\lib\site-packages (from khoj-assistant) (0.7.1)
Requirement already satisfied: pyyaml==6.0 in c:\python310\lib\site-packages (from khoj-assistant) (6.0)
Requirement already satisfied: sentence-transformers==2.2.2 in c:\python310\lib\site-packages (from khoj-assistant) (2.2.2)
Requirement already satisfied: jinja2==3.1.2 in c:\python310\lib\site-packages (from khoj-assistant) (3.1.2)
Requirement already satisfied: dateparser==1.1.1 in c:\python310\lib\site-packages (from khoj-assistant) (1.1.1)
Requirement already satisfied: pydantic==1.9.1 in c:\python310\lib\site-packages (from khoj-assistant) (1.9.1)
Requirement already satisfied: schedule==1.1.0 in c:\python310\lib\site-packages (from khoj-assistant) (1.1.0)
Requirement already satisfied: pytest==7.1.2 in c:\python310\lib\site-packages (from khoj-assistant) (7.1.2)
Requirement already satisfied: uvicorn==0.17.6 in c:\python310\lib\site-packages (from khoj-assistant) (0.17.6)
Requirement already satisfied: fastapi==0.77.1 in c:\python310\lib\site-packages (from khoj-assistant) (0.77.1)
Requirement already satisfied: openai==0.20.0 in c:\python310\lib\site-packages (from khoj-assistant) (0.20.0)
Requirement already satisfied: pillow==9.3.0 in c:\python310\lib\site-packages (from khoj-assistant) (9.3.0)
Requirement already satisfied: pyqt6==6.3.1 in c:\python310\lib\site-packages (from khoj-assistant) (6.3.1)
Requirement already satisfied: torch==1.13.1 in c:\python310\lib\site-packages (from khoj-assistant) (1.13.1)
Requirement already satisfied: regex!=2019.02.19,!=2021.8.27,<2022.3.15 in c:\python310\lib\site-packages (from dateparser==1.1.1->khoj-assistant) (2022.3.2)
Requirement already satisfied: pytz in c:\python310\lib\site-packages (from dateparser==1.1.1->khoj-assistant) (2022.7.1)
Requirement already satisfied: python-dateutil in c:\python310\lib\site-packages (from dateparser==1.1.1->khoj-assistant) (2.8.2)
Requirement already satisfied: tzlocal in c:\python310\lib\site-packages (from dateparser==1.1.1->khoj-assistant) (4.2)
Requirement already satisfied: starlette==0.19.1 in c:\python310\lib\site-packages (from fastapi==0.77.1->khoj-assistant) (0.19.1)
Requirement already satisfied: MarkupSafe>=2.0 in c:\python310\lib\site-packages (from jinja2==3.1.2->khoj-assistant) (2.1.2)
Requirement already satisfied: tqdm in c:\python310\lib\site-packages (from openai==0.20.0->khoj-assistant) (4.64.1)
Requirement already satisfied: requests>=2.20 in c:\python310\lib\site-packages (from openai==0.20.0->khoj-assistant) (2.28.2)
Requirement already satisfied: openpyxl>=3.0.7 in c:\python310\lib\site-packages (from openai==0.20.0->khoj-assistant) (3.1.0)
Requirement already satisfied: pandas>=1.2.3 in c:\python310\lib\site-packages (from openai==0.20.0->khoj-assistant) (1.5.3)
Requirement already satisfied: pandas-stubs>=1.1.0.11 in c:\python310\lib\site-packages (from openai==0.20.0->khoj-assistant) (1.5.3.230203)
Requirement already satisfied: typing-extensions>=3.7.4.3 in c:\python310\lib\site-packages (from pydantic==1.9.1->khoj-assistant) (4.4.0)
Requirement already satisfied: PyQt6-Qt6>=6.3.0 in c:\python310\lib\site-packages (from pyqt6==6.3.1->khoj-assistant) (6.4.2)
Requirement already satisfied: PyQt6-sip<14,>=13.4 in c:\python310\lib\site-packages (from pyqt6==6.3.1->khoj-assistant) (13.4.1)
Requirement already satisfied: atomicwrites>=1.0 in c:\python310\lib\site-packages (from pytest==7.1.2->khoj-assistant) (1.4.1)
Requirement already satisfied: iniconfig in c:\python310\lib\site-packages (from pytest==7.1.2->khoj-assistant) (2.0.0)
Requirement already satisfied: tomli>=1.0.0 in c:\python310\lib\site-packages (from pytest==7.1.2->khoj-assistant) (2.0.1)
Requirement already satisfied: colorama in c:\python310\lib\site-packages (from pytest==7.1.2->khoj-assistant) (0.4.6)
Requirement already satisfied: packaging in c:\python310\lib\site-packages (from pytest==7.1.2->khoj-assistant) (23.0)
Requirement already satisfied: pluggy<2.0,>=0.12 in c:\python310\lib\site-packages (from pytest==7.1.2->khoj-assistant) (1.0.0)
Requirement already satisfied: attrs>=19.2.0 in c:\python310\lib\site-packages (from pytest==7.1.2->khoj-assistant) (22.2.0)
Requirement already satisfied: py>=1.8.2 in c:\python310\lib\site-packages (from pytest==7.1.2->khoj-assistant) (1.11.0)
Requirement already satisfied: transformers<5.0.0,>=4.6.0 in c:\python310\lib\site-packages (from sentence-transformers==2.2.2->khoj-assistant) (4.21.0)
Requirement already satisfied: torchvision in c:\python310\lib\site-packages (from sentence-transformers==2.2.2->khoj-assistant) (0.14.1)
Requirement already satisfied: numpy in c:\python310\lib\site-packages (from sentence-transformers==2.2.2->khoj-assistant) (1.24.2)
Requirement already satisfied: scikit-learn in c:\python310\lib\site-packages (from sentence-transformers==2.2.2->khoj-assistant) (1.2.1)
Requirement already satisfied: scipy in c:\python310\lib\site-packages (from sentence-transformers==2.2.2->khoj-assistant) (1.10.0)
Requirement already satisfied: nltk in c:\python310\lib\site-packages (from sentence-transformers==2.2.2->khoj-assistant) (3.8.1)
Requirement already satisfied: sentencepiece in c:\python310\lib\site-packages (from sentence-transformers==2.2.2->khoj-assistant) (0.1.97)
Requirement already satisfied: huggingface-hub>=0.4.0 in c:\python310\lib\site-packages (from sentence-transformers==2.2.2->khoj-assistant) (0.8.1)
Requirement already satisfied: h11>=0.8 in c:\python310\lib\site-packages (from uvicorn==0.17.6->khoj-assistant) (0.14.0)
Requirement already satisfied: click>=7.0 in c:\python310\lib\site-packages (from uvicorn==0.17.6->khoj-assistant) (8.1.3)
Requirement already satisfied: asgiref>=3.4.0 in c:\python310\lib\site-packages (from uvicorn==0.17.6->khoj-assistant) (3.6.0)
Requirement already satisfied: anyio<5,>=3.4.0 in c:\python310\lib\site-packages (from starlette==0.19.1->fastapi==0.77.1->khoj-assistant) (3.6.2)
Requirement already satisfied: filelock in c:\python310\lib\site-packages (from huggingface-hub>=0.4.0->sentence-transformers==2.2.2->khoj-assistant) (3.9.0)
Requirement already satisfied: et-xmlfile in c:\python310\lib\site-packages (from openpyxl>=3.0.7->openai==0.20.0->khoj-assistant) (1.1.0)
Requirement already satisfied: types-pytz>=2022.1.1 in c:\python310\lib\site-packages (from pandas-stubs>=1.1.0.11->openai==0.20.0->khoj-assistant) (2022.7.1.0)
Requirement already satisfied: six>=1.5 in c:\python310\lib\site-packages (from python-dateutil->dateparser==1.1.1->khoj-assistant) (1.16.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\python310\lib\site-packages (from requests>=2.20->openai==0.20.0->khoj-assistant) (1.26.14)
Requirement already satisfied: certifi>=2017.4.17 in c:\python310\lib\site-packages (from requests>=2.20->openai==0.20.0->khoj-assistant) (2022.12.7)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\python310\lib\site-packages (from requests>=2.20->openai==0.20.0->khoj-assistant) (3.0.1)
Requirement already satisfied: idna<4,>=2.5 in c:\python310\lib\site-packages (from requests>=2.20->openai==0.20.0->khoj-assistant) (3.4)
Requirement already satisfied: tokenizers!=0.11.3,<0.13,>=0.11.1 in c:\python310\lib\site-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers==2.2.2->khoj-assistant) (0.12.1)
Requirement already satisfied: joblib in c:\python310\lib\site-packages (from nltk->sentence-transformers==2.2.2->khoj-assistant) (1.2.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in c:\python310\lib\site-packages (from scikit-learn->sentence-transformers==2.2.2->khoj-assistant) (3.1.0)
Requirement already satisfied: pytz-deprecation-shim in c:\python310\lib\site-packages (from tzlocal->dateparser==1.1.1->khoj-assistant) (0.1.0.post0)
Requirement already satisfied: tzdata in c:\python310\lib\site-packages (from tzlocal->dateparser==1.1.1->khoj-assistant) (2022.7)
Requirement already satisfied: sniffio>=1.1 in c:\python310\lib\site-packages (from anyio<5,>=3.4.0->starlette==0.19.1->fastapi==0.77.1->khoj-assistant) (1.3.0)

and upon running
khoj --no-gui -vv
the backend issue was solved and the search bar could be used; however it could not find anything. (tried looking for 'limitations of the african accent', which was one of my recent notes)

C:\Users\jonat>khoj --no-gui -vv
INFO: 2023-02-13 12:12:41,560: src | Starting Khoj...
WARNING: 2023-02-13 12:12:41,561: src.configure | Khoj is not configured.
Configure it via khoj GUI, plugins or by editing C:\Users\jonat.khoj\khoj.yml.
INFO: Started server process [19336]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO: 127.0.0.1:50366 - "GET /api/config/data HTTP/1.1" 200 OK
INFO: 127.0.0.1:50366 - "GET /api/config/data/default HTTP/1.1" 200 OK
INFO: 127.0.0.1:50366 - "GET /api/config/data/default HTTP/1.1" 200 OK
INFO: 127.0.0.1:50366 - "POST /api/config/data HTTP/1.1" 200 OK
INFO: 127.0.0.1:50367 - "GET /api/search?q=t&n=6&r=false&t=markdown HTTP/1.1" 200 OK
INFO: 127.0.0.1:50367 - "GET /api/search?q=te&n=6&r=false&t=markdown HTTP/1.1" 200 OK
INFO: 127.0.0.1:50367 - "GET /api/search?q=tes&n=6&r=false&t=markdown HTTP/1.1" 200 OK
INFO: 127.0.0.1:50367 - "GET /api/search?q=test&n=6&r=false&t=markdown HTTP/1.1" 200 OK
INFO: 2023-02-13 12:13:10,851: src.routers.api | Return response from query cache
INFO: 127.0.0.1:50367 - "GET /api/search?q=test%20&n=6&r=false&t=markdown HTTP/1.1" 200 OK
INFO: 2023-02-13 12:13:12,227: src.routers.api | Return response from query cache
INFO: 127.0.0.1:50367 - "GET /api/search?q=test&n=6&r=false&t=markdown HTTP/1.1" 200 OK
INFO: 2023-02-13 12:13:12,371: src.routers.api | Return response from query cache
INFO: 127.0.0.1:50367 - "GET /api/search?q=tes&n=6&r=false&t=markdown HTTP/1.1" 200 OK
INFO: 2023-02-13 12:13:12,562: src.routers.api | Return response from query cache
INFO: 127.0.0.1:50367 - "GET /api/search?q=te&n=6&r=false&t=markdown HTTP/1.1" 200 OK
INFO: 2023-02-13 12:13:12,691: src.routers.api | Return response from query cache
INFO: 127.0.0.1:50367 - "GET /api/search?q=t&n=6&r=false&t=markdown HTTP/1.1" 200 OK
INFO: 2023-02-13 12:13:12,916: src.routers.api | No query param (q) passed in API call to initiate search
INFO: 127.0.0.1:50367 - "GET /api/search?q=&n=6&r=false&t=markdown HTTP/1.1" 200 OK
INFO: 127.0.0.1:50367 - "GET /api/search?q=a&n=6&r=false&t=markdown HTTP/1.1" 200 OK
INFO: 127.0.0.1:50367 - "GET /api/search?q=af&n=6&r=false&t=markdown HTTP/1.1" 200 OK
INFO: 127.0.0.1:50367 - "GET /api/search?q=afr&n=6&r=false&t=markdown HTTP/1.1" 200 OK
INFO: 127.0.0.1:50367 - "GET /api/search?q=afri&n=6&r=false&t=markdown HTTP/1.1" 200 OK
INFO: 127.0.0.1:50367 - "GET /api/search?q=afric&n=6&r=false&t=markdown HTTP/1.1" 200 OK
INFO: 127.0.0.1:50367 - "GET /api/search?q=africa&n=6&r=false&t=markdown HTTP/1.1" 200 OK
INFO: 127.0.0.1:50367 - "GET /api/search?q=african&n=6&r=false&t=markdown HTTP/1.1" 200 OK
INFO: 2023-02-13 12:13:14,010: src.routers.api | Return response from query cache
INFO: 127.0.0.1:50367 - "GET /api/search?q=african%20&n=6&r=false&t=markdown HTTP/1.1" 200 OK
INFO: 127.0.0.1:50367 - "GET /api/search?q=african%20a&n=6&r=false&t=markdown HTTP/1.1" 200 OK
INFO: 127.0.0.1:50367 - "GET /api/search?q=african%20ac&n=6&r=false&t=markdown HTTP/1.1" 200 OK
INFO: 127.0.0.1:50367 - "GET /api/search?q=african%20acc&n=6&r=false&t=markdown HTTP/1.1" 200 OK
INFO: 127.0.0.1:50367 - "GET /api/search?q=african%20acce&n=6&r=false&t=markdown HTTP/1.1" 200 OK
ERROR: 2023-02-13 12:13:15,313: src.routers.api | 'utf-8' codec can't decode byte 0x95 in position 1691: invalid start byte
INFO: 127.0.0.1:50366 - "GET /api/update?t=markdown HTTP/1.1" 500 Internal Server Error
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [19336]

and when i try to run again
khoj --no-gui -vv
instead i get

also, would terminal text or screenshots be preferred?

[debanjum]

1. Given that the manual numpy upgrade succeeded, you can avoid the pipx setup.
   Though for reference, the fix for "'pipx' is not recognized" should be 1. running pipx ensurepath in the directory where pipx was installed, which should be C:\Users\<your-username>\AppData\Roaming\Python\Python3x\Scripts and 2. run pipx command from new cmd terminal
2. Based on the logs you've shared, I'd suggest re-running my previous instruction with some updates:
   1. Delete the .khoj folder in your home folder again (on windows at C:\Users\<your-windows-username>\.khoj)
   2. Restart the khoj backend with khoj --no-gui -vv
   3. Re-install/re-enable your Khoj obsidian plugin
   4. Wait for the khoj backend to successfully index your files
      A successful indexing is seen when the backend logs
      1. Do not have this error show up ERROR: 2023-02-13 12:13:15,313: src.routers.api | 'utf-8' codec can't decode byte 0x95 in position 1691: invalid start byte
      2. And you see something INFO: 127.0.0.1:50366 - "GET /api/update?t=markdown HTTP/1.1" 200 OK
   5. Now try search from Obsidian

It's strange that you're getting the same UnicodeDecodeError in similar portion of the code (i.e jsonl.py) that should have been resolved in 11517ba.
Current hypothesis for cause seems to be the jsonl.gz index files in ~/.khoj were written using the old jsonl.py:compress_jsonl_data() method without utf-8 encoding. This is causing the read as utf-8 to fail in the jsonl.py:load_jsonl() method.

Please do share logs of khoj backend from step 2. onwards

the backend issue was solved and the search bar could be used; however it could not find anything.

The search didn't return any results as initially the indexing of your notes wasn't complete and after the backend failed to successfully index your markdown files because of the UnicodeDecodeError

also, would terminal text or screenshots be preferred?

Screenshots mildly preferred (for readability) but fine with whatever is easier for you