BM25 index creation blocked by corporate proxy

[colexbruhn]

Hey there! Super excited to get up an running with Parade :)
I'm going through the example docs, and I'm running into an issue where it looks like our corporate proxy is blocking some sort of external request during the paradedb.create_bm25 call and causing things to break.

Here's the error itself:
SQL Error [XX000]: ERROR: error inserting document during build callback: WriterClientError(ServerError("<corporate proxy html dump here>")) Where: SQL statement "CREATE INDEX search_idx_bm25_index ON public.mock_items USING bm25 ((mock_items.*)) WITH (key_field='id', text_fields='{description: {tokenizer: {type: "en_stem"}}, category: {}}', numeric_fields='{rating: {}}', boolean_fields='{}', json_fields='{}');" PL/pgSQL function create_bm25(text,text,text,text,text,text,text,text) line 42 at EXECUTE

This is running on the paradedb docker image on a RHEL 8 EC2 instance.
Unfortunately, I can't tell from the logs where the request is being made, otherwise I could put in an unblock request. So maybe someone on the team would know?
Are there any packages that don't come pre-bundled ahead of time that I could install in order to get around this?

Still getting up to speed with everything, so please let me know if there's anything that I missed!
Thanks!

[neilyio]

@colexbruhn happy to help. A little bit about the architecture of pg_search. When the extension is initialized, we create a`"writer" background worker which is responsible for all writes/updates/deletes to the index. This is very important for Tantivy, our underlying search engine, which is strict about only one thread/process at a time having write access to the index.

When you connect a client like to Postgres, a corresponding "backend" process is created for the client. If you have 10 clients connected to Postgres, you have 10 "backend" processes... but still only one pg_search "writer" process.

When your client performs a insert/update/delete query, the "backend" process makes an HTTP request to the "writer" process with the data from your query. This is a blocking operation. The "backend" process waits for HTTP request to return successfully, indicating the write to the index was completed in the "writer" process.

To make this possible, the "writer" process creates an HTTP server on a port that is randomly assigned when you initialize Postgres. So to communicate with it, "backend" processes are making HTTP requests to localhost:<random-port>.

Its possible that your proxy is preventing this local network communication between processes.

[colexbruhn]

Hey Neil! Thank you for the super thorough response. That indeed was the case. I got everything working just as I saw your message lolol

I have some code that pulls public images, slaps some corporate boilerplate on it, then pushes it up to our image repo. Basically just had to scrap the proxy bit of the boilerplate, and then everything worked perfectly! I was checking on the proxy and egress settings on my EC2 instance rather than the modified parade image (which I totally forgot was modified).

tl;dr - don't mess with the proxies in the parade image

Thanks for your help!

[philippemnoel]

Sweet! @colexbruhn Let us know if you need anything else. Would love to get on a call to better understand your use case for pg_search and see how we can best support you