Potential problem with large (> 2GB) files on Linux.

[alexiv1965]

Proposal:

Almost all read actions in searchd go through sphReadThrottled() (with write actions - all the same). Let's note that Linux read() function (called later) cannot read one piece that is larger 2GB.

Here is selection of "step size" for single read action. The rt_merge_maxiosize config option if set - will limit the size of every read/write actions by its value. But this option is not set by default and very few admins will take it into account. So, most often scenario is zero value for g_iMaxIOSize global variable, and hence - the whole size will be read in one chunk. Thereafter it'll produce very unclear read error.

I suppose to set strict upper limit for g_iMaxIOSize value (not zero) - to make sure that Linux read() can do its job without error - this value should be a little bit less than 2GB - see read() man for details.

By the way: the following call of sphRead() have very rough fix for Windows warning: cast to int of read chunk size. Without described above solution it leads to more unclear read errors: large positive value is casted to negative value and then in call to read() it is casted to very-very large positive value. Suppose this should be also fixed.

Once again: the same option and global variable rules both read and write actions.

Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

• Implementation completed
• Tests developed
• Documentation updated
• Documentation reviewed
• Changelog updated
• OpenAPI YAML updated and issue created to rebuild clients

[sanikolaev]

The related message in TG - https://t.me/manticore_chat/7200/16974

[sanikolaev]

Thereafter it'll produce very unclear read error.

Please provide more details about it. What was the error in your case?

[alexiv1965]

That is error in log: "global IDF unavailable - IGNORING". It concerns only my case: I've tried to set up very large global.idf file for the index, but this file cannot be read. In other situations there'll be other error messages.

[sanikolaev]

Can you share this idf file by uploading it to our write-only S3 storage? https://manual.manticoresearch.com/Reporting_bugs#Uploading-your-data