Add support for storing large ipset's in RAM disks

[netravnen]

Does it make sense to support storing large (>63000B) files in RAM disks before processing?

On Mikrotik CHR, CCR 2116, CCR 2216, RB 5009, RB 1100, CCR 2004, RB 3011 with >= 1G of RAM / Memory. It would make sense to allow sotring the fetched ipset files on a RAM disk before processing. This could alleviate the need to truncate the fetched ipset's.

warning: The list is huge and may be truncated: https://iplists.firehol.org/files/botscout_30d.ipset
warning: The list is huge and may be truncated: https://iplists.firehol.org/files/firehol_level2.netset
warning: The list is huge and may be truncated: https://iplists.firehol.org/files/firehol_level3.netset
warning: The list is huge and may be truncated: https://iplists.firehol.org/files/firehol_level4.netset

Currently, I have a ram-disk (1 GB) defined as follows,

/disk/add nfs-export=yes slot=tmp smb-export=yes smb-user=GUEST tmpfs-max-size=1073741824 type=tmpfs

on the following platform,

[admin@routeros] > /system/resource/print without-paging 
— cropped —
               board-name: CCR2116-12G-4S+
— cropped —

[eworm-de]

The limit is not defined by me, but RouterOS: The fetch command does not download more than ~ 64kB to a variable.

Actually it could download the complete list to a file, but that just moves the problem and makes it worse: Reading the content of a file is limited even more.

I have reported this to support (SUP-132297 "fetch data truncated"), and the answer was:

It is an expected limit. We will note your request.

... and ...

If there will be similar requests, we will consider increasing the output limit.

So feel free to send your own request...

[eworm-de]

Well, this is not what you expected or hoped for... Do you want to accept it as correct answer anyway?

[eworm-de]

Well, my comment made me think about it again, and I think I found a way to achieve support for "huge" downloads.

Want to give it a try? You need to switch global-functions and fw-addr-lists to branch staging.

This requires code from RouterOS 7.13, so it will possibly stay in staging for some time. Not yet sure when to bump...

[eworm-de]

Ok, so looks like works in general, but it is quite hard to import really huge lists.

Feel free to post more results and findings.

What still bothers me... Any idea if things worked with 7.13 already? (I think so...)

[netravnen]

Just downgraded a CHR to 7.13, but could not reproduce any download issues. Possibly this always worked, but you did not give it the time to finish?

Correct.

[netravnen]

What still bothers me... Any idea if things worked with 7.13 already? (I think so...)

Will retry 7.13.5 to se what times I get.

Isolating firehol_level4.netset to its own address-list assist in getting times for huge lists. Which I will use as a benchmark for the results going forwardens in this discussion.

[netravnen]

:global PrintDebug true; added this to global-config-overlay to see if I can get more accurate times in the log file.

[eworm-de]

Did another change to reduce the number of regular expression matches... Not sure if the change is noticeable.

Want to update and give it a try?

[eworm-de]

The new code made it through the next branch and into the main branch, and is available for anybody now. The size of a list is no longer limited by the script, but by devices' resources. (Also added a warning on this in documentation.)

[eworm-de]

(I think this is to be marked as correct answer...)

[netravnen]

Needed to change the category from Ideas to Q&A (link) before being able to mark this as the answer.