[ldap] LDAP query fails to fetch user/group data with lots of results

[nicpenning]

Using the 0.0.1 Entity Analytics Active Directory integration on 8.15.0, I have not been able to ingest user/group data.

In the event logs I see the following error log:

message : Error running full sync
log.origin.file.name : activedirectory/activedirectory.go
log.origin.function : github.com/elastic/beats/v7/x-pack/filebeat/input/entityanalytics/provider/activedirectory.(*adInput).Run
error.message : failed to get group details: LDAP Result Code 4 "Size Limit Exceeded": 
failed to get user details: LDAP Result Code 4 "Size Limit Exceeded": 

For confirmed bugs, please report:

• Version: 8.15.0
• Operating System: Windows Server 2019
• Steps to Reproduce: Have an active directory environment where you have more than 15K+ users and 1k+ groups.

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[nicpenning]

With the paging, I still get an OOM error and it crashes with no results. I will post more details later. I tried paging at 256 and 1000 with the same behavior. Stay tuned.

[nicpenning]

Still did not succeed when running on Windows.

However, using the paging in 0.2.0 on a Linux OS, I successfully pulled in data. Trouble now is it seems to be doing a full sync every 15 minutes. I suppose that is a different issue, though.

So currently Linux works (to a degree) but Windows does not. Doing some offline conversations, this may be due to how command prompt handles the results but I will that to @efd6 to convey.

[nicpenning]

Excited to test this as it becomes available.