Fewer events and results returned when using adhoc_search_level = verbose

[john-corcoran]

Describe the bug
Identical non-transforming searches run across fast, smart and verbose modes (modes selected using adhoc_search_level parameter) will return fewer events and results when using verbose mode. This may be expected behaviour due to additional system constraints when using verbose mode, however no warning is raised and I haven't found any documentation that indicates this is a risk (apologies if it exists and I've missed it).

To reproduce

• Example search: "search index=indexname"
• When performed using fast and smart modes (by changing adhoc_search_level), eventCount and resultCount for the job will be the same figure (150K for an example index I'm running it on). Pulling the events and results in separate operations (using results.JSONResultsReader(job.events) and results.JSONResultsReader(job.results)) will return equivalent datasets (for each, data is pulled in 50K paginated batches).
• When performing the same search in verbose mode, the eventCount will be the same figure as above, however the resultCount will be a smaller figure (30K in my example). Pulling the events and results using the JSONResultsReader operations listed above will result in equivalent datasets, however they will both be for the smaller figure - i.e. the eventCount, although listed as the correct figure (150K), is inaccurate and the returned events will also be fewer than they should be (pagination is still attempted in 50K batches for the 150K size eventCount, but only 30K of events are returned in my example).

Expected behavior
That verbose non-transforming searches would have the same event and result counts returned as fast and smart searches.

Splunk:

• Version: Splunk Enterprise (v8.2.4)(Docker version: https://hub.docker.com/r/splunk/splunk/)
• Deployment: Docker container on local system

SDK:

• Version: 1.6.19
• Language Runtime Version: Python 3.10
• OS: Docker Python image (https://hub.docker.com/_/python)

[tdhellmann]

@john-corcoran can I get some additional information from you?

• Which Splunk platform versions are you encountering this issue on?
• Are you encountering this issue on the Splunk Cloud Platform or on Splunk Enterprise?
• What Splunk platform deployment architecture are you using (standalone/single-instance deployment, search head clustering, etc. - I do not think this last bit matters, but I want to be able to communicate it to the Search API team)

[john-corcoran]

Hi @tdhellmann - this is from the Docker version of Splunk Enterprise (v8.2.4) (https://hub.docker.com/r/splunk/splunk/), running locally on a single host (using sample data) with search head / clustering defaults (I'm not sure on the specifics for Splunk in Docker, but can find out if needed). (I've mostly copied notes from my previous message, please let me know if there's more detail you needed that I've missed.)