Broken pagination function of Stash provider

[Porsh33]

go-scm/scm/driver/stash/util.go

Lines 15 to 27 in f73f51e

	func encodeListOptions(opts scm.ListOptions) string {
	params := url.Values{}
	if opts.Page > 1 {
	params.Set("start", strconv.Itoa(
	(opts.Page-1)*opts.Size),
	)
	}
	if opts.Size != 0 {
	params.Set("limit", strconv.Itoa(opts.Size))
	}
	return params.Encode()
	}

Reference to docs https://docs.atlassian.com/bitbucket-server/rest/5.16.0/bitbucket-rest.html

Important: If more than one page exists (i.e. the response contains "isLastPage": false), the response object will also contain a nextPageStart attribute which must be used by the client as the start parameter on the next request. Identifiers of adjacent objects in a page may not be contiguous, so the start of the next page is not necessarily the start of the last page plus the last page's size. A client should always use nextPageStart to avoid unexpected results from a paged API.

[bradrydzewski]

We have not experienced any issues with the below logic, so we would kindly ask that you provide more details. For example, can you narrow it down to a subset of endpoints and can you describe in more detail the error message or problems you are experiencing as a result of the below logic? Also, are you experiencing this problem through your use of Drone, or are you using this library separately?

	if !out.pagination.LastPage.Bool {
		res.Page.First = 1
		res.Page.Next = opts.Page + 1
	}

[Porsh33]

I use the Abstruse CI, which has code like this test: https://github.com/drone/go-scm/blob/master/scm/driver/stash/repo_test.go#L257

// ListRepos returns list of repositories.
func (s SCM) ListRepos(page, size int) ([]*scm.Repository, error) {
	repos, _, err := s.client.Repositories.List(s.ctx, scm.ListOptions{Page: page, Size: size})
	return repos, err
}

But i think this test is incorrect (reference data). Because in real Bitbucket Enterprise the API answer with pagination looks like this:

porsh@i109885904 ~/work/github [15:16] $ curl -sH "Authorization: Bearer <token>" "https://bb.domain.name/rest/api/1.0/repos?limit=1" | jq -r
{
  "size": 1,
  "limit": 1,
  "isLastPage": false,
  "values": [
    {
      "slug": "12341234",
      "id": 5063,
      "name": "12341234",
      "scmId": "git",
      "state": "AVAILABLE",
      "statusMessage": "Available",
      "forkable": true,
      "project": {
        "key": "~RUDSKOY",
        "id": 2809,
        "name": "Vladislav Rudskoy",
        "type": "PERSONAL",
        "owner": {
          "name": "rudskoy",
          "emailAddress": "[email protected]",
          "id": 6612,
          "displayName": "Vladislav Rudskoy",
          "active": true,
          "slug": "rudskoy",
          "type": "NORMAL",
          "links": {
            "self": [
              {
                "href": "https://domain.name/users/rudskoy"
              }
            ]
          }
        },
        "links": {
          "self": [
            {
              "href": "https://bb.domain.name/users/rudskoy"
            }
          ]
        }
      },
      "public": false,
      "links": {
        "clone": [
          {
            "href": "https://bb.domain.name/scm/~rudskoy/12341234.git",
            "name": "http"
          },
          {
            "href": "ssh://[email protected]/~rudskoy/12341234.git",
            "name": "ssh"
          }
        ],
        "self": [
          {
            "href": "https://bb.domain.name/users/rudskoy/repos/12341234/browse"
          }
        ]
      }
    }
  ],
  "start": 0,
  "nextPageStart": 3
}

if we use the result of func encodeListOptions(Page: 2, Size: 1), the request will be

porsh@i109885904 ~/work/github [15:16] $ curl -sH "Authorization: Bearer <token>" "https://bb.domain.name/rest/api/1.0/repos?limit=1&start=1" | jq -r
{
  "size": 1,
  "limit": 1,
  "isLastPage": false,
  "values": [
    {
      "slug": "12341234",
      "id": 5063,
      "name": "12341234",
      "scmId": "git",
      "state": "AVAILABLE",
      "statusMessage": "Available",
      "forkable": true,
      "project": {
        "key": "~RUDSKOY",
        "id": 2809,
        "name": "Vladislav Rudskoy",
        "type": "PERSONAL",
        "owner": {
          "name": "rudskoy",
          "emailAddress": "[email protected]",
          "id": 6612,
          "displayName": "Vladislav Rudskoy",
          "active": true,
          "slug": "rudskoy",
          "type": "NORMAL",
          "links": {
            "self": [
              {
                "href": "https://domain.name/users/rudskoy"
              }
            ]
          }
        },
        "links": {
          "self": [
            {
              "href": "https://bb.domain.name/users/rudskoy"
            }
          ]
        }
      },
      "public": false,
      "links": {
        "clone": [
          {
            "href": "https://bb.domain.name/scm/~rudskoy/12341234.git",
            "name": "http"
          },
          {
            "href": "ssh://[email protected]/~rudskoy/12341234.git",
            "name": "ssh"
          }
        ],
        "self": [
          {
            "href": "https://bb.domain.name/users/rudskoy/repos/12341234/browse"
          }
        ]
      }
    }
  ],
  "start": 1,
  "nextPageStart": 3
}

As you can see, server returns the same data as the first time. This is due to the fact that the value of the "nextPageStart" key is not taken into account when forming the URL. And we can't provide this value "as is" using this function

A new data can be received using request:

porsh@i109885904 ~/work/github [15:31] $ curl -sH "Authorization: Bearer <token>" "https://bb.domain.name/rest/api/1.0/repos?limit=1&start=3" | jq -r
{
  "size": 1,
  "limit": 1,
  "isLastPage": false,
  "values": [
    {
      "slug": "2021-09-homework-01",
      "id": 16135,
      "name": "2021-09-homework-01",
      "scmId": "git",
      "state": "AVAILABLE",
      "statusMessage": "Available",
      "forkable": true,
      "project": {
        "key": "BACKENDSCHOOL",
        "id": 24862,
        "name": "Backend School",
        "description": "https://domain.name/services/backendschool",
        "public": false,
        "type": "NORMAL",
        "links": {
          "self": [
            {
              "href": "https://bb.domain.name/projects/BACKENDSCHOOL"
            }
          ]
        }
      },
      "public": false,
      "links": {
        "clone": [
          {
            "href": "ssh://[email protected]/backendschool/2021-09-homework-01.git",
            "name": "ssh"
          },
          {
            "href": "https://bb.domain.name/scm/backendschool/2021-09-homework-01.git",
            "name": "http"
          }
        ],
        "self": [
          {
            "href": "https://bb.domain.name/projects/BACKENDSCHOOL/repos/2021-09-homework-01/browse"
          }
        ]
      }
    }
  ],
  "start": 3,
  "nextPageStart": 5
}

[bradrydzewski]

if we use the result of func encodeListOptions(Page: 2, Size: 1), the request will be
porsh@i109885904 ~/work/github [15:16] $ curl -sH "Authorization: Bearer " > "https://bb.domain.name/rest/api/1.0/repos?limit=1&start=1" | jq -r

Just a note that in your example the curl statement still has start=1 and not start=2 which could explain why the contents are the same in both requests.

[Porsh33]

if we use the result of func encodeListOptions(Page: 2, Size: 1), the request will be
porsh@i109885904 ~/work/github [15:16] $ curl -sH "Authorization: Bearer " > "https://bb.domain.name/rest/api/1.0/repos?limit=1&start=1" | jq -r

Just a note that in your example the curl statement still has start=1 and not start=2 which could explain why the contents are the same in both requests.

It's ok because strconv.Itoa((opts.Page-1)*opts.Size) is equal (2-1)*1 = 1. But even if I try to get data with start=2 I'll get same result as first one (because the new data providing by server starts from "nextPageStart": 3)

porsh@i109885904 ~/work [15:49] $ curl -sH "Authorization: Bearer <token>" "https://bb.domain.name/rest/api/1.0/repos?limit=1&start=2" | jq -r
{
  "size": 1,
  "limit": 1,
  "isLastPage": false,
  "values": [
    {
      "slug": "12341234",
      "id": 5063,
      "name": "12341234",
      "scmId": "git",
      "state": "AVAILABLE",
      "statusMessage": "Available",
      "forkable": true,
      "project": {
        "key": "~RUDSKOY",
        "id": 2809,
        "name": "Vladislav Rudskoy",
        "type": "PERSONAL",
        "owner": {
          "name": "rudskoy",
          "emailAddress": "[email protected]",
          "id": 6612,
          "displayName": "Vladislav Rudskoy",
          "active": true,
          "slug": "rudskoy",
          "type": "NORMAL",
          "links": {
            "self": [
              {
                "href": "https://domain.name/users/rudskoy"
              }
            ]
          }
        },
        "links": {
          "self": [
            {
              "href": "https://bb.domain.name/users/rudskoy"
            }
          ]
        }
      },
      "public": false,
      "links": {
        "clone": [
          {
            "href": "https://bb.domain.name/scm/~rudskoy/12341234.git",
            "name": "http"
          },
          {
            "href": "ssh://[email protected]/~rudskoy/12341234.git",
            "name": "ssh"
          }
        ],
        "self": [
          {
            "href": "https://bb.domain.name/users/rudskoy/repos/12341234/browse"
          }
        ]
      }
    }
  ],
  "start": 1,
  "nextPageStart": 3
}