Pagination Updated - Items per page option added #1625

Rishi-source · 2024-10-19T07:51:30Z

Add pagination to VCIO & SCIO #1616 #1617 #1618

This change aligns Package Search functionality and Vulnerability Search, improving consistency across the application and enhancing user control over result display.

For Package Search:

For Vulnerability Search:

Signed-off-by: Rishi Garg [email protected]

Rishi-source · 2024-10-19T11:58:22Z

Hi @pombredanne @TG1999 @keshav-space @Hritik14 , Can you please review this PR It has passed all workflow tests and was also working fine on my local server. Thank you

pombredanne · 2024-10-25T10:26:30Z

@TG1999 ping .... @tdruez input welcomed too, as eventually we want to use the same approach across all projects.

vulnerabilities/templates/vulnerabilities.html

vulnerabilities/views.py

vulnerabilities/templates/packages.html

Signed-off-by: Rishi Garg <[email protected]>

Rishi-source · 2024-11-02T06:52:25Z

@tdruez please see the changes now workflow tests are also successful .

Rishi-source · 2024-11-12T02:34:31Z

@tdruez I would request you to please review the PR I have made changes In the code after you requested some changes after first review.

tdruez

See the various comments.
Generally, this implementation seems too complex and not "reusable" enough at it requires to many moving parts.

tdruez · 2024-11-12T07:13:08Z

vulnerabilities/forms.py

@@ -3,7 +3,7 @@
 # VulnerableCode is a trademark of nexB Inc.
 # SPDX-License-Identifier: Apache-2.0
 # See http://www.apache.org/licenses/LICENSE-2.0 for the license text.
-# See https://github.com/aboutcode-org/vulnerablecode for support or download.


Unwanted changes

tdruez · 2024-11-12T07:13:24Z

vulnerabilities/forms.py

@@ -12,26 +12,87 @@

 from vulnerabilities.models import ApiUser

+from .models import *


Avoid * imports.

tdruez · 2024-11-12T07:14:14Z

vulnerabilities/forms.py

+        widget=forms.Select(
+            attrs={
+                "class": "select is-small",
+                "onchange": "handlePageSizeChange(this.value)",


Avoid inline JavaScript.

tdruez · 2024-11-12T07:14:33Z

vulnerabilities/forms.py

+    def clean_search(self):
+        return self.cleaned_data.get("search", "")


What is the purpose of this?

tdruez · 2024-11-12T07:15:10Z

vulnerabilities/forms.py

+    def get_queryset(self, query=None):
+        """
+        Get queryset with search/filter/ordering applied.
+        Args:
+            query (str, optional): Direct query for testing
+        """
+        if query is not None:
+            return self._search(query)
+
+        if not self.is_valid():
+            return self.model.objects.none()
+
+        return self._search(self.clean_search())


What's the need for this logic?

tdruez · 2024-11-12T07:15:34Z

vulnerabilities/forms.py

+    def _search(self, query):
+        """Execute package-specific search logic."""
+        return (
+            self.model.objects.search(query)
+            .with_vulnerability_counts()
+            .prefetch_related()
+            .order_by("package_url")
+        )


This is not following the Django forms conventions.

tdruez · 2024-11-12T07:16:04Z

vulnerabilities/forms.py

+    def _search(self, query):
+        """Execute vulnerability-specific search logic."""
+        return self.model.objects.search(query=query).with_package_counts()


Same comment as above.

tdruez · 2024-11-12T07:17:15Z

vulnerabilities/templates/includes/pagination.html

+        <a href="?page={{ page_obj.previous_page_number }}&search={{ search|urlencode }}&page_size={{ page_obj.paginator.per_page }}" 
+           class="pagination-previous">Previous</a>


What happens if other filters are active, those are lost when using the pagination?

tdruez · 2024-11-12T07:20:12Z

vulnerabilities/templates/includes/pagination.html

All the complexity of those templates should likely be handled in Python.

tdruez · 2024-11-12T07:22:49Z

vulnerabilities/views.py

-    model = models.Package
-    template_name = "packages.html"
-    ordering = ["type", "namespace", "name", "version"]
+class BaseSearchView(ListView):


Could we use the django_filters.views.FilterView instead of re-implementing it?

Rishi-source · 2024-11-19T08:44:36Z

See the various comments. Generally, this implementation seems too complex and not "reusable" enough at it requires to many moving parts.

@tdruez what do you mean by "reusable" here. I would code the implementation again to make it more simpler and satisfy the required needs.

tdruez · 2024-11-20T05:59:57Z

@tdruez what do you mean by "reusable" here. I would code the implementation again to make it more simpler and satisfy the required needs.

@pombredanne as eventually we want to use the same approach across all projects.

@Rishi-source Let's say I want to add this pagination on another Django view, could you document the few steps I need to do with this current implementation? This would help to figure out where we can improve the "reusability".

Signed-off-by: Rishi Garg <[email protected]>

Rishi-source · 2024-12-08T06:44:32Z

vulnerabilities/pagination_mixin.py

+        {"value": 100, "label": "100 per page"},
+    ]
+
+    def get_paginate_by(self, queryset=None):


This function handles page size from URL parameters. Sets default to 20 items, max of 100. Validates to ensure positive integers only. Returns default if invalid values provided.

Rishi-source · 2024-12-08T06:46:34Z

vulnerabilities/pagination_mixin.py

+        except (ValueError, TypeError):
+            return self.paginate_by
+
+    def get_context_data(self, **kwargs):


It adds pagination data to template context - current page size, size choices (can be handled by altering PAGE_SIZE_CHOICES) , total count and page range. it also preserves search params across pagination.

Rishi-source · 2024-12-08T06:48:06Z

vulnerabilities/pagination_mixin.py

+
+        return context
+
+    def _get_page_range(self, paginator, current_page, window=2):


it creates page number sequence. Shows all numbers for small sets (<5 pages). For larger sets, shows first/last pages around current page with (...) for gaps.

Rishi-source · 2024-12-08T06:49:46Z

vulnerabilities/views.py

@@ -36,6 +36,8 @@
 from vulnerabilities.utils import get_severity_range
 from vulnerablecode.settings import env

+from .pagination_mixin import PaginatedListViewMixin


Import the Mixin and the PaginatedListViewMixin class.

Rishi-source · 2024-12-08T06:51:19Z

vulnerabilities/views.py

@@ -62,25 +64,21 @@ def get_purl_version_class(purl: models.Package):
    return purl_version_class


-class PackageSearch(ListView):
+class PackageSearch(PaginatedListViewMixin, ListView):


Add the PaginatedListViewMixin in it and this would convert any ListView into paginated class.

Rishi-source · 2024-12-08T06:56:00Z

Hi @tdruez , I have altered code to improve the reusability for that I have made a file pagination_mixin.py which contains PaginatedListViewMixin class . This logic can convert and ListView into a paginated class and can improve the reusability as now you would have to just add this class to you views.

Rishi-source · 2024-12-08T07:08:49Z

vulnerabilities/templates/includes/pagination.html

+    window.location.href = newUrl;
+}
+</script>


This js function manages URL parameters when changing pagination size. It update the page size parameter while preserving search terms, and remove the current page parameter to reset pagination.

is there any risk of XSS there?

tdruez · 2024-12-09T05:03:48Z

@Rishi-source Thanks but looking at multiple comments is hard to follow. Could you also provide a combined documentation of the few steps required to reuse this pagination in another app?

Rishi-source · 2024-12-09T09:40:14Z

Pagination Implementation Documentation

Core Components of Code

Pagination Mixin (in pagination_mixins.py):
Pagination Template (in templates/includes/pagination.html):

Implementation Steps

Copy the Mixin
- Create pagination_mixins.py in your app directory
- Copy the PaginatedListViewMixin class code into it
Create the Template
- Create pagination.html in your templates directory
- Copy the pagination template code into it
Use PaginatedListViewMixin in Your View

from django.views.generic import ListView
from .pagination_mixins import PaginatedListViewMixin # Make sure to import the mixin

class MyListView(PaginatedListViewMixin, ListView): # Please make sure to put "PaginatedListViewMixin" Above all the mixins in case if there are 2 or more mixins are already their as position  of mixin will matter here
    model = MyModel
    template_name = "my_template.html"
    
    def get_queryset(self):
        return super().get_queryset().order_by('id')

Include in Your Pagination Template

{% extends "base.html" %}

{% block content %}

    {# Like This #}
    {% if is_paginated %}
        {% include 'pagination.html' with page_obj=page_obj %}
    {% endif %}

{% endblock %}

Customization Options

Change Page Sizes

class MyListView(PaginatedListViewMixin, ListView):
    PAGE_SIZE_CHOICES = [
        {"value": 10, "label": "10 per page"},
        {"value": 25, "label": "25 per page"},
        {"value": 50, "label": "50 per page"},
    ]
    paginate_by = 20  # Set the Default size

Adjust Window Size

def get_context_data(self, **kwargs):
    # Change window variable to show more/fewer pages around current page
    context = super().get_context_data(**kwargs)
    context['page_range'] = self._get_page_range(
        context['paginator'], 
        context['page_obj'].number,
        window=3  # Show 3 pages on each side i.e. left and right   
    return context

Notes

JavaScript is embedded directly in the template via the onchange handler
All pagination state is managed through URL parameters
All Filter terms are preserved across pagination
The mixin automatically handles invalid page numbers and sizes

Common Issues and Solutions

Missing Page Numbers
- Check that page_range is being passed correctly in context
URL Parameters Not Preserved
- Check template URLs include all necessary parameters
- Verify parameter names match in views and templates
Page Size Not Working
- Verify page_size parameter is being passed correctly
- Check get_paginate_by implementation in mixin
Styling Issues
- Ensure Bulma CSS is included if using default styling
- Or customize template classes to match your CSS framework

Remember that this implementation assumes:

You're using Django's class-based views
You want URL parameter-based pagination
You're handling search through GET parameters

Rishi-source · 2024-12-09T09:42:52Z

@tdruez Here’s the documentation on implementing pagination using this approach. Please review and let me know if any changes are required.

Signed-off-by: Rishi Garg <[email protected]>

Rishi-source · 2024-12-10T14:30:32Z

Hi @tdruez and @pombredanne , I have implemented this pagination code in both VCIO and SCIO and it is working nice.It preserved all the filter and add a option for the user to select the items per page. I have already shared the screenshot for the VCIO pagination view here are the screenshot for SCIO pagination.

@tdruez you just have to follow the steps which I have provided in the Documentation for implementing this pagination.

Rishi-source · 2024-12-25T04:03:41Z

Hi @TG1999 @tdruez , I request you to please review this PR I have completed the code .

pombredanne

Thanks
Here is some feedback!
The PR title should be using imperative style:
Not Pagination Updated - Items per page option added

But rather:
"Improve pagination in web ui"
then add details in the body.

Could you also add tests? PaginatedListViewMixin is not trivial code

Also as @tdruez mentioned in late November, the template should have little to no logic and instead the logic should be in the Python code.

pombredanne · 2025-01-06T14:02:08Z

vulnerabilities/pagination_mixin.py

+        Get the number of items to paginate by from the request.
+        """
+        try:
+            page_size = int(self.request.GET.get("page_size", self.paginate_by))


Is this the standard approach to get a request arg?

pombredanne · 2025-01-06T14:06:31Z

vulnerabilities/pagination_mixin.py

+
+    def get_pagination_context(self, paginator, page_obj):
+        """
+        Generate pagination-related context data, preserving filters.


Suggested change

Generate pagination-related context data, preserving filters.

Return a mapping of pagination-related context data, preserving filters.

pombredanne · 2025-01-06T14:10:15Z

vulnerabilities/pagination_mixin.py

+        query_params = self.request.GET.copy()
+        query_params.pop("page", None)
+
+        base_query_string = query_params.urlencode()


Is this the right way to generate a URL query string? Is there a better way?

pombredanne · 2025-01-06T14:11:02Z

vulnerabilities/templates/includes/pagination.html

+    window.location.href = newUrl;
+}
+</script>


is there any risk of XSS there?

pombredanne · 2025-01-06T14:11:42Z

vulnerabilities/pagination_mixin.py

@@ -0,0 +1,87 @@
+class PaginatedListViewMixin:
+    """
+    A mixin that adds pagination functionality to ListView-based views.


I would call the file pagination.py instead

tdruez requested changes Oct 29, 2024

View reviewed changes

Rishi-source force-pushed the Page-Updated branch 4 times, most recently from d8068aa to c28765a Compare October 29, 2024 18:44

Rishi-source requested a review from tdruez October 29, 2024 18:50

Rishi-source force-pushed the Page-Updated branch 2 times, most recently from e4a1a18 to 8d177c8 Compare November 2, 2024 05:37

Minor Changes to Get Workflow tests corrected

6ca25ee

Signed-off-by: Rishi Garg <[email protected]>

Rishi-source force-pushed the Page-Updated branch from 8d177c8 to 6ca25ee Compare November 2, 2024 05:47

Rishi-source added 3 commits November 8, 2024 01:24

Merge branch 'main' into Page-Updated

c28dd61

Merge branch 'main' into Page-Updated

74ec708

Merge branch 'main' into Page-Updated

397210d

tdruez requested changes Nov 12, 2024

View reviewed changes

Rishi-source and others added 3 commits December 4, 2024 14:21

Merge branch 'aboutcode-org:main' into Page-Updated

29ff24c

Merge branch 'aboutcode-org:main' into Page-Updated

76afc2b

Pagination via Mixin - Resolving Reusability issue

1302bed

Signed-off-by: Rishi Garg <[email protected]>

Rishi-source force-pushed the Page-Updated branch from 170df8c to 1302bed Compare December 8, 2024 06:34

Rishi-source commented Dec 8, 2024

View reviewed changes

Rishi-source requested a review from tdruez December 8, 2024 06:56

Rishi-source commented Dec 8, 2024

View reviewed changes

Merge branch 'main' into Page-Updated

2e33c5b

Rishi-garg03 added 2 commits December 10, 2024 19:35

Resuable Pagination Code - SCIO+VCIO

3b21946

Signed-off-by: Rishi Garg <[email protected]>

Workflow Checks

5752e23

Signed-off-by: Rishi Garg <[email protected]>

Merge branch 'main' into Page-Updated

7b1de51

Rishi-source closed this Dec 26, 2024

Rishi-source reopened this Dec 26, 2024

pombredanne requested changes Jan 6, 2025

View reviewed changes

Merge branch 'main' into Page-Updated

1851616

		@@ -12,26 +12,87 @@

		from vulnerabilities.models import ApiUser

		from .models import *

		def clean_search(self):
		return self.cleaned_data.get("search", "")

		<a href="?page={{ page_obj.previous_page_number }}&search={{ search\|urlencode }}&page_size={{ page_obj.paginator.per_page }}"
		class="pagination-previous">Previous</a>


		return context

		def _get_page_range(self, paginator, current_page, window=2):

	Generate pagination-related context data, preserving filters.
	Return a mapping of pagination-related context data, preserving filters.

Pagination Updated - Items per page option added #1625

Are you sure you want to change the base?

Pagination Updated - Items per page option added #1625

Conversation

Rishi-source commented Oct 19, 2024 • edited Loading

For Package Search:

For Vulnerability Search:

Rishi-source commented Oct 19, 2024

pombredanne commented Oct 25, 2024

Rishi-source commented Nov 2, 2024

Rishi-source commented Nov 12, 2024

tdruez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rishi-source commented Nov 19, 2024

tdruez commented Nov 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rishi-source commented Dec 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tdruez commented Dec 9, 2024

Rishi-source commented Dec 9, 2024 • edited Loading

Pagination Implementation Documentation

Core Components of Code

Implementation Steps

Customization Options

Notes

Common Issues and Solutions

Rishi-source commented Dec 9, 2024

Rishi-source commented Dec 10, 2024

Rishi-source commented Dec 25, 2024

pombredanne left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rishi-source commented Oct 19, 2024 •

edited

Loading

Rishi-source commented Dec 9, 2024 •

edited

Loading