Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(search): Adds logic to download search results #4893

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

ERosendo
Copy link
Contributor

@ERosendo ERosendo commented Jan 6, 2025

This PR implements the backend logic for exporting search results (#599).

Key changes:

  • Introduces a new rate limiter to throttle CSV export requests to 5 per day

  • Adds a new setting named MAX_SEARCH_RESULTS_EXPORTED (default: 250) to control the maximum number of rows included in the generated CSV file.

  • Refactors the view.py file within the search module. Helper functions related to fetching Elasticsearch results have been moved to the search_utils.py file for better organization and clarity.

  • Introduces two new helper functions fetch_es_results_for_csv and get_headers_for_search_export

  • Adds a new task that takes the user_id and the query string as input. It then sends an email with a CSV file containing at most MAX_SEARCH_RESULTS_EXPORTED rows.

@freelawproject freelawproject deleted a comment from semgrep-app bot Jan 6, 2025
@ERosendo ERosendo force-pushed the 599-feat-add-logic-to-download-results branch 2 times, most recently from 535f37a to e8f6fd3 Compare January 13, 2025 18:31
@ERosendo ERosendo force-pushed the 599-feat-add-logic-to-download-results branch from b3ba8d5 to 8721c41 Compare January 16, 2025 02:32
Copy link

semgrep-app bot commented Jan 16, 2025

Semgrep found 3 avoid-pickle findings:

Avoid using pickle, which is known to lead to code execution vulnerabilities. When unpickling, the serialized data could be manipulated to run arbitrary code. Instead, consider serializing the relevant data as JSON or a similar text-based serialization format.

Ignore this finding from avoid-pickle

Semgrep found 1 direct-use-of-jinja2 finding:

Detected direct use of jinja2. If not done properly, this may bypass HTML escaping which opens up the application to cross-site scripting (XSS) vulnerabilities. Prefer using the Flask method 'render_template()' and templates with a '.html' extension in order to prevent XSS.

Ignore this finding from direct-use-of-jinja2

@ERosendo ERosendo force-pushed the 599-feat-add-logic-to-download-results branch from 8721c41 to ae29bba Compare January 16, 2025 04:07
This commit refactors the search module by moving helper functions from `view.py` to `search_utils.py`. This improves code organization and makes these helper functions reusable across different modules.
@ERosendo ERosendo force-pushed the 599-feat-add-logic-to-download-results branch from ae29bba to 92cddf5 Compare January 16, 2025 04:19
@ERosendo ERosendo marked this pull request as ready for review January 16, 2025 05:59
@ERosendo ERosendo requested a review from mlissner January 16, 2025 05:59
Copy link
Member

@mlissner mlissner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I gave this a once-over and it feels about right. Concerns I'll highlight for you guys to consider:

  1. Memory: We're putting the CSV in memory, which sure is handy. I think this is fine b/c it'll be pretty small, a couple hundred KB, right? This must be fine, but it's on my mind.

  2. The fields in the result might be annoying with columns that aren't normalized to human values (like SOURCE: CR or something, and local_path: /recap/gov.xxxx.pdf instead of https://storage.courtlistener.com/recap/gov.xxx.pdf). I didn't see code to fix that, but it's probably something we should do if we can. This CSV is supposed to be for humans, in theory.

I appreciate the refactor, but I'd suggest it in a separate PR in the future, so it's not mixed in.

But this looks about right to me otherwise. :)

@mlissner mlissner assigned albertisfu and unassigned mlissner Jan 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: To Do
Development

Successfully merging this pull request may close these issues.

3 participants