2.4.0 Test Plan

2.4.0 QA Checklist

For both upgrades and fresh installs, here is a list of functionality that requires testing. You can use this for copy/pasting into your QA report.

If you have submitted a QA report already for a 2.4.0 release candidate with successful basic server testing and application acceptance testing sections, then you can skip these sections in subsequent reports, unless otherwise indicated by the Release Manager. This is to ensure that you focus your QA effort on the release-specific changes as well as changes since the previous release candidate.

Environment

• Install target:
• Tails version:
• Test Scenario:
• SSH over Tor:
• Release candidate:
• General notes:

Basic Server Testing

• After installing the testinfra dependencies, all tests in ./securedrop-admin verify are passing:
  • Install dependencies on Admin Workstation with cd ~/Persistent/securedrop && ./securedrop-admin setup -t
  • Run tests with ./securedrop-admin verify (this will take a while)
  • Remove test dependencies: rm -rf admin/.venv3/ && ./securedrop-admin setup
• After performing Application Acceptance Testing, ./securedrop-admin verify tests still pass.
• QA Matrix checks pass

Command Line User Generation

• Can successfully add admin user and login

Administration

• I have backed up and successfully restored the app server following the backup documentation
• If doing upgrade testing, make a backup on 2.3.2 and restore this backup on this release candidate
• "Send Test OSSEC Alert" button in the journalist triggers an OSSEC alert and an email is sent
• Can successfully add journalist account with HOTP authentication

Application Acceptance Testing

Source Interface

Landing page base cases

• JS warning bar does not appear when using Security Slider high
• JS warning bar does appear when using Security Slider Low

First submission base cases

• On generate page, refreshing page produces a new 7-word codename
• On submit page, empty submissions produce flashed message
• On submit page, short message submitted successfully
• On submit page, file greater than 500 MB produces "The connection was reset" in Tor Browser quickly before the entire file is uploaded
• On submit page, file less than 500 MB submitted successfully

Returning source base cases

• Nonexistent codename cannot log in
• Empty codename cannot log in
• Legitimate codename can log in
• Returning user can view journalist replies - need to log into journalist interface to test

Journalist Interface

Login base cases

• Can log in with 2FA tokens
• incorrect password cannot log in
• invalid 2fa token cannot log in
• 2fa immediate reuse cannot log in
• Journalist account with HOTP can log in

Index base cases

• Filter by codename works
• Starring and unstarring works
• Click select all selects all submissions
• Selecting all and clicking "Download" works

Individual source page

• You can submit a reply and a flashed message and new row appears
• You cannot submit an empty reply
• Clicking "Delete Source Account" and the source and docs are deleted
• You can click on a document and successfully decrypt using application private key

Basic Tails Testing

After updating to this release candidate and running securedrop-admin tailsconfig

• The Updater GUI appears on boot
• Updating to 2.3.2 is successful

2.4.0 release-specific changes

Web application

• #6350 Make sources.filesystem_id non-nullable

  • Before upgrading to 2.4.0, create a source and log in with it
  • After upgrading to 2.4.0, verify that logging in with the source from the step above is still possible

• #6315 Source Interface Redesign Omnibus, #6419 Viewport shouldn't always show a scrollbar & #6429 Update tor2web/onion site proxy warnings for SI redesign

  • Design
    • Using the latest Tor Browser, nothing shows up distorted, overlaps when it obviously should not, or jumps unexpectedly (on hover, focus, or click)
      • Expected jump: If the codename is long enough to be broken up in two lines, the widget will expand to accommodate two lines instead of one
      • Linux
      • Windows
      • Mac
    • Basic liquid layout works
      • Resize browser window for a viewport size smaller than 800px (Tor Browser changes viewport in 100px increments):
        • Index page: "First submission" and "Return visit" on index page sit on top of each other instead of next to each other
        • Everywhere: Logo/locale sits on top of the main content instead of being in a column to the left (right in rtl languages)
        • "Skip to main content" and "Skip to notification" links (that only appear when tabbing through the page) show up above the logo without any content jumping (in English)
    • (Optional, if you encounter a line-break) The codename widget "weeble" icon is always vertically centered, even when the codename is long enough for a line break
    • Visit the /tor2web-warning page
      • See icons for both the flash message and New Identity
      • See a text-only link to the Tor Browser download page
    • There are no scrollbars visible on pages that are too short to trigger a scrollbar
  • UX
    • Button order for regular actions is: [ Cancel ] [ Action ]
    • Button order for destructive actions (replies): [ Destructive Action ] [ Cancel ]
    • When the locale menu is visible, it shows creates a margin under the menu making the shadow of the menu visible even in pages that would otherwise be too short
  • Accessibility
    • Use the entire workflow of the SI by keyboard only
      • Links can be clicked and forms submitted by pressing return
      • [ ] Show Codename widget checkbox can be interacted with by pressing the spacebar
      • "Skip to main content" link appears on all pages (except the index page)
      • "Skip to notification" link appears on pages which show a notification (except the index page)
      • The logo does not jump when the "Skip to …" links appear
    • Use the entire workflow of the SI via screen-reader
      • alt text of images (f.e. the shield icon for the Tor Browser security settings) is read out loud
      • On all pages except the index page, a "Skip to main content" link can be heard at the beginning of the page
      • When a flashed message is present, a "Skip to notification" link can be heard at the beginning of the page
      • There are no unexpected elements without descriptions read
      • The codename is read when the "Show Codename" checkbox is checked
      • The codename widget is not read when the "Show Codename" checkbox is not checked
      • Using the SI with a screen-reader allows interaction with SecureDrop as expected
    • Items in (tab rather than mouse-click) focus show a 3 pixel big cyan border, text inputs a light blue border
    • Buttons "grow" a border on hover
  • I18n
    • Right-to-left languages (currently only Arabic)
      • Flips the column order
      • Codename stays left-to-right because it's currently English-only
      • Shadow of codename widget "leans" to the left
      • Icons (flash messages, weeble in codename widget and login form) are shown on the right
      • Checkbox in the codename widget is shown on the right side of the text
      • Date/time and trash cans for deletion of replies are shown on the left
    • Arabic specifics
      • Arabic does not have any letter spacing applied
      • Codename (which uses an English word list) but has letter spacing applied

• #6245 Updates for flask and jinja deprecations

  • From the journalist UI, send a message to a source with newlines and HTML tags. View the message in the source UI and see that the newlines are converted into <br> and the HTML is escaped properly.
  • Download multiple files from the journalist UI so it gets turned into a zip file and verify the filename on the zip is as expected, like {source_name}--{timestamp}.zip.

• #6368 Run string formatting after gettext() call

  • In JI, enable minimum length
  • In SI, change your language to German
  • Submit a message that would be deemed too short as per the policy
    • Verify no {} appears in the resulting error message, instead the correct character count is shown

• #6398 Use gpg with --trust-model direct

  • As the www-data user on the application server, run while true; do grep "g[p]g2"<<<$(ps axwww); sleep 1; done. While this script is running:
    • Add a new user in the SI and confirm that gpg2 commands complete quickly (~3sec max) and include the --trust-model direct option
    • Submit a message and confirm that gpg2 commands complete quickly and include the --trust-model direct option
    • Log into the JI and reply to the source, confirming that gpg2 commands complete quickly and include the --trust-model direct option
    • Log into the SI as the original source and confirm that the reply is visible, and gpg2 commands complete quickly with the option as per above
    • In the JI, download and decrypt the source's message, confirming that it can be successfully decrypted.
    • In the JI, delete the source account and confirm that deletion is successful, and that gpg2 commands complete quickly and include the --trust-model direct option

• #6406 feat: validate SDConfig.SUPPORTED_LANGUAGES for usable locales

  • During installation, configure support 3 languages, for example de_DE, it_IT and en_US
  • Configure the default locale to be de_DE (override DEFAULT_LOCALE in /var/www/securedrop/config.py)
    • Confirm that the language menu works normally
    • Configured language loses support
      • Log into the app server and run sudo rm -rf /var/www/securedrop/translations/it_IT and sudo systemctl restart apache2
        • Receive OSSEC alert ERROR in i18n: Configured locales {Locale('it', territory='IT')} are not in the set of usable locales …
    • Default language loses support
      • Log into the app server and run sudo rm -rf /var/www/securedrop/translations/de_DE and sudo systemctl restart apache2
        • Receive OSSEC alert ERROR in i18n: Configured locales {Locale('de', territory='DE')} are not in the set of usable locales …
    • No usable fallback locale
      • Log into the app server, remove en_US from SUPPORTED_LOCALES in /var/www/securedrop/config.py, and run sudo systemctl restart apache2
        • SecureDrop instance is inaccessible
        • Receive OSSEC alert ending in ValueError: None of the default locales {Locale('en', territory='US'), Locale('de', territory='DE')} is in the set of usable locales set()

Operations

• #6401 Disable fwupd-refresh.timer, triggers OSSEC warnings

  • Login to app and mon servers
    • Run sudo systemctl is-enabled fwupd-refresh.timer and see it print "disabled"
    • Run sudo systemctl list-timers which does not list not list fwupd-refresh.timer

• #6448 Updated release key expiry

  • On app and mon servers:
    • run apt-cache policy securedrop-keyring and verify that v0.1.6 is installed
    • run sudo apt-key list and verify that the key with fingerprint 2359E6538C0613E652955E6C188EDD3B7B22E6A3 is installed and has an expiry date of 2023-07-04, including subkeys.
    • run sudo apt-get update and confirm that it completes without error

Preflight testing

Basic testing

• Install or upgrade occurs without error
• Source interface is available and version string indicates it is 2.4.0
• A message can be successfully submitted

Tails

• The updater GUI appears on boot
• The update successfully occurs to 2.4.0
• After reboot, updater GUI no longer appears