arXiv submission UI v0.1-alpha

Release notes - arXiv-NG - Version submission-0.1-alpha

Sub-task

• [ARXIVNG-267] - Implement flask API stubs
• [ARXIVNG-398] - Implement low level file type guess and TeX analysis code. Originates from TypeGuess.pm in legacy Perl system.
• [ARXIVNG-399] - Implement file class that provides interface to various details about uploaded files.

Bug

• [ARXIVNG-2007] - TeX log summary disappears if you navigate away from a successful compile and then return
• [ARXIVNG-2034] - Populate help bubbles on metadata pages

New Feature

• [ARXIVNG-215] - Investigate accessibility in TeX-based PDFs
• [ARXIVNG-245] - Upload check: remove *.synctex files from submission source packages
• [ARXIVNG-405] - Implement API routes for upload
• [ARXIVNG-479] - Upload check: Attempt to Fix/Repair illegal Filename (illegal characters)
• [ARXIVNG-480] - Upload check: Identify files that are not allowed in submission and remove them.
• [ARXIVNG-481] - Upload check: Identify file types that require additional processing.
• [ARXIVNG-482] - Upload check: Fix top level directory
• [ARXIVNG-483] - Upload check: Make sure permissions are set to reasonable values
• [ARXIVNG-484] - Upload check: unmacify - Edit source files to fix up carriage returns/newlines
• [ARXIVNG-488] - Upload check: Detect case where submitter fails to include the .bbl for references.
• [ARXIVNG-490] - Implement oversize check (per file/aggregate)
• [ARXIVNG-491] - Upload check: Strip non-compliant embedded TIFF bitmaps from Postscript file
• [ARXIVNG-838] - Upload: Add logging for file upload processing.
• [ARXIVNG-852] - Upload check: Check for extraneous directories and remove them.
• [ARXIVNG-872] - User interface to verify submitter information
• [ARXIVNG-873] - Interface for submitter to confirm authorship/proxy status
• [ARXIVNG-874] - Interface for submitter to select license
• [ARXIVNG-875] - Interface for submitter to acknowledge/accept policy statement
• [ARXIVNG-876] - Interface for submitter to select primary classification
• [ARXIVNG-877] - Interface for submitter to select secondary classification
• [ARXIVNG-878] - Interface for submitter to add metadata to submission
• [ARXIVNG-904] - Submission events must support "deletion"
• [ARXIVNG-1017] - Change submission "view" button link to give nice file name with extension (.pdf etc.)
• [ARXIVNG-1041] - Macro/template code for UI progress bar
• [ARXIVNG-1061] - Style progress bar
• [ARXIVNG-1072] - Ability to download content (source files) from the file management service
• [ARXIVNG-1073] - Ability to delete individual files in file management service
• [ARXIVNG-1074] - Ability to delete all files in file management service
• [ARXIVNG-1075] - Ability to delete released workspace in file management service
• [ARXIVNG-1077] - Ability to lock an upload workspace, so that replacements are rejected
• [ARXIVNG-1078] - Ability to unlock a workspace in file management service, so that replacements can be uploaded
• [ARXIVNG-1079] - Ability to release/unrelease upload workspaces, so that files no longer needed can be deleted
• [ARXIVNG-1105] - UI for process-files/compilation error messaging and various pages
• [ARXIVNG-1130] - File management service should return workspace size, last upload size
• [ARXIVNG-1132] - Help page for upload error conditions
• [ARXIVNG-1135] - When additional files are added to an upload workspace, they should be highlighted (success) in the upload list
• [ARXIVNG-1142] - Work out issues with handling of + signs in file names
• [ARXIVNG-1318] - Converter: Document arxiv-converter TeX compilation Docker image creation and use.
• [ARXIVNG-1340] - Submission core must support replacement
• [ARXIVNG-1341] - Paper owner may submit a withdrawal request via UI
• [ARXIVNG-1342] - Paper owner may submit a replacement submission via UI
• [ARXIVNG-1343] - Paper owner may add JREF/DOI to an already published paper via the UI
• [ARXIVNG-1360] - User should be able to delete their submission entirely via the UI
• [ARXIVNG-1364] - The submitter should get feedback early in the submission process about their endorsement status
• [ARXIVNG-1366] - Integration with plain text extraction service in arxiv.submit
• [ARXIVNG-1367] - Integration with classifier service in arxiv.submit
• [ARXIVNG-1369] - User can see a preview of their submission abstract just like it will appear on the abstract page
• [ARXIVNG-1370] - User can preview the PDF version of their submission prior to finalization
• [ARXIVNG-1371] - The submitter should be notified during the submission process that their TeX source will be public
• [ARXIVNG-1372] - It should be clear to the submitter what the difference between withdrawal and replacement is. When a submitter initiates a withdrawal, they should be prompted to consider replacement before proceeding
• [ARXIVNG-1375] - Generate and send confirmation email upon successful submission
• [ARXIVNG-1376] - Integrate with classic email system
• [ARXIVNG-1408] - Paper owner may submit a cross-list request via UI
• [ARXIVNG-1409] - Crosslist template: layout, presentation, copywriting
• [ARXIVNG-1412] - File management service should expose upload type (pdf only, tex, html, etc) via API
• [ARXIVNG-1415] - Upload check: delete missfont.log file and generate informative message.
• [ARXIVNG-1416] - Upload check: Generate warning for "*.tex_", "*.tex.bak", "*.tex~" files.
• [ARXIVNG-1417] - Converter: Investigate and fix stamp PDF process in converter image.
• [ARXIVNG-1419] - Displaying TeX Errors: Determine what is possible in submission-0.1
• [ARXIVNG-1423] - Endorsements: send message to UI if unendorsed
• [ARXIVNG-1642] - Compiler service must support compilation of PS to PDF
• [ARXIVNG-1783] - Migrations and versioning for event classes
• [ARXIVNG-1804] - Submitters should be able to abandon/cancel jref, cross-list, and withdrawal requests
• [ARXIVNG-1987] - Upload check: Repair Encapsulated Postscript file: Strip out TIFF bitmap
• [ARXIVNG-1988] - Upload check: Repair Postscript file.
• [ARXIVNG-1994] - AutoTeX Log: highlight lexicon terms in autotex.log
• [ARXIVNG-2006] - Submission confirmation e-mail should include a link to preview the submission

Task

• [ARXIVNG-22] - Implement Docker image for plain text extraction
• [ARXIVNG-23] - Implement text extraction agent + API
• [ARXIVNG-54] - Request cert and CNAME for fulltext.arxiv.org
• [ARXIVNG-81] - Dockerize fulltext API
• [ARXIVNG-83] - Redeploy refactored fulltext
• [ARXIVNG-105] - Revise submission integration WP based on meeting notes
• [ARXIVNG-112] - Prototype JSON Schema for submission API, incorporating suggested improvements to SWORD
• [ARXIVNG-113] - Draft submission API workflow(s) based on SWORD + suggested improvements, and ORCID/OAuth2 scenario
• [ARXIVNG-155] - Cleanup TeX and Ghostscript containers and migrate to NG repository and workflow.
• [ARXIVNG-161] - Redeploy reference and fulltext extraction in k8s
• [ARXIVNG-201] - Rebuild arxiv-fulltext images on arxiv base Docker image
• [ARXIVNG-211] - Update submit API docs based on current implementation
• [ARXIVNG-225] - Submission event controller should be a stand-alone service with internal API
• [ARXIVNG-274] - Update arXiv submission API docs based on file management and compilation design
• [ARXIVNG-404] - Upload check: Add test to verify that nested archives are being unpacked properly.
• [ARXIVNG-413] - Docker compose config should start a working minimal submission API
• [ARXIVNG-456] - Add instructions for contributors to repo
• [ARXIVNG-457] - Generate a wide range of examples of submissions
• [ARXIVNG-458] - Implement validation in core events package for metadata update commands
• [ARXIVNG-459] - Implement validation on classification events using taxonomy in arXiv base
• [ARXIVNG-461] - Generate examples of moderation scenarios, and expose appropriate status to client
• [ARXIVNG-478] - Upload check: Check for ancillary files and make sure they are identified properly
• [ARXIVNG-485] - Upload check: Check Postscript file (edits postscript)
• [ARXIVNG-571] - Upload check: Add pdfpages example test files
• [ARXIVNG-840] - Design progress bar with indicators for all steps
• [ARXIVNG-841] - Work out a submission UI user testing plan with Gail
• [ARXIVNG-842] - Finish UI prototype
• [ARXIVNG-859] - Build initial UI shim templates
• [ARXIVNG-879] - Verify with admin team requirements for manipulating submission content
• [ARXIVNG-880] - Finish preparation for first user test of prototype
• [ARXIVNG-883] - Set up README, documentation in submission-ui project
• [ARXIVNG-892] - Add endorsement validation to SetPrimaryClassification event
• [ARXIVNG-897] - Add InvalidStack to error handling
• [ARXIVNG-899] - Add shim of each controller to accelerate progress on ARXIVNG-[875-880]
• [ARXIVNG-909] - Remove primary classification from secondary classification selection options
• [ARXIVNG-910] - Dynamic population of subjects
• [ARXIVNG-911] - Migrate GET requests to POST on all forms
• [ARXIVNG-912] - Pass submission_id to progress bar links
• [ARXIVNG-914] - Create naming guide for core events
• [ARXIVNG-959] - Documentation of test cases for file upload
• [ARXIVNG-985] - Change labels on items with confusing terms
• [ARXIVNG-986] - Clarify design for trash can icon
• [ARXIVNG-987] - Label box with Selected Categories for cross-list page and category page
• [ARXIVNG-989] - Template for delete file(s) confirmation page
• [ARXIVNG-990] - Design and create template for file upload page
• [ARXIVNG-991] - Update the OpenAPI spec based on submission discussion
• [ARXIVNG-992] - Implement file management service integration against OpenAPI spec
• [ARXIVNG-993] - Implement controllers for upload
• [ARXIVNG-994] - Update error message response format
• [ARXIVNG-1042] - Upload: Revise upload request to OpenAPI specification.
• [ARXIVNG-1063] - Upgrade fulltext extraction agent to use arxiv.base.agent
• [ARXIVNG-1064] - Remove modules/functionality from fulltext project that are redundant to arXiv base
• [ARXIVNG-1065] - Update fulltext service docs based on design discussions
• [ARXIVNG-1066] - Refactor fulltext store component to use s3
• [ARXIVNG-1069] - Fulltext extraction worker does not discriminate between PDF and HTML (pending render) responses
• [ARXIVNG-1076] - Bring file management service up to spec with current version of arxiv.users package
• [ARXIVNG-1081] - Decide on a format and wording of file upload error messages
• [ARXIVNG-1082] - Implement package upload and error message reporting in submission UI
• [ARXIVNG-1098] - Upload status does not reflect deleted files until new file is uploaded
• [ARXIVNG-1099] - Add modified date to individual file status
• [ARXIVNG-1102] - Submission system must integrate with classic filesystem: implement function for putting source, preview PDF in the correct location
• [ARXIVNG-1103] - Write OpenAPI schema for compilation service
• [ARXIVNG-1104] - Clean up UI presentation issues
• [ARXIVNG-1106] - Implement a service module that retrieves a source package from the file management system
• [ARXIVNG-1107] - Implement a task function that performs compilation with a Docker image
• [ARXIVNG-1108] - Implement a storage service integration for compiled PDFs
• [ARXIVNG-1109] - Implement a route + controller to handle requests for compilation
• [ARXIVNG-1131] - Modified date on individual files does not reflect upload time
• [ARXIVNG-1134] - When a file is renamed on an upload, the old name and path are returned
• [ARXIVNG-1136] - Tarball with relative paths can escape upload workspace
• [ARXIVNG-1150] - Implement API methods to download upload workspace log, top-level upload logs
• [ARXIVNG-1151] - Set up Flask app for compilation
• [ARXIVNG-1152] - Upload TeX tree image to private AWS Docker repository
• [ARXIVNG-1153] - Set up private docker repository for TeX tree image
• [ARXIVNG-1175] - Implement /anc creation and storage for Ancillary checkbox
• [ARXIVNG-1183] - Investigate effort required (and feasibility for v0.1) to have category descriptions after selection
• [ARXIVNG-1189] - Implement integration for compilation service
• [ARXIVNG-1190] - Implement logic to evaluate step in submission, and enforce rule that users cannot skip ahead
• [ARXIVNG-1191] - Add CSRF protection to all submission pages
• [ARXIVNG-1195] - Implement FM API support for downloading individual content files.
• [ARXIVNG-1215] - Make sure links to all external services have a url_for
• [ARXIVNG-1216] - Documentation for arxiv-compiler
• [ARXIVNG-1231] - arxiv.auth integration for fulltext service
• [ARXIVNG-1235] - Uploading a directory NOT named anc and marking it Ancillary places directory into the main file list
• [ARXIVNG-1248] - Decide on file size and submission size limit for NG submissions
• [ARXIVNG-1259] - Implement admin feedback - round 2 of UI cleanup
• [ARXIVNG-1270] - Implement additional help info from JH session
• [ARXIVNG-1300] - File uploads to anc and special characters support
• [ARXIVNG-1301] - Implement new file size/total submission limits
• [ARXIVNG-1303] - Upload: Fix initial state mismatch when top level directory is removed.
• [ARXIVNG-1319] - Upgrade requests package to >=2.20.0 in submission-core repo
• [ARXIVNG-1322] - describe compiler docker container for use in compiler service
• [ARXIVNG-1323] - refactor compilation task function to use "real" compiler image
• [ARXIVNG-1324] - stand up celery worker to test async compilation
• [ARXIVNG-1325] - stand up web app to handle api requests and create tasks for celery
• [ARXIVNG-1326] - test compilation against a whole bunch of existing papers
• [ARXIVNG-1333] - add localstack:s3 integration to docker-compose for compiler service
• [ARXIVNG-1334] - bring test coverage up to standard in compiler service
• [ARXIVNG-1338] - Submission core must support JREF event
• [ARXIVNG-1339] - Submission core must support withdrawal event
• [ARXIVNG-1346] - More investigation of converter Docker image
• [ARXIVNG-1365] - Link to TeX specific help pages in compilation
• [ARXIVNG-1382] - Spacing for file upload list is uneven
• [ARXIVNG-1390] - Converter: Disable 'chroot' in arXiv-converter
• [ARXIVNG-1395] - Jref template: layout, presentation, copywriting
• [ARXIVNG-1396] - Withdrawals template: layout, presentation, and copywriting
• [ARXIVNG-1397] - Replacement template: layout, presentation, copywriting
• [ARXIVNG-1398] - Preview template: layout, presentation, copywriting
• [ARXIVNG-1399] - Confirmation template: copywriting and layout
• [ARXIVNG-1400] - Compilation template: check and refine layout and copy
• [ARXIVNG-1401] - Crossref template: layout, presentation, copywriting
• [ARXIVNG-1410] - Replacement workflow should add the ID and title of the paper that is being replaced on every screen
• [ARXIVNG-1413] - Implement post-submission classification check
• [ARXIVNG-1414] - Implement post-submission title check
• [ARXIVNG-1422] - Implement admin log integration in events
• [ARXIVNG-1425] - File upload messaging inconsistent for the case of success-with-warnings
• [ARXIVNG-1449] - Upload check: warning messages not getting returned properly
• [ARXIVNG-1451] - Endorsements: not applying long list to submission UI
• [ARXIVNG-1452] - Uploading an individual file produces confirmation message "Unpacked 2 files"
• [ARXIVNG-1551] - Write docker-compose for submission UI service group
• [ARXIVNG-1625] - Process Template needs to have errors and warning toggled by variables
• [ARXIVNG-1629] - Document event binding, async mechanisms in submission core
• [ARXIVNG-1643] - Develop a lexicon of words that should get highlighted in TeX logs: warnings, errors
• [ARXIVNG-1673] - Design HTML template for submission e-mails
• [ARXIVNG-1693] - create_database in submission-core does not work with mysql/mariadb backend
• [ARXIVNG-1773] - Metadata page produces 500 error
• [ARXIVNG-1775] - At start of replacement, process step is already highlighted
• [ARXIVNG-1776] - Communicate DB failure and other process interrupts to the UI
• [ARXIVNG-1781] - AutoTex log: style viewport in submit.css
• [ARXIVNG-1788] - Uploading PDF file via submission UI results in 500 error. Submission UI becomes non functional.
• [ARXIVNG-1789] - Submission UI: Process Pages: "View raw log" appears immediately after pressing process files button.
• [ARXIVNG-1790] - Submission UI: Process Pages: Previous list of compilations does not always display state.
• [ARXIVNG-1791] - Submission UI: Process Pages: autotex log only displayed in TeXLive Compiler Summary when compilation is successful.
• [ARXIVNG-1973] - Design changes to submission start page
• [ARXIVNG-1974] - Use a small cookie to keep track of which steps the user has completed (esp for optional steps with no data)
• [ARXIVNG-1975] - Width of main container varies from step to step
• [ARXIVNG-2012] - Fix watermark in preview PDF
• [ARXIVNG-2013] - Minor refactor of progress bar status
• [ARXIVNG-2015] - Write alpha testing survey
• [ARXIVNG-2016] - Finalize submission release plan
• [ARXIVNG-2020] - Stage submission services in development namespace
• [ARXIVNG-2022] - Ensure that authenticated sessions have sufficient scope for submission system
• [ARXIVNG-2052] - Converter: make fonts routine is failing
• [ARXIVNG-2144] - Verify that upstream services are available upon startup
• [ARXIVNG-2145] - Integrate Vault with submission agent

Story

• [ARXIVNG-55] - Fulltext API should support text/plain content type
• [ARXIVNG-263] - Implement File Management Service
• [ARXIVNG-275] - Submission event controller supports all existing submission UI actions/commands
• [ARXIVNG-276] - Submission event controller updates classic DB as expected
• [ARXIVNG-277] - Fulltext service should generate and serve PSV text as a separate resource
• [ARXIVNG-684] - Submitters should be able to include a display name for each author
• [ARXIVNG-1786] - It should be clear to the submitter that their submissions cannot be anonymous