Add support for importing `.wpress` files #497

danielbachhuber · 2024-09-02T14:33:47Z

Related #496 and https://github.com/Automattic/dotcom-forge/issues/8380

Proposed Changes

Adds support for importing .wpress files from All-in-One WP Migration and Backup.

A lot of this code was generated by Cursor + Claude 😊

Props to https://github.com/ofhouse/wpress-extract/blob/main/lib/wpress-extract.js for guidance on how .wpress files are structured.

Testing Instructions

Install All-in-One WP Migration and Backup plugin on Atomic or self-hosted site
Change theme, upload media, install plugin, add post and page etc.
Prepare file backup
Import .wpress file in Studio
Confirm that other formats are still imported

Pre-merge Checklist

Have you checked for TypeScript, React or other console errors?

danielbachhuber · 2024-09-02T15:08:17Z

Could I get an initial review on this? Also, what sort of tests would you suggest (particularly given .wpress' custom backup format?

fluiddot

Could I get an initial review on this?

The PR looks great, thanks @danielbachhuber for creating it 🙇 . I added some comments to the code but apart from using async file system operations, the approach looks solid.

However, I encountered an issue when testing it. After importing a wpress file, the active theme points to an empty theme. This makes the site render a blank page.

Also, what sort of tests would you suggest (particularly given .wpress' custom backup format?

You check examples of other tests we made for other formats here. We mock the file reading operations, so I presume the tests for the wpress format would be similar to the ones we have.

Let me know if there's anything I can help further to move the PR forward 🙂.

src/lib/import-export/import/handlers/backup-handler-wpress.ts

fluiddot · 2024-09-05T11:55:04Z

src/lib/import-export/import/importers/importer.ts

+		} );
+
+		rl.on( 'line', ( line: string ) => {
+			writeStream.write( line.replace( /SERVMASK_PREFIX/g, 'wp' ) + '\n' );


I wonder if we could expand on why wpress format needs to modify the SQL code before importing it.

Do you refer that we should add a comment explaining why why we replace this fake prefix?
It seems that wpress replaces the original prefix with SERVMASK_PREFIX, probably to avoid prefix collisions when importing to a shared database.

Do you refer that we should add a comment explaining why why we replace this fake prefix? It seems that wpress replaces the original prefix with SERVMASK_PREFIX, probably to avoid prefix collisions when importing to a shared database.

Yep, I was curious why we need this. Thanks @sejas for explaining it 🙇 !

src/lib/import-export/import/handlers/backup-handler-wpress.ts

sejas

Whoa! It's almost working!
Great work and All-in-One WP Migration and Backup format will be a great addition.

I've experienced the same error, where the theme is not selected. After accessing twice to /wp-admin/themes.php the 2024 theme was selected automatically instead of the correct theme used in the original site.

At first I thought it was some errors with SQLite import, like not supporting "START TRANSACTION;", but after debugging it, I've found that my .wpress file has an empty string for the template and stylesheet options. It seems that the All in one plugin removes those options from the SQL dump on purpose.

src/lib/import-export/import/handlers/backup-handler-wpress.ts

danielbachhuber · 2024-09-23T10:42:20Z

However, I encountered an issue when testing it. After importing a wpress file, the active theme points to an empty theme. This makes the site render a blank page.

As it turns out, the .wpress archive includes a package.json file with some site configuration data:

I guess we need to process that post-import too.

wojtekn · 2024-09-25T07:14:23Z

src/lib/import-export/import/importers/importer.ts

+		await fsPromises.rename( tempOutputPath, tmpPath );
+	}
+
+	protected async parseWpressPackage(): Promise< {


Would it make sense to treat this package file as the metafile so we could implement this as a part of the parseMetaFile() method?

Yeah, I was thinking in something like that.

Currently it's working and we set correctly the theme, but creating a more abstract way to set it can benefit future adaptations.

I didn't mean to create a more abstract way. I meant to use the abstraction that already exists (parseMetaFile), instead of adding new one (parseWpressPackage).

I've updated the code to reuse the parseMetaFile function.

Now MetaFileData has WP and PHP versions as optional and template and stylesheet also optionals.

Changed on: 4a08615

…ort-wpress-files

sejas · 2024-09-30T21:29:15Z

@wojtekn , @fluiddot , I've updated the code. Let me know what you think.
I'll add some tests in the mean time, but feel free to review the PR.
Thanks!

wojtekn · 2024-10-01T13:24:22Z

@sejas thanks for taking this over. I've just updated the testing steps.

It works well for me, allowing me to import a .wpress file. I noticed two issues so far:

I can't drag and drop file. I need to use a file selector to pick up the file.
Plugins on imported sites are inactive.

sejas · 2024-10-01T19:20:47Z

@wojtekn , thanks for reviewing it.

I can't drag and drop file. I need to use a file selector to pick up the file.

Good catch! It took me a while find that it was failing only in the add site form.
Fixed! 876bcf2

UbxWD5.mp4

Plugins on imported sites are inactive.

I've found that wpress creates a variable with the active plugins on package.json > Plugins.
I'm creating a sql file to activate them. 952f8d6

src/lib/serialize-plugins.ts

wojtekn

I've found that wpress creates a variable with the active plugins on package.json > Plugins.
I'm creating a sql file to activate them. 952f8d6

I couldn't make it work - after importing .wpress from the Atomic site, none of the plugins is active.

src/lib/import-export/import/importers/importer.ts

src/lib/import-export/import/handlers/backup-handler-wpress.ts

fluiddot · 2024-10-02T16:33:59Z

src/lib/import-export/import/importers/importer.ts

+		} );
+
+		rl.on( 'line', ( line: string ) => {
+			writeStream.write( line.replace( /SERVMASK_PREFIX/g, 'wp' ) + '\n' );


Do you refer that we should add a comment explaining why why we replace this fake prefix? It seems that wpress replaces the original prefix with SERVMASK_PREFIX, probably to avoid prefix collisions when importing to a shared database.

Yep, I was curious why we need this. Thanks @sejas for explaining it 🙇 !

src/lib/import-export/import/validators/wpress-validator.ts

src/lib/serialize-plugins.ts

fluiddot

LGTM 🎊 ! I confirmed I can import a .wpress backup file and themes and plugins match the original site.
Great work @sejas !

I've added some comments but none should block the PR.

fluiddot · 2024-10-02T16:56:07Z

@sejas following #497 (comment), it would be great to add unit tests to cover the import of this backup type. WDYT?

…ort-wpress-files

Add support for importing .wpress files

4b5c9a8

danielbachhuber self-assigned this Sep 2, 2024

Fix linting errors

0f5be5c

danielbachhuber mentioned this pull request Sep 2, 2024

Feature Request: Import site into Studio from anywhere with free backup/migrate plugin #496

Open

Merge branch 'trunk' into improve/import-wpress-files

9b8978f

danielbachhuber requested review from kozer, sejas and jeroenpf September 2, 2024 15:06

danielbachhuber mentioned this pull request Sep 2, 2024

Make it more obvious to the developer when the server needs to be restarted #499

Open

fluiddot reviewed Sep 5, 2024

View reviewed changes

sejas reviewed Sep 6, 2024

View reviewed changes

src/lib/import-export/import/handlers/backup-handler-wpress.ts Outdated Show resolved Hide resolved

Merge branch 'trunk' into improve/import-wpress-files

d4d8f2e

danielbachhuber assigned sejas Sep 23, 2024

Update theme and childtheme from wpress package json

a6182b6

wojtekn reviewed Sep 25, 2024

View reviewed changes

sejas added 7 commits September 30, 2024 20:26

Merge branch 'trunk' of github.com:Automattic/studio into improve/imp…

603c91b

…ort-wpress-files

Remove optional parameter on fd.read

8e7300d

add returns comment

66dceeb

Move chunk size to read as a constant

8bc06a8

use async ensureDir

2dda7df

Use buffer compare

34d773c

use parse meta file instead of custom method

4a08615

sejas marked this pull request as ready for review September 30, 2024 21:16

sejas requested review from fluiddot and wojtekn September 30, 2024 21:16

sejas added 2 commits September 30, 2024 22:24

remove unused offset

e16f250

remove console log

a2aad3c

sejas requested a review from a team September 30, 2024 21:29

sejas added 2 commits October 1, 2024 19:28

Refactor file types to use them globally

876bcf2

create SQL to auto activate plugins that were installed

952f8d6

move serializePlugins to its own file

19aeaf8

sejas reviewed Oct 1, 2024

View reviewed changes

src/lib/serialize-plugins.ts Show resolved Hide resolved

wojtekn reviewed Oct 2, 2024

View reviewed changes

src/lib/import-export/import/importers/importer.ts Outdated Show resolved Hide resolved

sejas added 2 commits October 2, 2024 15:51

Fix conflict on option already existing to active the plugins

ce7cbd3

remove unnecessary debug line to duplicate sql file

11dd3e2

fluiddot reviewed Oct 2, 2024

View reviewed changes

src/lib/import-export/import/validators/wpress-validator.ts Outdated Show resolved Hide resolved

src/lib/serialize-plugins.ts Show resolved Hide resolved

fluiddot approved these changes Oct 2, 2024

View reviewed changes

wojtekn approved these changes Oct 3, 2024

View reviewed changes

sejas added 9 commits October 3, 2024 15:40

check if file exists

2514225

replace empty dir sync for async version

a4eda36

simplify construction of extractionDirectory

350bfd3

Merge branch 'trunk' of github.com:Automattic/studio into improve/imp…

a1a815f

…ort-wpress-files

update error message

de7aec8

remove unnecessary function

4b06370

add test for serialize plugins function

2707ddd

add tests for wpress validator

c5ee97f

add package.json as required file

ae6e28c

sejas merged commit bae0104 into trunk Oct 7, 2024
10 checks passed

sejas deleted the improve/import-wpress-files branch October 7, 2024 06:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for importing `.wpress` files #497

Add support for importing `.wpress` files #497

danielbachhuber commented Sep 2, 2024 •

edited by wojtekn

Loading

danielbachhuber commented Sep 2, 2024

fluiddot left a comment

fluiddot Sep 5, 2024

sejas Sep 30, 2024

fluiddot Oct 2, 2024

sejas left a comment

danielbachhuber commented Sep 23, 2024

wojtekn Sep 25, 2024

sejas Sep 27, 2024

wojtekn Sep 27, 2024

sejas Sep 30, 2024 •

edited

Loading

sejas commented Sep 30, 2024

wojtekn commented Oct 1, 2024

sejas commented Oct 1, 2024

wojtekn left a comment

fluiddot Oct 2, 2024

fluiddot left a comment •

edited

Loading

fluiddot commented Oct 2, 2024

Add support for importing .wpress files #497

Add support for importing .wpress files #497

Conversation

danielbachhuber commented Sep 2, 2024 • edited by wojtekn Loading

Proposed Changes

Testing Instructions

Pre-merge Checklist

danielbachhuber commented Sep 2, 2024

fluiddot left a comment

Choose a reason for hiding this comment

fluiddot Sep 5, 2024

Choose a reason for hiding this comment

sejas Sep 30, 2024

Choose a reason for hiding this comment

fluiddot Oct 2, 2024

Choose a reason for hiding this comment

sejas left a comment

Choose a reason for hiding this comment

danielbachhuber commented Sep 23, 2024

wojtekn Sep 25, 2024

Choose a reason for hiding this comment

sejas Sep 27, 2024

Choose a reason for hiding this comment

wojtekn Sep 27, 2024

Choose a reason for hiding this comment

sejas Sep 30, 2024 • edited Loading

Choose a reason for hiding this comment

sejas commented Sep 30, 2024

wojtekn commented Oct 1, 2024

sejas commented Oct 1, 2024

wojtekn left a comment

Choose a reason for hiding this comment

fluiddot Oct 2, 2024

Choose a reason for hiding this comment

fluiddot left a comment • edited Loading

Choose a reason for hiding this comment

fluiddot commented Oct 2, 2024

Add support for importing `.wpress` files #497

Add support for importing `.wpress` files #497

danielbachhuber commented Sep 2, 2024 •

edited by wojtekn

Loading

sejas Sep 30, 2024 •

edited

Loading

fluiddot left a comment •

edited

Loading