Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release 3.2.0 #722

Closed
13 tasks done
kelson42 opened this issue May 8, 2021 · 54 comments
Closed
13 tasks done

Release 3.2.0 #722

kelson42 opened this issue May 8, 2021 · 54 comments
Assignees
Milestone

Comments

@kelson42
Copy link
Collaborator

kelson42 commented May 8, 2021

The 3.2 milestone is huge and has still a lot of work to do. I see work to do for at least 6 months. But the new ZIM support has been implemented and the support of this new type of ZIM files would benefit of an early release. How should we deal with this dilema?

@Jaifroid
Copy link
Member

Jaifroid commented May 8, 2021

@kelson42 I think there are quite a lot of things in that milestone that should be bumped off to a later milestone because not so urgent. I'd support an early release with just a few more bugfixes. Ideally I had wanted my tab-opening PR to be in the next release, as it's pretty much finished and working well, but there's an intractable test failure that appears to be an artefact of the testing system that I can't get to the bottom of and is blocking that one. I've implemented it in Kiwix JS Windows instead, where it is working really nicely even on the UWP app.

But I think we could release without that PR, as the no-namespace support is indeed important and was major work that would justify releasing.

@mossroy
Copy link
Contributor

mossroy commented May 8, 2021

I agree with @Jaifroid : we usually release a new version when there are significant changes, and move the remaining github issues accordingly.
BUT we were waiting for newer ZIM files to test (AFAIK we could only test with old ZIM files converted to the new format). It was also necessary to implement unit/UI tests on a sample one

@kelson42
Copy link
Collaborator Author

kelson42 commented May 9, 2021

@mossroy @Jaifroid Thank you for the quick feedback. Who is going to move the tickets?

@Jaifroid
Copy link
Member

Jaifroid commented May 9, 2021

Once we've decided on a release, we usually move the tickets just before actual release. @mossroy I guess we could release current master marking the no-namespace support as "preliminary", and say that the implementation could change. If you think it's worth it. We'd have three changes to list: 1. Preliminary support for no-namespace ZIMs; 2. Preliminary support for ZIMs with a version 1 article index; 3. Home key focuses search bar. We could probably add some quick bug-fixes to that, for example the race condition caused by the transition animation.

@mossroy
Copy link
Contributor

mossroy commented May 10, 2021

Do we now have ZIM files in the newer format, created from scratch? (i.e. not converted from older format). I think it's necessary to test some before releasing.
Do we now have a sample ZIM file in this new format, suitable for unit tests? (like the Ray Charles one)

@mossroy
Copy link
Contributor

mossroy commented May 11, 2021

@kelson42 : my last questions were for you :-)

@kelson42
Copy link
Collaborator Author

@rgaudin I forward the question to you.

@kelson42
Copy link
Collaborator Author

@Jaifroid
Copy link
Member

That engineering ZIM wouldn't be suitable for automated tests becuase it is 3.5GB, and the tests need to load the ZIM into memory in order to access it without the "user" having picked the file.

If it looks like the production of no-namespace ZIMs is going to be delayed considerably, I don't think it need be a blocker on us making new releases here, so long as we test manually on the ZIMs we currently have, until we have a reference Ray Charles (or similar) to use. And we mark the feature as "experimental" or "preliminary", and say it is support for the "specification" rather than for any actual ZIM.

Alternatively, we could develop the tests with the current no-namespace "converted" ZIMs we have, and slot in a reference ZIM when available. The risk is having to develop the tests twice if something important changes.

@mossroy
Copy link
Contributor

mossroy commented May 17, 2021

Can #720 be fixed before considering a release?

It's true that we should release a new version of kiwix-js before the newer ZIM files are publicly available : do we have an approximate time frame for that?

If we only have one such ZIM file privately available (the engineering one above), maybe it's too early to release kiwix-js. I'm not asking for a coverage of all kinds of ZIM files, but one looks too few. We should at least have some mediawiki-based ones (because it's the main use-case), and some other contents (videos, stackoverflow etc).

We also should add unit tests on a reference small one (like Ray Charles) as soon as it's available. I don't think it's relevant to develop tests on converted ZIM files.
Unit tests are there to help us release with confidence, and secure future refactorings. A major change like this one deserves such tests. If such reference file is not available on time, we can consider implementing them after the release, but it would be sad to have to take the risk.

Or are we in a hurry to deploy some of the other changes from kiwix-js changelog? (I don't think so)

@Jaifroid
Copy link
Member

Agreed, and yes I can tackle #720 first.

@mossroy
Copy link
Contributor

mossroy commented Jul 14, 2021

@kelson42 : do we now have no-ns ZIM files, suitable for more tests before a release? (both manual and automated tests : see above)

@kelson42
Copy link
Collaborator Author

New sotoki can made new zim files. Will try tomorrow to make a run so you can check.

@rgaudin
Copy link
Member

rgaudin commented Jul 15, 2021

@mossroy here's the smallest sotoki Zim built without NS. Uses Illustration API for the Icon as well

beer_stackexchange_com_2021-07.zim

@Jaifroid
Copy link
Member

Jaifroid commented Jul 15, 2021

Hmm, the detection of the API is failing badly on this ZIM (with our code). Did something change between the sample ZIMs we tested on and this production ZIM? It seems we can't get the article count or the article pointer position, despite the minor version being 1, hence we fall back to the Version 0 API. Our diagnostic is shown below.

image

(@mossroy There is also another issue caused by some error in DOM tokens -- to open this ZIM you have to comment out or try/catch around line 1360 of app.js -- you'll see the Exception which gives the line number depending on which branch you're on.)

@kelson42
Copy link
Collaborator Author

@Jaifroid I guess this is a question for @mgautierfr. I can not help here, AFAIK it should have been the same.

@rgaudin
Copy link
Member

rgaudin commented Jul 15, 2021

Well, we've updated the libzim to a more recent version of the master. I don't know the internals of all this so let me know if you need a specific info. from me.

@Jaifroid
Copy link
Member

@rgaudin We don't use libzim, hence we have to reproduce the API in JS. But the previous ZIMs we had to work with were all converted using a script, and they all validated fine. This one fails. I need to trace the logic step-by-step and will feed back to @mgautierfr (unless he knows of a change we need to incorporate -- @mgautierfr , please see my comment above #722 (comment)).

@Jaifroid
Copy link
Member

I've re-opened #708, where this should be discussed. I think I've determined the issue (an empty v1 directory listing). Please see #708 (comment).

@Jaifroid
Copy link
Member

Jaifroid commented Aug 16, 2021

@mossroy Since we have a production ZIM that works with our implementation, might we be in a position to think about hte steps necessary for releasing 3.2.0 with no-namespace support? As production no-namespace ZIMs are close to public release, if not already released publicly, I think we should do this sooner rather than later. It would be good to include #740.

@mossroy
Copy link
Contributor

mossroy commented Aug 20, 2021

Yes, it's a good step being done.
@kelson42 : Do we have a small ZIM file to implement unit/UI tests on, like the Ray Charles one? It would at the same time allow to test a mediawiki-based ZIM file (which is the most common use-case).
When do you plan to publicly release the no-namespace ZIM files?

And yes, we should include #740 in 3.2.0 : I'll have a look at it.

@kelson42
Copy link
Collaborator Author

kelson42 commented Aug 20, 2021

@mossroy We have started a repository to manage a reference test suite of ZIM file here https://github.com/openzim/zim-testing-suite. If you have not what you need there. I would recommend to request/put it there. I think first no-namespace ZIM files will be published in September (as soon as libzim7 will be published which can now happen almost any time).

@mossroy
Copy link
Contributor

mossroy commented Aug 20, 2021

@kelson42 , I opened openzim/zim-testing-suite#5 to ask for a small sample ZIM file suitable for our tests (the existing ones are not)
It's sad that this schedule does not give us enough time to implement these automated tests. It would be better to involve us earlier next time a major change is considered and planned.

Anyway, the wikipedia ZIM files from https://github.com/openzim/zim-testing-suite/tree/main/data/nons are enough to check compatibility with wikimedia-based ZIM files : I did not see any issue.

So I would suggest the following plan for kiwix-js :

@mossroy
Copy link
Contributor

mossroy commented Aug 22, 2021

I've added the usual task list on this ticket.
I might handle the milestones & move the issues right now
@Jaifroid : if you're available, you might update the changelog? Else I'll do it myself

@Jaifroid
Copy link
Member

I'm happy to do the changelog.

@mossroy
Copy link
Contributor

mossroy commented Aug 22, 2021

I moved to a new v3.3 the v3.2 issues we plan to work on at the hackathon.
I moved the other ones to v3.4
I also moved to "backlog" all the jQuery-specific enhancements

@mossroy
Copy link
Contributor

mossroy commented Aug 22, 2021

I tested many different ZIM files on my Firefox OS device and in a Firefox extension (both in jQuery mode) and found no issue.
I did not see any issue either on Chromium in SW mode.
But there is a blocking issue in a Chromium extension : the decompressor API says "Error loading ZSTD decompressor!" :

Could not instantiate any ZSTD decoder! RuntimeError: abort(CompileError: WebAssembly.instantiate(): Wasm code generation disallowed by embedder). Build with -s ASSERTIONS=1 for more info.
at A (zstddec-wasm.js:15)
at zstddec-wasm.js:18

and nothing works afterwards.

I seems to come from a CSP constraint

@mossroy
Copy link
Contributor

mossroy commented Aug 22, 2021

There is a workaround for this issue that I tested successfully by adding a line in manifest.json : https://stackoverflow.com/questions/48523118/wasm-module-compile-error-in-chrome-extension (works with unsafe-eval or wasm-eval)
Maybe we should open a separate issue for that, to consider compatibility with different versions of Chromium-based browsers (is wasm-eval supported by all of them?), compatibility with Firefox, and security risks that it might raise
I have to leave for now, will come back this afternoon

@mossroy
Copy link
Contributor

mossroy commented Aug 22, 2021

Just created #751 for this issue

@Jaifroid
Copy link
Member

Good catch. IMHO it's safe to add unsafe-eval because we only execute offline code. We also have a separate CSP for the iframe, which is where ZIM content is executed.

@Jaifroid
Copy link
Member

JFYI, I'm on my phone only for much of the rest of today.

@mossroy
Copy link
Contributor

mossroy commented Aug 22, 2021

@Jaifroid : if I implement #751 with the unsafe-eval , can I do the release of 3.2.0?
i.e do you think you made enough tests on Windows platforms (including IE11) for the release?

@mossroy
Copy link
Contributor

mossroy commented Aug 22, 2021

Let's not rush, and take the time to review #754, including testing it as an Edge extension.
Hopefully the CI instability will be fixed in the following days (it might have blocked our release anyway)

@Jaifroid
Copy link
Member

Sorry it turned out I was in a no signal area all day. I've tested the release a lot via the last PR, so happy for you to release when you wish. I'm also happy to wait until we're sure we don't have a blocker with WASM.

@mossroy
Copy link
Contributor

mossroy commented Aug 22, 2021

No problem!
You must be overwhelmed by github notifications today : take your time

@mossroy
Copy link
Contributor

mossroy commented Aug 23, 2021

The unsigned packages have been nicely generated by GitHub Actions, and uploaded in https://download.kiwix.org/nightly/2021-08-23/
I uploaded the Firefox package in their store : there's a new warning about our unsafe-eval policy : https://addons.mozilla.org/fr/developers/upload/f8bcf19b4fdb45698be20b3a04232e60 , but it has been accepted.
I've also uploaded the package for Ubuntu Touch

@mossroy
Copy link
Contributor

mossroy commented Aug 23, 2021

@kelson42 and @Jaifroid : please upload the package https://download.kiwix.org/nightly/2021-08-23/kiwix-chrome-unsigned-extension-3.2.0.zip for the Chrome and Edge stores

Regarding the Firefox store, it would be safer that both of you also could also manage the addon : uploading a new release, or any other need.
I can easily give you rights on this addon, but you first need to create an account on https://accounts.firefox.com/ , and give me the corresponding e-mail address. Please activate 2FA on your accounts for more security.

@mossroy
Copy link
Contributor

mossroy commented Aug 23, 2021

@ernesst : Could you please test this new version on your device? It's already uploaded on https://open-store.io/app/kiwix
Could you also give us the "API status" you get at the end of your "Configuration" section? (snapshot or copy/paste)
Last but not least, could you try to activate the ServiceWorker mode in this section, and tell us if there is an error in the API status, and if you can browse ZIM files in this mode? We plan to switch to this mode by default in next release so I'd like to make sure it also works on Ubuntu Touch.

@Jaifroid
Copy link
Member

I've uploaded the extension to the Edge Store. Annoyingly, since the last submission the Store has come out of Beta, and it forced me to rebuild the page from scratch. I took the opportunity to update the screenshots. It says it might take 7 business days to review.

image

@ernesst
Copy link
Contributor

ernesst commented Aug 24, 2021 via email

@mossroy
Copy link
Contributor

mossroy commented Aug 24, 2021

Thanks for your answer @ernesst : is it temporary or permanent?
If it's permanent, do you know someone else who could test our releases?
According to the stats of open-store.io, the previous version (3.1.0) has been downloaded 415 times (surprisingly twice as much as the 207 times of version 3.0.0), which is not negligible.

If it's temporary, we have no rush on our side : the tests can be done later (any rough idea of when you would get a suitable device?)

@kelson42
Copy link
Collaborator Author

Release is waiting validation on Chrome Extension Store. But I have already upload the CRX to download.kiwix.org.

@ernesst
Copy link
Contributor

ernesst commented Aug 25, 2021 via email

@kelson42
Copy link
Collaborator Author

Kiwix JS 3.2.0 has finally been published on the Chrome Store

@Jaifroid
Copy link
Member

@Jaifroid
Copy link
Member

Jaifroid commented Aug 29, 2021

Actually, there now appear to be two Kiwix extensions in the Microsoft Addons Store. Remember I mentioned I had to rebuild the previous version because the previous one had been in the Beta Store? Well it doesn't appear to have taken the place of the old one, so we now have two versions, 3.1 and 3.2. I think I'm going to have to sort this out with Microsoft, because I don't seem to have access to the old version. Grrr. In the meantime, please don't advertise the URL I gave in the post above, because there are 274 users of the old one. Maybe I missed a step somewhere. It would be preferable to update the old version with all those users.

@Jaifroid
Copy link
Member

OK, I've sorted out the issue and have managed to submit the package to the correct place now. I'll delete the rogue version.

@kelson42
Copy link
Collaborator Author

Let me know when everything is ready for the annoucement.

@Jaifroid
Copy link
Member

@kelson42 From my perspective it's fine for you to announce. The Edge Addon will update in due course (in a few days), but there's no reason to delay the general announcement.

@mossroy
Copy link
Contributor

mossroy commented Aug 29, 2021

OK for me to announce too

@mossroy
Copy link
Contributor

mossroy commented Sep 3, 2021

The last remaining task is to announce this version, for @kelson42
I'll close this issue, to be able to close the milestone.

@mossroy mossroy closed this as completed Sep 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants