When will the 4.8.0 version be released？ #793
Comments
|
This question was asked a few months ago and the answer hasn't likely changed much. Please see #778. In that thread you will see references to #437 Comment which contains a wealth of information on the topic. In all of this, one thing you should keep in mind is that many people (even Microsoft!) currently use portions of Lucene 4.8 in production. So that is to say the product is already very stable. Companies using 4.8 in production just need to be awere there may be some minor API changes on the road to a final 4.8 release. But honestly, that's not a lot different than using a Production release that has a major new release. |
|
In my opinion, if you know people are using it, and you feel it is stable, you might as well call it a release and remove the beta designation. I am confident it works for most use cases. However, I can't get some developers to consider it. If it gets escalated, it will most likely get dropped. I feel it is time to go forward. |
|
I'm definitely in agreement with @jeffreywstevens here. I know there are some recent commits and changes done in the past several months which would warrant a new beta release, but after that I also think we should procure an RTM release. Any changes after that can just be patch versions. Would need buy in from @rclabo + @NightOwl888 |
|
I'm totally on board with that. In the past two years I've had to make way more changes to my code base due to ASP.NET Core API changes than I have had to for Lucene.NET 4.8 API changes. I feel that LuceneNET 4.8 is super stable (hats off to @NightOwl888!!!) and is worthy of an RTM release. Sure there are some aspects that may not be perfectly on par with Java Lucene 4.8 (OpenNLP comes to mind), but those areas tend to be auxiliary functionality that have no easy route for porting. The core functionality seems rock solid and awesome from my perspective. And I think so many more devs will use this amazing project if it's RTM. |
|
I know @NightOwl888 is under a lot of pressure right now due to a deadline on another project, so he might not be able to chime in for a while. That doesn't mean he isn't interested in this discussion; it just means he is juggling a lot at the moment. |
|
@rclabo IIRC there were quite a lot of commits and fixes since the last release, do you think we should look to ship one more beta? |
|
@Shazwazza Probably, but honestly...I feel like that question is above my pay grade :-) I have enormous respect for @NightOwl888 and would certainly defer to his judgment. |
|
I have some time on my hands and can dedicate helping out with the efforts. Just from looking over the NuGet download stats, the 4.8 beta packages outnumber the last production 3.3 release, in terms of downloads: https://www.nuget.org/stats/packages/Lucene.Net?groupby=Version
These numbers could be misleading and inflated due to automated CI builds but still paint a good picture for 4.8 usage. Some of us are using 4.8 betas in prod without issues, and anecdotally we hear about that from other people too. If another beta release makes prod 4.8 a reality, let's go for it. Until @NightOwl888 can chime in, I will start pulling together the changelog and see what the new beta release would look like, and we can try pushing that out and get the ball rolling. |
|
I put together a draft for the next release; I believe people with commit access should be able to see it in the releases page: https://github.com/apache/lucenenet/releases It's pretty meaty. As the next step, I will review the communication from @NightOwl888 from the previous beta build and see what we need to do to proceed. From the top of my head, there is a PMC vote and then the publishing of the NuGet packages if the vote passes. I also want to set up local tests for handling indexes produced by previous versions to ensure the current version can work and open it, etc. |
|
Just a quick status update, I am going through the steps outlined here to ensure I have all the bits correctly set up locally to do the release. One thing I am not clear about is the Azure Pipelines and the access that is needed there to make a release. But I haven't gotten to that part yet, so I haven't explored it too deeply. |
|
A quick update. We have sorted out access etc., and are actively working on finishing up a few remaining things that will allow us to push 4.8 beta 17. I can dedicate a decent amount of time now and have been pushing PRs with the remaining fixes. Shad has chimed in as well and has started some work too. No ETA, but we are back up and pushing to the finish line. You can observe our progress here by watching PRs coming in and out: https://github.com/apache/lucenenet/pulls?q=is%3Apr+is%3Aclosed Our focus areas are 1) fixing the findings by SonarCloud code scans that indicate a problem with converted code where base class constructors call virtual methods that can be overridden in subclasses and cause issues with the state not being initialized properly. 2) Fix Close/Dispose issue with the analyzers #271 And then we will regroup and see where we are at with 4.8 release. There is still some work/considerations to be made about ICU4N. No ETA, as I don't think we can estimate how long this will take, but we can take it as we go and make regular status updates. |
|
I think we're ready, |
|
I had to go on a month+ trip but back now. I haven't heard much from @NightOwl888 recently, he must be busy with some other commitments. The last piece of work I pushed before taking off was this #852 . It's not entirely clear if I can pull into main what we have there or if Shad was considering more changes to the approach. Having re-familiarized myself with the project and talking with Shad more about why it's been difficult to make a production release, I think I can give this explanation for it:
The difficulty lies in what to do if we release 4.8.1 and find a bug. OK, we make a patch release, 4.8.2 that fixes that bug. But now, Java Lucene does not have 4.8.2 version. Worse, what if the issue we discover requires a change that's a breaking change, and we in theory would increment the minor version, end up with 4.9.x release which would have API/changes that are not compatible with Java Lucene 4.9.x releases that exist. And with fixes to those we would have releases that don't exist in Java world once again, e.g. 4.9.3. And I think that's the main issue why Shad has been extremely careful and reluctant to do production releases of the project. We know that bugs are lurking in the code base, but with each pass they are more and more difficult to find, and we can't guarantee that 4.8.1 we release will not require changes. A careful discussion and consideration is needed here, but one way forward would be to come as a group with the remaining committers that still at least chime in and perhaps draw a line in the sand and say ok, 4.8.1 prod release we are making attempts to be as close as possible to Lucene 4.8.1 release. All releases going forward from that will attempt to stay close at the "major" version but all the minor/patch releases can and will deviate greatly. I am not proposing this lightly, but it does seem to offer some sort of way forward with making a production release and potentially allowing for a more frequent prod update cadence without keeping ourselves accountable for those versions to be one-to-one mapped to Java world. |
|
@laimis I think that makes a lot of sense. Given that we haven't previously had an approach for versioning when rolling bug fixes or breaking changes once Lucene.NET 4.8 is released it's very understandable that we have held a very high bar to what needs to be achieved before doing a production release. I personally think that what you propose as a solution seems reasonable. And who knows, perhaps someone will offer up other solutions that may be even better. But I think as a dev community we need to rally around some versioning approach whatever it is. Having a versioning approach and an understanding of what versions align with Java Lucene and which ones don't will give us the freedom to get Lucene.NET released. Doing a production release of the library will untie the hands of developers that would love to use it but who are restricted from doing so due to company policies not allowing pre-release software into production environments. Releasing the software will thus grow our developer community and hopefully our committer pool as well. Also, releasing the software will grow the use cases that are actively being utilized and provide valuable feedback on where the library can be improved. What you propose seems reasonable however it's a bit challenging that this is a release of 4.8.1 rather than 4.0. As such we only have 4.9 as a potential breaking change release, then we hit a major version 5.0. This could cause us to be forced to release a braking change as a point release, say 4.9.1. This challenge of course goes away in the future if the next major release of Lucene.NET has a low minor release number. e.g. 10.3 but we have the same issue in the future if the next major release of Lucene.NET has a high minor release number like 9.7 (the current version of Java Lucene). It's a bit challenging I guess, but we may just have to get comfortable with the idea of a breaking change in a point release. ie. 4.91. (shrug) |
|
@rclabo thank you for chiming in. Curious about this part that you mention:
After 4.9, wouldn't we have 4.10.x as an option? 4.11.x after, etc? |
|
@laimis That really made me laugh (at myself). You are totally right. For some reason when I wrote that it didn't even occur to me that we could have a 4.10.x! That's pretty funny. Definitely, after 4.9 we can have 4.10.x as an option, and after that 4.11.x. Thanks for being gracious in your question. ;-) |
|
I've been reading your latest comments about the problems with versioning and a production release and think I have an idea to solve this problem. What if we use the Lucene version as is and then add an extra number to the end to signal the current iteration of the version? That way you still have the consistency of matching the Lucene Java version and .Net version but can still make improvements like bug fixes that weren't caught in a preview/beta phase. (See this image for an example)
We also have to remember that no one can be sure that they have bug-free software and that unforeseen problems do come up no matter how long we work on something. So I think it would be better to use iterations instead of neverending beta releases like there have been for a while now with the current 4.8.0 release. This also gives a better signal of when you can use Lucene in production as has been mentioned time and time again that many people do even though it's in beta. |
|
@nikcio - I think this is a fine proposal and in some ways, I like this approach better because it makes it more clear which version LuceneNET is in rough alignment with, The one thing lost with this approach is the ability to tell, via the version number, if an iteration of a version is a breaking change. But honestly, that doesn't bother me personally a bit. In my case, if I'm upgrading to a newer version of LuceneNET for my project, then I'm probably reading the release notes to see what new goodies it includes. And in that process, I'd be made aware of any breaking changes and the nature of those changes. That's sufficient for me and probably for a lot of devs. However, I know versioning can be an opinionated topic so it will be interesting to see how others on the dev mailing list feel. |
|
I realize this is a lot bigger topic, but I think the maintainers of this project should seriously consider breaking off from the exact version scheme of the upstream Java Lucene. As a consumer of this library, naturally I would like to know what API version of Lucene this corresponds to, but that could easily be solved by a version mapping table in documentation. Examples such as
indicate just how hard it is to keep the versions of distinct code bases the same. Especially the patch number is troublesome as that typically designates implementation and bug fixes, but I think the same applies to minor and major. By releasing yourself from this constraint you would have the flexibility to release stable versions of the functionality that you have implemented without waiting for 100% feature parity with a given upstream Java version. This way you may opt to never be 100% feature complete with Java Lucene 4.8 (for example), because the community is more in need for some 7.x features that can then be prioritized over the long tail of rarely used 4.8 features (just as a made up example). By following your own version scheme you can instead document version X as "compatible with Lucene 4.8 minus features Y and Z". It would also possibly be easier to get contributors, as most consumers of a library would rather contribute a PR that just adds a feature from a later version that they need for their application. Sorry to be blunt, but it's going to be very hard to get contributors chasing the last bits 4.8 compatibility. The additional value is that you can now follow semantic versioning more strictly, something I would argue is an industry standard these days. It would sure make maintaining libraries that depend on Lucene.NET easier. |
|
First of all, the versioning scheme had been decided some time ago and is in fact documented and made part of the build. At this point I don't see any reason to go back and revisit this scheme which was part of the work that was done during the first 4.8.0 beta.
This assumes usability and API are the entire issue, but they are not. Lucene.NET is the most difficult application I have ever had the pleasure of debugging in my 25 years as a developer. When we go off the map like this, we literally throw away our best debugging tool, which is to run the same version of Lucene and Lucene.NET side by side to see where the execution paths diverge. I don't have an answer for how we could debug if we combine different versions of Lucene. Do you? Furthermore, the binary structure of the index does change from one version to the next, making them incompatible and making it literally impossible to bring many Lucene 9.x features back to Lucene.NET 4.x. We had this issue with back-porting the analyzers-nori package. We have 100% compatibility with creating an index in Lucene and opening it in Lucene.NET with the same version and plan to keep it that way going forward (and it worked once the other way around, but hasn't been tested in quite a while). The index isn't the only binary format that is also kept in sync between versions. There are other problems with disjointed versioning between Lucene and Lucene.NET. Case and point: Lucene.NET 3.0.3. There was no release of Lucene 3.0.3. Despite trying to sleuth an answer I have no idea what commit Lucene.NET 3.0.3 is a port of. I could guess that it is a port from 3.0.1 (which actually was released), but I can't be 100% sure. I didn't even know what commit in this repo corresponded to the 3.0.3 release until I found it on an obscure blog (they released 3.0.3 RC2 by renaming it, but didn't make a tag corresponding to the 3.0.3 release). Both of these issues are the primary reason we have never done a maintenance release of Lucene.NET 3.0.3. While we could incorporate the actual version number as part of the For usability, there are also issues. Existing Lucene blog posts may not be useful if the API is different than the major version of Lucene the post is about. The bottom line is there is no maintenance plan for making a Frankenstein version of Lucene that incorporates features from different versions. The best way is to try to sync the entire project to a single Git commit. The story goes way beyond keeping the API in sync. It also means keeping the execution paths, binary formats, tests, and documentation in sync. While we could simply abandon 4.8.0 and start working on the latest version of Lucene now, we would be stuck in a situation where we have all of the same work to finish we do now plus an estimated 1800 hours of upgrading work. This upgrade estimate could be off if we run into any major gaps that mean more JDK features we need to find or build replacements for. Right now, we are in a situation where our remaining work still has an undefined scope because of gaps that we may not know about. The plan is to try to close all of the gaps so when we finally do start working on the upgrade we have a mostly well-defined scope of work instead of a fuzzy "research this and figure out what we need to do here" situation, where research is often most of the work (meaning to create an issue about it, we need to do most of the work first to define the scope of the issue). Also, seems like a total waste do to that. Most of the work that is remaining is on ICU4N. I have almost convinced myself that we may be able to release ICU4N as stable earlier by not strictly following the ICU versioning scheme but instead allowing each major release to have breaking API changes until we stabilize it (we are 13 versions behind so we have some wiggle room, but it does mean we will have to do a full upgrade every time we make a breaking API change). But we should probably still conditionally compile out the "draft" APIs and other APIs that are considered unstable in the NuGet package or at least make them invisible to the IDE. There are still other issues to deal with, such as the fact that NuGet doesn't actually deploy resource files for cultures it doesn't recognize. There are many decisions to make like that in ICU4N where there are gaps between Java and .NET. Unfortunately, nobody here seems willing to talk about the actual work that remains. Most want to move on to the next version of Lucene and pretend that we don't need to do this work for the upgrade, anyway. We could alternatively move on to 4.8.0 release while keeping the It really sucks for us to have to reject what would ordinarily be good ideas from the community, but unfortunately, most of these ideas never take everything into consideration when providing such advice, only the "normal stuff" that most projects deal with. |
|
Shad, thank you for that. I feel like it just pulled me back into reality. So I guess what you are saying is we can't have a "stable" Lucene.NET release unless its dependencies are stable and currently Lucene.NET.ICU is a work in progress with a changing API surface. I'm reading into that, ICU4N, which Lucene.NET.ICU depends on, is also probably a work in progress. And it's certainly worth noting that ICU support is something the Java Lucene team got for free in the JDK that unfortunately isn't included in the .NET Framework (full or core). Hence the need to create ICU4N to provide that support. A nontrivial endeavor in its own right. In using Lucene.NET to create a search index for an e-commerce marketplace, I've never hit any ICU-related functionality that was missing that I felt I needed. Unfortunately, I have no prior history with ICU so my only learnings about it have been here on the Lucene.NET project. So I guess for me, it's often an out-of-sight, out of mind, portion of Lucene. But when I review the docs for Lucene.Net.ICU and see what's included, it feels very central to a search library and encompasses such basic functionality as finding word boundaries and line break boundaries. While this seems trivial in languages like English it's anything but trivial in languages like Chinese 要弄清楚如何分解中文單字是很困難的。or Japanese 中国語で単語を区切る方法を理解するのは難しいです. Given that a great many of the developers using Lucene.NET only use it for English text, or other languages that use the Latin alphabet, it's easy to see how we can sometimes lose sight of what ICU is and why it's so important. Based on your post, I now better understand why Lucene.NET hasn't had a public release yet. Still, it seems very unfortunate that such a stable product (at least for indexing Latin languages) has a current version (beta) that doesn't indicate it's production-ready for Latin languages. I'm with you a 100% that doing a Frankenstein version of Lucene that incorporates features from different versions. is a non-starter. Being able to compare execution paths with a corresponding Java version is too valuable to give up. |
Not exactly. We could do a release if we go over the API surface of the core and other completed components to finalize it AND build a multi-release scheme so we have 2 different release labels, one for the stable components and one for the unstable components. While the API work is something we have to do anyway, changing the build, release policy, Git labeling scheme, etc. isn't exactly free.
Yes, ICU4N is still a work in progress. There are several tests that either still fail, often due to gaps that we haven't yet covered. There are also some concurrency bugs to track down. Since it is only a partial port, we have lots of tests to go through that might be able to be ported, as well. The intention is not to port any more of the production code (except for perhaps some of the formatters and parsers because that is where most of its funding has come from so far). The ICU4J functionality is not in the JDK. Instead ICU4N is a port of ICU4J. But it is hard to integrate because the gap between Java and ICU4J is not the same as the gap between .NET and ICU4N. Although, it is made easier because ICU is documented pretty well. In short ICU4/J extend the text processing capabilities of .NET and Java by providing rules-based versions of some of the included components (such as the We use the ICU4N
Actually, there are several use cases that even make it valuable even to Western European languages. For example, for removing diacritics from words. In .NET, this cannot be done without a hack because the normalization feature is missing the case fold option that ICU has. I have seen many people post this hack in their questions about Lucene.NET even though they could just use the These make it so words with accent characters such as resume, résumé, and resumé all normalize to the same root word for searches. Although the components inside of the |
|
Thank you for the really nice and transparent explanation, @NightOwl888! Ultimately, it is down to a fundamental architectural decision on whether this is a line-by-line, version-by-version port of the Java Lucene or if this is a full-text search library based on Java Lucene. This decision is one that would be made by the maintainers, and respected by the users of this library.
If I read the entire thread correctly, there was never a suggestion to just abandon 4.8, but instead to decide the API is stable and focus on bug fixes, then release 4.8 and figure out a different way to version the library so that API changes can be done later. This way, going from beta to release would mean the current feature set is stable, but without the guarantees of implementing 100% of the APIs of the Java version. Just to give an example, speaking only from my experience with the library, I personally was not aware of the desire to keep on-disk binary formats the same between Java and .NET. We are only using a subset of all this functionality, and we would definitely not be using the Java version, let alone on the same data. We don't care about Java Lucene at all, we just want a really good .NET full text search engine (actually we don't care about on-disk format at all as we are 100% in memory, but that's a different story).
I respect the decision to do a line-by-line port of Java Lucene, but I do like to point out that porting the most relevant features would not necessarily lead to a "Frankenstein" version. Obviously any feature that goes into the codebase have to be well architected and any technical dependencies for this feature have to be implemented properly. But consider if the goal was just to make the best .NET full text search engine out there, maybe omitting the long tail of rarely used features to not have to spend 1800 hours on version 4.8, instead focusing on the most popular features (again, building on robust foundation) may be serving the community better. This could perhaps lead to a higher engagement from the community (in terms of collaboration/PRs and possibly funding). You could still use Java Lucene as a blueprint for the implementation, but with the additional insight in what turned out well and what did not turn out so well there, without being burdened like they have by keeping compatibility also with less used and less well designed features.
To be blunt, and in all respect, it might get hard to find funding for hundreds or thousands of dev hours fixing the ICU library to support rare scripts and languages, until someone with a clear business case for it turns up. Just for comparison, if some company needed, say, vector valued fields (just as a random example) they might have the resources to fund the maintainers directly or devote professional developers to work with you on implementing this feature. But since I understand it you want to go to 9.something directly after 4.8, maybe we'll see a lot more contributions coming in as the field will be more open for new features.
If you have policies against pre-release libraries this is probably also a no go. I think policies like this are based on the assumption that pre-release means unstable implementation, while you mean unstable API. This is probably the core of this discussion, as it is clear that the code base is very stable from a bugs point of view. It sounds like you have made a well-motivated and conscious decision w.r.t the versioning policy and the way to integrate new features. Your code, your versioning policy. Thank you for an awesome effort! |
You are making some assumptions that just aren't true here.
Without keeping the binary formats the same, we would have to recreate all of the corrupt indexes for the tests. Arguably, the index format is the one thing that the Lucene team gave the most thought to about making Lucene portable across programming languages. Granted, we could use the documented format and try to reinvent the wheel for the rest, but there are a lot of components that would have to be analyzed at a high level so they could be recreated. In addition, Lucene also has pluggable codecs so a newer version of Lucene can read the binary format from an older version so users can upgrade the software first and then upgrade the index later. Maybe you don't use this feature, but for users of apps with high availability, this feature is a must. There are over 3000 code files in Lucene and it is not documented well - it could easily take years of analysis before we even start writing anything. We wouldn't even have much of an idea which features are important and which are not without tons of analysis and research. And when we are finished, there would be no reasonable way to incorporate features of new versions of Lucene (which is what happened on the NUnit project). As for upgrading a single feature ahead of where it is in Lucene, this is where we run into problems. We have no idea before porting it what other patches it depends upon and whether any of those depend on binary formats that have changed. So we could start off porting to get the "future" feature in 4.8.0 only to find out later that it is incompatible and all of the work porting that one feature would go out the window. It would take much longer to port Lucene feature by feature than it would be to port the diff between 2 commits to get to a higher version. And we would always be sure to have a version that works (at least as well as it worked in Java).
That is true about funding. But the fact of the matter is that ICU4N has had more funding than Lucene.NET even though it is an alpha with unstable APIs and we still are working out how to properly package it. Maybe it is easier to get people to fund Lucene.NET if ICU4N is a done deal, but Lucene.NET moves on without ICU4N my fear is that ICU4N will never be released. It is a tough sell to "release" Lucene.NET 4.8.0 and then ask for funding to "finish" it (which is basically to subsidize ICU4N). And it doesn't seem right to sell people on the idea that we are collecting funding for the upgrade only to shift that funding to finish ICU4N. It is far easier to finish ICU4N first, then release it, then release Lucene.NET, then ask for Lucene.NET funding for the 1800 hours to upgrade it (which is a pretty well defined scope). You are right in that doing it in this order means there is less help on Lucene.NET, but that isn't really where the help is needed until the upgrade anyway. We have analyzed this pretty well and this is by far the fastest path (even though it is taking years because of limited funding and help).
For the most part, yes. There are a few intermittently failing tests we have yet to track down. We mostly just have several APIs that are likely to break before the release. Since |
These are excellent points. Lucene is relatively easy to use as a library so it's easy not to realize just how sophisticated it is under the hood. It's hands down the most sophisticated software I have ever worked on. The amount of brilliant propeller head thinking that has gone into this product can't be overstated. Some of the best minds in search have contributed to Lucene. It's truly an amazing piece of software. And making changes to its internals is not for the faint-hearted. :-) |
Points like these really sold your "line by line" approach to me. The (incorrect) assumption that I made was that most/all of the contributors and maintainers are as familiar with the (Java) Lucene codebase as the core Lucene devs, or the degree of communication between the projects. Admittedly, this was an assumption I made without looking it up. If this is not true, then any other approach would fail, agreed. |
|
Just to clarify, Lucene has a lot of documentation, and Lucene.NET has it's flavor of that documentation as well. By many standards, it's decent documentation. But it's one thing to document how developers can use an expansive library like Lucene, and quite another to document why each design choice was made the way it was and how the specific implementation details of that design enable the insanely fast overall indexing and search speeds of Lucene. There are many small aspects of the system that use such advanced software engineering approaches that a dev could easily spend more than a month if they wanted to understand that aspect of the system deeply. Lucene's use of automata is one example. Here is a video at a conference that does a high-level overview of how and why Lucene uses automata. If a dev wants to understand automata they will need to watch videos like that one and ultimately hunt down the whitepapers. Once those whitepapers have been digested, maybe the dev will have the ability to understand that portion of the code. Maybe. We are assuming a very senior dev here. A dev is not going to find deep documentation on automata in Lucene's source code or external documentation. (shrug) There is, of course, the Lucene dev mailing list archive, an archive of completed issues, and PR notes. All three of which contain a fantastic amount of history and insights. |
Indeed explanation, so when I was working on adding the Sequence Number feature found the same issue really uncertain about the Lucnenet roadmap. @NightOwl888 / @rclabo can anyone list the issues where I can work on to me open to contributions focusing on production-grade features. |
@NightOwl888 I am a Lucene Java programmer myself and am happy to help in any efforts to maintain two-way compatibility between Lucene and Lucene.NET. |
|
Any way of getting updated nuget packages from master or do we need to build on our own? |
@turowicz - We are working on a Lucene.NET beta release now. And we could use some help as there are several tasks to complete before the release. This is still a work in progress, but I have a milestone setup for J2N with some open tasks: https://github.com/NightOwl888/J2N/milestone/3. The plan is to roll out a release of ICU4N and there are several tasks to work on in this project as well. |
|
does it mean just building the master from source is not a good idea then? for .net8 |
|
You are welcome to build from source. We don't currently have a target for It seems that help is coming from some PMC members to work on the release, but they are not available immediately. So, between that and the fact that we have to have a release vote (which takes 3 days), it could be 2 - 3 weeks or so until a release is available on NuGet. Unless of course we get more volunteers. |
@superkelvint - Sorry for the late reply. I didn't see your comment back in April. Thanks for offering to help with compatibility. One way you might be able to help us is to add support (even if it is unofficial) to the latest version of Lucene to read 4.8.0 codecs. The backwards-codecs package only goes back to Lucene 5.x. Our plan is that once Lucene.NET 4.8.0 is stable to jump ahead to the current version, so it would be beneficial if Lucene.NET users could upgrade the software first and upgrade their index at some later point. It would save us some time if we didn't have to grab the 4.x codecs from the last version that supported them and try to splice them into the backwards-codecs package, as offhand I don't really know what is involved. Another way you could help us (since you linked to the issue) is to provide some guidance on the analysis-nori module (latest work here). We got most of it working, but there are 3 test failures that were difficult to find an answer for. The tests are Now, since the kuromoji module is almost identical and it runs on 4.8.0, I suspect there is a solution. I have already asked the Lucene team, but their advice was just to wait until we upgrade. However, if we have someone who is willing to help us find a solution, maybe we can make this available sooner. |
Happy to wait for 3-4 weeks for a nuget release. Unfortunately I can't contribute. |
|
@NightOwl888 3 weeks have passed, how's it going 😄 |
|
I can't speak for the group, but here's my perspective. I had hoped to help out some starting a week ago but got bogged down in another task at work that has a hard deadline with big consequences for missing. So I'm probably another week out before I can help at all. @NightOwl888 is amazing, but many hands make light work, and right now, we appear to be short of hands. So I wouldn't be surprised if it takes a few months to get the next version out. We'll see. |
|
In the meantime, how one should go about using the latest version? How to tell what is "stable"? |
|
We generally keep the We are getting some traction on this release, but there are still many tasks to complete. Most importantly, getting J2N and ICU4N in a releasable state and fixing the API doc generation. |
@turowicz you may have a specific reason that you want to build from master or to get nugets from the nightly build, and if so, that's cool . But it's probably worth mentioning that Lucene.Net 4.8.0-beta00016 at nuget.org is also very stable and it's what I'm currently using. |
|
re: blockers for the next beta release, I've been looking into #911 and upgrading DocFx to the latest version, but it is not a cakewalk. I've made good progress and can mostly build docs without plugins using the combined docfx json file, but the individual files don't work with the TOC yet. At some point we might have to decide to push out a new beta release without docs (or with half-broken docs), unless anyone with experience with this can jump in and help me get this across the finish line. |
|
I had put together the docfx docs together before. IIRC if you update to
latest docfx, it might not support the old plugin system which we need.
I'll try to find time next week to see what I can do.
|
|
@Shazwazza Thanks, I'd appreciate any assistance! It indeed does not support the old plugins, but those just replace markdown elements with things like environment variables, so I'm punting on that for now and saving that for the end. So currently I'm just trying to get the docs to build correctly without plugins enabled, and that's causing issues. It's complaining about a duplicate toc.yml file, which Shad told me you were able to work around previously. I think this is a large part of why the individual docfx json files (scripted via docs.ps1) fail to generate correct output for the individual library "sites." You can check out my progress here so you don't have to start from the beginning: https://github.com/paulirwin/lucene.net/tree/issue/911 - I've improved the docs.ps1 script to install the latest verison of docfx as a global tool, fixed a lot of doc build warnings, etc. Running the docfx global tool manually on the combined file generates a site that mostly works (again without plugins yet, so env vars don't work), but if you clean your repo and run docs.ps1 you'll see that the site is broken for all of the libraries off the main site. |
|
@paulirwin ok sounds good, ideally we're running latest docfx for sure so if we can get there, that will be the best bet. Converting the docs is a pretty crazy challenge, especially when we have to deal with the craziness of docfx too - though believe it or not, its less crazy today :P I'll try to make time next week and let you know. Have pinged you now on Slack |
|
Hey @Shazwazza and @paulirwin I had a little look at what you've been doing and added a bit to it. I've added a description in this PR of what progress I've made: #958 Just close the PR when you're done using it I don't think I have the correct knowledge to add what is still missing 😅. |
|
An update on this thread, #961 gets the docfx build working again and has the added bonus of using the latest docfx, so we're now up to date and can even build docs cross-platform. Thanks again to @nikcio, @Shazwazza, and @rclabo for the assitance! I'm now looking into helping @NightOwl888 with ICU4N and J2N as he needs to help get that done. |
But more .NET system (even Microsoft!) have already chosen elastic search(lucene),or meilisearch. lucenet.net Just like the elderly, move too slowly and need to pick up the pace. |
|
Let's please leave out comments like that about elderly people. However, I understand and sympathize with the sentiment of the rest of your message. Those of us that are actively working on this project are currently giving it all we can to get this done. To provide an update which is due anyways, we are nearly done with preparing the next release. The ICU4N and J2N releases to support .NET 8 and satellite assemblies are done and dependencies upgraded, so that we can now target .NET 8, which is PR #928 that was just merged today. This should hopefully be the last major item for the beta 17 release. We are reviewing now to see if there are any remaining issues that need to be included. This will be our first beta release in a couple years and we're eager to get it done, but we also want to get any breaking changes out of the way all at once. After the beta 17 release, my personal goal is to help us get to a final "RTM" release of 4.8 this calendar year if at all possible. We'll need the community's help in testing the beta 17 release to make sure it's solid. This means not only filing bugs, but letting us know in the affirmative that it's working well for your workload so that we can have some confirmation of our work and how it is being used. Please do not jump in to port post-4.8 features at this time; any PRs for these will likely be closed and your efforts might be wasted. Likewise, we apologize but we are not accepting significant architectural changes (library structure, .NET version, C# language features, etc.) at this time. We can't have this be a moving goalpost; we need to focus on getting the release done. Stay tuned, we're pushing hard, and we'll hopefully have a beta 17 release ready for testing very, very soon. |
|
As a .net developer, I'm almost 40 years old. I'm old. I now want to do some amateur website type applications based on .NET. And lucene.net seems to be difficult for me to muster up the courage to choose, it is not as functional as lucene, and it is immature. And it doesn't have distributed support. |
|
I'm older than you 😂 I have an amazing career in .Net and with lucenenet. Lucene.NET 4 even in beta is used in countless production apps and websites but also know Lucene itself is a pretty low level beast. To implement it correctly depending on your requirements needs a lot of knowledge... i.e elasticsearch, azuresearch, solr, examine, etc. |
If you know and really use Lucene.net for a long time, you know how stable and powerful it is, and the amount of work paid by the developers behind it is amazing, and if you know the java version of lucene, you know that they also spent quite a long time due to the architecture adjustment and upgrade, and then began to release the new version quickly. Lucene.net 4.8 is doing just that, and once it is officially released, the progress of subsequent updates can be accelerated. |
No one denies the developers' efforts. But the world is essentially competitive. If you are behind, you will lose your share, and your users will be reduced, and the development progress will be slow, forming a vicious circle. |
You are right. The numbers don't lie. It is a vicious circle. Time for a quick math lesson. Take a seat. I worked 7 days this week for 12 hours each day. How many days are in a week? 7 days. Quick math - how many more days can I work in a week? That is right, 0. Lucene.NET has over 450,000 lines of code. It depends on functionality that doesn't exist in the BCL. All together, we are maintaining over 600,000 lines of code including those dependencies. This is not simple code. It goes deep into dozens of different technologies. Most of these things I have never had to deal with in over 20 years of programming. And the way things are done in Java simply don't translate well to .NET in a multitude of ways. All that we have left are the gaps at the edges of the application, but there are some very big gaps. Now, how many people regularly contribute to Lucene.NET? 4 or 5? Yeah, that is about right. But look closer. How many are contributing to the code? Currently, only 2. Most of the time it is 1. Quick math. What is 600,000 divided by 2? How about divided by 1? Do you see the problem now? Ever write a randomized test framework that is capable of repeating the random scenario so the test fails reliably with the same seed? Ever package an app with satellite assemblies representing over 750 languages? Ever implement the part of Unicode that .NET doesn't support? Ever copy a formatter out of the BCL just so you can add functionality to it because Microsoft didn't make it extensible and we need that feature? How about writing code that can put a ref struct or an interface into the same variable without any boxing? Probably not. How long do you think it would take you? Now how about if you multiply it by 30 things you have never worked with before? I have over 6,000 hours of my time invested in this project and its dependencies, and that is being conservative. I have worked with more diverse technologies in the past 3 years than most people will ever touch in their entire career. Paul Irwin returned recently to ensure the hard work that he invested in this project over a decade ago will be released. He is doing his part, and is invaluable to the project. What was your contribution? One complaint? Make that two. Do you seriously think a complaint is going to make us go faster? Do you think we have time to consider that you are being clobbered by your competition because you decided to sit back and wait instead of jump in and help? We don't spend one second thinking about this. We are completely overwhelmed by the amount of work there is left to do. It is beyond absurd to expect us to find extra time to include anything that isn't already in Lucene or isn't something directly related to closing a gap. The project isn't being delayed. It is going as fast as we can go. But there is only so much a couple of people working on the code can achieve when it comes to the release. The people working on the website and helping keep up with the revolving door of target frameworks are also important. They are at least doing their part to lighten the load. As for "losing shares", here are some real numbers:
So, we are not losing shares. It is you who is losing. You depend on a stable release. But you don't care enough to help get it done. The decision to wait instead of help was entirely yours, and you are losing because you made the wrong decision. Lucene.NET is not stable. The API has planned breaking changes that are not even started yet.
There is more than this, but my point is that we have breaking changes planned and we only get one shot at the version number 4.8.0. The next version number we will use is 4.8.0.1 because version 4.8.1 implies that we have ported Lucene 4.8.1. This doesn't leave much room for breaking APIs since nobody expects 4.8.0.1 to be a breaking change from 4.8.0. I have some ideas on how to work around this for obscure APIs, but for the primary ones, breaking binary changes are not acceptable for a stable release. We just fixed the last known concurrency bug in the Lucene.Net core module, which will be fixed in 4.8.0-beta00017. We also put in some low level native API calls to implement the fsync functionality that we were missing. But we recently found and ported a missing test that revealed that our ACID support is broken. About 1 power outage out of 200 during a commit will result in a corrupted index. Does that sound stable to you? This is database 101 kind of stuff, and we are not there yet. I am out of my depth with this problem and we could use someone who is familiar with the implementation of database fault tolerance and durability (or someone willing to learn) to step in and help. The vicious circle is that for companies who require a stable release, every year that passes by they are losing out by not helping or funding us so we can get more help. We need players on the field to pass the ball to or we cannot move any faster. The Lucene.NET project is not letting the Lucene.NET community down. But the Lucene.NET community is letting the Lucene.NET project down by not helping or providing funding to get it done. We have an entire plan in place to catch up with Lucene's release schedule. But we will never hit a moving target with this many lines of code and so few people. It would be absurd to even try it. So, now the ball is in your court. And by you, I am referring to the entire Lucene.NET community reading this. If you want a stable Lucene.NET release, you have 3 options:
The action you take next will demonstrate just how serious you are about wanting a stable Lucene.NET release. You have much more power to change the release date than I do. There is a push to try to get the production release out before the end of the year. I don't think it can be done with only 2 people working on the 4.8.0 blockers even if we take as many shortcuts as we can. But with a few more people working on the blockers and helping to drive ICU4N to a stable (but incomplete) state, I think it is plausible. Now for one last bit of math. Looking at my watch, it has been 4 hours since I started typing this. So did your complaint-as-a-contribution to Lucene.NET shorten or lengthen the time it will take us to release? That's right, you have just delayed the release by half a day by prompting me to reply. It took me away from other tasks I was doing. I am sure the Lucene.NET community as a whole really appreciates your contribution. But don't worry, you can submit a PR for a blocker to put us back on track again. |
No description provided.
The text was updated successfully, but these errors were encountered: