Is Project Active?

[RussCannon]

Is this project in active development? If so, is it serious work or, say, hobbyist type? Just wondering. I have not been pleased with the direction the KSH2020 project has been going, and it seems to have petered out anyway. I am just wondering where this project is going and what plans are in works.

Cheers,
Russ

[marcastel]

@RussCannon in short yes, the project is active but still in early organisation phases. And all helping hands are welcome :-)

The motivation, on the short term is not necessarily to bring in new features, but to make sure that the KornShell and its command interpreter remain part of all POSIX UNIX and Linux distributions. We want to ensure that it builds smoothly for packagers and to fix essential bugs.

As this is not a funded project, participation is mostly a best effort. We have not yet laid down a roadmap or anything of the sort, as we are still a very small team. The late worldwide situation has somewhat reorganised some of our priorities; but rest assured that this is NOT a dead initiative.

[hyenias]

100% agree with @marcastel. @RussCannon, I have elected to support this ksh-community's efforts. I have been learning so much from the team over the last couple of months. Yes, our ksh-community's version of the Korn Shell is in active development. Currently, I have been working on fixing rounding issues with floating points, see #6.

[saper]

still in early organisation phase

Is there some group or organisation behind it discussing things offline? or is just about pull requests we put in here...

What kind of organisation do you think is needed?

[marcastel]

@saper by organisation I essentially meant a dedicated group of people with a particular purpose : the KornShell. We need to learn how to work together, and organise ourselves to distribute work and be proactive to issues, ensure stability across platforms, further document, and so much more!

Obviously we need some sort of communication channel and more formal processes; the later are underway with, for instance, working instructions for the life cycle management of the repository. Please bare with us as this gets... humm... organised :-)

[McDutchie]

Well, I think that getting a proper bugfix release out there is now urgent, or the world will simply give up on ksh. Meanwhile there has not been a single commit to this repo in more than three months.

The last stable 2012 release is the least buggy one available, but still has many serious bugs. I've been following the work done on the abandoned ksh2020 branch, and while it certainly had fatal flaws, there was also a lot of good work done: actual fixes of longstanding bugs that can be backported to the stable release. As my understanding of the codebase was growing, I also figured out a number of fixes myself.

So, I got sick of waiting, and went to work. My 93u+m branch is now about three weeks old and under active development. It already has a number of significant bugfixes, all of which are properly documented with rationales in the git log. I got the horribly broken regression test suite into a working state, so everyone can verify the fixes don't cause breakage. I'm also testing everything against the regression test suite of my cross-platform shell library, modernish, which has been helping me discover many corner case bugs in all the POSIX shells for a number of years now.

Unlike the ksh2020 developers, I work incrementally. I don't believe in massive sweeping overhauls, I try not to change anything I don't understand, and I'm scrupulous about not causing regressions.

The 93u+m branch/fork is here: https://github.com/modernish/ksh

This is not intended as a hostile fork. When your community gets started, I would be more than happy to contribute my fixes back (as long as it doesn't require jumping through a bunch of hoops, at least; in that case feel free to take them yourself :-)). And I hope the need for my fork will eventually disappear as you get going.

Meanwhile, if anyone else is impatient, contributions to the 93u+m branch are more than welcome. There is no organisation. Let's just use github and get to work.

[kdudka]

@McDutchie Your fork looks indeed promising. Thank you very much for working on it!

[jelmd]

@McDutchie We are all very busy and founding an org which does not operate in a try and error manner/chaotic way takes some time. IMHO it is better to wait for some task done by other members (even if it takes a little bit longer), than to merge stuff without any policy/reviews according to its own taste. ksh2020 branch has shown, that this leads to unstable, unusable SW.

IMHO, there is no reason to "push" a new release just because there was none for the last 8 years. Indeed, I prefer to release when ready. Ready means in general tested by some people for a certain time (beta) w/o problems/regressions. Right now to get "Ready" for the next milestone (beta) the plan is to review all known vendor patches, aggregate and merge if appropriate. I think that it should be doable to get a release candidate this autumn and possibly a new release around x-mas.

Any contributor is welcome and forking ksh-community/ksh master, creating a branch for a fix/problem/feature one wanna work on and when ready creating a new pull request is the right thing to do - see also https://github.com/ksh-community/docs/blob/master/workflow-HowTo.md .

So thanks for your work - I'll definitely have a look at it when reviewing the patches from apple. But that takes time and at least I'm still very busy wrt. stuff around covid19 ... However, if you intent to become a long term contributor, you may join the the ksh-community, create new discussion teams, review PRs or ask to get invited to already existing teams, where we usually discuss all the organizational stuff, which is not source code/doc relevant.

[jghub]

@McDutchie
same from here: happy to see what you are doing. you are of course right, that ksh-community has had next to no world-visible activity (prior to the corona-mess there was quite some discussion internally, though) and that things should get moving faster ASAP. so AFAIAC, it is good that you have not waited any longer.

some of the other (few) guys formally constituting ksh-community so far will probably respond separately, but I feel the bottleneck so far was that @jelmd has had not enough spare time recently to really start with integration of existing patch/bug fixes (he is the guy in this small group who volunteered for this and probably is most competent to do it in my view -- personally, I'd rather avoid to take that over, e.g.). and everybody else was obviously also very busy with other things (pandemic etc....)

regarding how to proceed: you of course might also contemplate to simply join ksh-community if you are interested (whose name is not an accident...) and drive things forward from there rather than in a fork. I am sure no one would object since from everything I have seen in the ast repo (and now in your fork) your approach sure is compatible with ours (and you confirm that in your statement above). just my 2c.

[saper]

Meanwhile, if anyone else is impatient, contributions to the 93u+m branch are more than welcome. There is no organisation. Let's just use github and get to work.

I saw some fixes from @lkoutsofios in the master. I think we could also incorporate changes to the INIT system from the 93v (which in this case is better). @McDutchie Would you be interested in accepting some patches to INIT and maybe tagging a new "INIT" release out (2020-06-XX)?

I have also noticed that setting CCFLAGS in the package breaks shared library building because ast makefiles try to set the CCFLAGS on their own.

I am also working to get the debug=1 to build again

[jelmd]

@saper As git log says: @lkoutsofios patches <= 2020-02-14 are already included in this repo. The only 2 missing patches are the ones from 2020-02-24. IIRC a free?BSD maintainer commented, that those changes are a noop because it uses something different anyway. For completeness they'll get merged, but obviously there is not really a need to give it a high prio tag.

[McDutchie]

Thanks @saper, yes please, I would absolutely be very interested in accepting patches to INIT. I've currently got it building on macOS, FreeBSD, OpenBSD, and Linux. But it doesn't currently build on (at least) DragonflyBSD, NetBSD, Solaris, AIX, HP-UX, QNX – usually because iffe fails to detect some DLL feature so we get error: unknown type name 'Dllscan_t' and other errors like it.

As for the fixes from lkoutsofios in the ast master, there was just the one fix dated after ksh-community forked from ast: a FreeBSD build fix. So I've reverted the FreeBSD fixes I ported from FreeBSD ports (which are more drastic) and applied the changes from lkoutsofios instead. Thanks for the heads-up about that. That made it stop building on DragonflyBSD though, but maybe your patches to INIT will fix that :)

[McDutchie]

Thanks @jelmd and @jghub for your kind responses. I've found the pandemic affects people in different ways, and I do have some time on my hands now. That is not going to last, though, so I need to apply the fixes that I can apply soon, or it may not happen later. I just need that free rein for now.

But to be honest, that's not all. Since I'm here I might as well be frank and put my cards on the table, so you know where you have me. So please take what follows in that spirit…

It looks like we have different viewpoints on the cause of the ksh2020 project crash. I don't think that was because they weren't acting as a formal organisation; I think it was mainly because (1) they made an initial mistake in basing their work off the incomplete and badly broken 93v- beta, (2) because they were too eager to refactor All The Things™ and not eager enough to fix bugs, and (3) because krader1961 (who was mostly driving it in practice) had an apparent disdain for the POSIX shell language that clouded his judgment in some ways. Organisations are perfectly capable of having flaws like that, and often do. Conversely, there are also highly successful projects, such as bash, that are led and driven by one person.

As I've said, there was a lot of good work done on ksh2020 as well. I've already backported a number of fixes I could not have figured out on my own, and more are waiting. krader1961 might not have much love for the POSIX shell, but he and siteshwar are much better C programmers than I am. I think the way AT&T ejected them after your campaign against them is simply disgraceful unfortunate. They did invest years of their life into this project, and without their work I would never have been able to do what I am doing now. So I am grateful to them, and I wish the situation could have been fixed without killing their appetite to work on ksh forever.

So perhaps you can see why I am a little reluctant to join your organisation. It's not only that I don't have time for the slowness, politics, interpersonal stuff, etc. that are inevitable when founding and running an organisation; it's also that you've already shown how you deal with people you disagree with. Having said that, I wish you well, and I hope you'll have great success. I would be very happy to collaborate on a more informal basis, and I remain absolutely willing to contribute all my work back whenever you're ready to accept it.

[McDutchie]

A note regarding vendor patches, which @jelmd mentioned. I have looked into some of them myself and found them mostly useless, as many of them are OS-specific hacks that make ksh build on that system while breaking things for others. If @saper can fix the build system, that would be much better.

Apple's patches in particular are actively harmful. I've no idea what they really do as there are no comments or documentation, but after I fixed the regression test suite, I found that they cause several regression test failures in things like read -n3 and others I can't remember right now. What was needed to get ksh to build on macOS were just the compiler flags from Apple's Makefile. I've integrated those into the cc.darwin wrapper script. Between that and krader1961's fix for the invalid use of memccpy(3) in sfputr(), all the regression tests now pass cleanly on the Mac.

Some of the patches from the Red Hat source RPM (which siteshwar had a hand in, of course) look useful, I'm looking into them right now. Unfortunately there is no documentation on what they do. Does anyone know where to find that?

[jghub]

@McDutchie

Thanks @jelmd and @jghub for your kind responses. I've found the pandemic affects people in different ways, and I do have some time on my hands now. That is not going to last, though, so I need to apply the fixes that I can apply soon, or it may not happen later. I just need that free rein for now.

thanks for bothering to explain your POV. I have these comments:

It looks like we have different viewpoints on the cause of the ksh2020 project crash. I don't think that was because they weren't acting as a formal organisation; I think it was mainly because (1) they made an initial mistake in basing their work off the incomplete and badly broken 93v- beta,

agreed regarding bad starting point. regarding "formal organisation": they quite obviously wanted to sail under the ATT flag to make their work "official". in my view this was de facto a counterfeit operation. regarding quality/success of their work, agreed that one does not need a formal organisation to achieve good results. but you need to be qualified to do it.

(2) because they were too eager to refactor All The Things™ and not eager enough to fix bugs, and (3) because krader1961 (who was mostly driving it in practice) had an apparent disdain for the POSIX shell language that clouded his judgment in some ways. Organisations are perfectly capable of having flaws like that, and often do. Conversely, there are also highly successful projects, such as bash, that are led and driven by one person.

agreed regarding (2). partly agreed regarding (3) -- regarding krader you probably know my position.: his dislike of the shell language sure only partly explains what has happened in ksh2020.

organisation vs. no organisation seems a non-issue to me, anyway. ksh-community technically is a "github organisation" (I believe that's the official term), otherwise it is a modest attempt of currently ≈5 people to keep ksh93 alive and hopefully in the long run provide a trustworthy upstream for package maintainers.

As I've said, there was a lot of good work done on ksh2020 as well. I've already backported a number of fixes I could not have figured out on my own, and more are waiting. krader1961 might not have much love for the POSIX shell, but he and siteshwar are much better C programmers

I think, whatever working and solid fixes are in the ksh2020 code that apply to ksh93u+ of course should be backported in due time.

than I am. I think the way AT&T ejected them after your campaign against them is simply disgraceful. They did invest years of their life into this project, and without their work I would

well, here I have to clearly object against your assessment:

1. there never was a "campaign" (which by definition would imply a concerted effort of a group of people to achieve a certain goal by probably questionable means). I am rather sure you have followed the story/history so I am a bit astonished that you arrive at this assessment. for newcomers to this repo, here's again the reference to the issue which ultimately triggered the decision of the ATT team to revoke commit rights for krader/siteshwar: segfault with extended globs att/ast#1464. so the truth of it is, that at a certain point, I personally decided to make the ATT team aware and to mince no words what I believed (and still believe) to be the correct decision. that they acted as promptly as they did, a) surprised me and b) shows how serious they took it (correctly, in my. view). you also seem to miss the fact that @gordonwoodhull very politely and diplomatically proposed that ksh2020 just go on in a fork, which would have been perfectly possible. that they (krader and siteshwar) dropped the project like a hot potato the moment they could no longer pretend that their work is the "official" ATT-sanctioned future of ksh is very telling in my view.

EDIT ON REREADING
2. I forgot to address your use of the very strong word "disgraceful" for describing the action of the ATT team. I believe this is really unfair. @gordonwoodhull in my view did an admirable job in providing them an "honourable" way out of the ATT repo and proposing they continue their work in a fork. you somehow seem to ignore that ksh2020 really was on a crassly divergent course from what the AST repo wants to represent. revoking commit rights to that repo (nothing more) sure was an overdue move.

never have been able to do what I am doing now. So I am grateful to them, and I wish the situation could have been fixed without killing their appetite to work on ksh forever.

nobody has "killed" anything. the decision to simply abandon ksh2020 instead of continuing the work in a separate project was entirely theirs.

see above. I only can suggest that you reread through the att/ast issue history what has and what has not been said or proposed by different people at the time. I maintain ksh2020 was heading in a very wrong direction and that mostly, nearly exclusively, was due to krader's approach. that does say nothing about his C capabilities and nothing against salvaging sensible fixes from the ksh2020 code.

So perhaps you can see why I am a little reluctant to join your organisation. It's not only that I

given your assessment, I can. but I don't agree with the assessment :).

don't have time for the slowness, politics, interpersonal stuff, etc. that are inevitable when

in fact, it very well could be that your aversions in this regard are not very different from mine. so far and in my view this is mostly a non-issue in the context of ksh-community.

founding and running an organisation; it's also that you've already shown how you deal with
people you disagree with.

sorry, this is a real non-sequitur to me (to put it mildly). I really believe the different conversations/discussions in the ATT/AST issue history tell a different story. it is now beating a dead horse anyway, but if there was a guy exhibiting agressive and intolerant behaviour there it was krader and krader alone. but whatever...

Having said that, I wish you well, and I hope you'll have great success. I
would be very happy to collaborate on a more informal basis, and I remain absolutely willing to contribute all my work back whenever you're ready to accept it.

amen. :).

[McDutchie]

Thank you for your reply. Your point that the ksh2020 maintainers could have continued in their own fork, just as you are intending to do here, is well taken. I accept that "disgraceful" was too strong a qualification on my part, so I will edit my reply. At the same time I can fully understand why they didn't accept being demoted by AT&T, so they would have to work harder at gaining visibility, etc. For us it's different as we never had AT&T blessing to begin with.

As for "organisation vs. no organisation" being a non-issue, I will just say that there is one repo with significant current activity, and this one ain't it… ;-)

I don't agree that krader1961 was the only person exhibiting aggressive and intolerant behaviour; I distinctly remember being put off by some of his opponents' behaviour as well. But, as you correctly point out, it's a dead horse so I'm happy to leave it and go back to fixing bugs. I just wanted to make my current position clear so we avoid future surprises/drama.

[jghub]

Thank you for your reply. Your point that the ksh2020 maintainers could have continued in their own fork, just as you are intending to do here, is well taken. I accept that "disgraceful" was too strong a qualification on my part, so I will edit my reply. At the same time I can fully understand

I appreciate this edit.

why they didn't accept being demoted by AT&T, so they would have to work harder at gaining visibility, etc. For us it's different as we never had AT&T blessing to begin with.

my best understanding is that they never had the blessing of the ATT/AST team for what was happening in ksh2020 and that the initial understanding was a different one (there were remarks to that extend by @lkoutsofios somewhere IIRC) -- which in turn explains and justifies the taken actions.

As for "organisation vs. no organisation" being a non-issue, I will just say that there is one repo with significant current activity, and this one ain't it… ;-)

non-issue in the sense that ksh-community is just an attempt at providing a context to do work at ksh but not some big self-paralysing org succumbing to parkinson's law. so if at the end of the day all good work will be done in your fork, it would tell something about capabilities and commitments of the guys involved, not about the "framework" :). personally, I will be content if there finally will be a trustworthy maintained upstream again. here or elsewhere. oh, and by the by, in case you find it useful and have not yet noted: here https://github.com/ksh-community/shbench are some benchmarks and a wrapper to drive them (triggered by my lengthy attempt to convince krader that there is a serious performance issue in ksh2020). maybe it helps to run those (and possibly augment them) by and then to prevent a "performance regression" creep into future ksh93u+ again.

I don't agree that krader1961 was the only person exhibiting aggressive and intolerant behaviour; I distinctly remember being put off by some of his opponents' behaviour as well. But, as you correctly point out, it's a dead horse so I'm happy to leave it and go back to fixing bugs. I just wanted to make my current position clear so we avoid future surprises/drama.

understood. I, too, am totally against drama :).

[jelmd]

@McDutchie in add. to @jghub comments: Yes, we all need time slots to do some work, and for ksh usually one needs bigger ones. So everybody is in the same position as you, with the difference that you have sufficient slots right now. And that's the advantage of the github workflow: I lets you work on your own pace and provides an easy way to submit your work as PRs to other repos, if you want to (and there is no requirement to join an org to be able to do this). One advantage of the org is, that one can use the "infrastructure" alias github to draw away non source code/documentation discussions, which annoys people/developers so much and draws away so many resources (e.g. time) instead of getting the work done. And having an org in place, which has some guidelines/rules of thumb helps a lot to avoid wasting valuable time with discussing the same things again and again (look at this thread and you get a slight idea of what I mean, why it takes so much time to workout such guidelines, get an org work).

So what you are doing is perfectly right and definitely helps. Actually you are doing, what I do for years: see a bug and depending on its importance I create an issue in the upstream repo, possibly fork, push in my fixes and submit it as PR to the upstream. After this I'm away, i.e. I do not really care whether it get merged or not, because I've already a fixed version, which is sufficient for my needs ;-) - I guess a lot of admins work these days like this. And here the org (upstream) slips in: there are hopefully some experts, which review the PR when it is time to and make it available for everybody. So in your case it frees you from the need to maintain your fork forever, which helps you probably a lot, because you already said, that you do not have the time for it (just some time slots right now).

So IMHO it is good, when you collect, merge, review, test BSD related patches - the less work for the upstream to integrate them (because most issues, tastes, policy stuff hopefully has already been discussed/fixed by the "BSD team") unless they are monster patches ...

[McDutchie]

@jelmd: I'm not sure you actually understand what I am doing. Have you taken a look at the 93u+m git log and NEWS file? I am way, way ahead of something like testing BSD-related patches... there is some proper work going on: the regression test suite is in a working state now, a considerable number of bugs have already been fixed (including some that have never been fixed before in any fork/branch), and I've already got four pull requests merged as well. For more plans, see the TODO file. You're more than welcome to take everything when you're ready to get started, but I'm making too much progress to send pull requests for individual patches.

[McDutchie]

@jghub: Thanks for that pointer to shbench, that is a great little thing to play with! And, so far, so good: the reported performance difference between 93u+ and 93u+m is in the range of rounding errors. Sometimes a test is very slightly faster on one, sometimes on the other, without any code changes. I will definitely include this in my future testing.

[McDutchie]

FYI: https://groups.google.com/d/msgid/korn-shell/2c0d6f5d-1289-9888-a187-46eb2d68d064%40inlv.org

[jghub]

@McDutchie:

@jghub: Thanks for that pointer to shbench, that is a great little thing to play with! And, so far, so good: the reported performance difference between 93u+ and 93u+m is in the range of rounding errors. Sometimes a test is very slightly faster on one, sometimes on the other, without any code changes. I will definitely include this in my future testing.

glad to hear that you find it useful (and it now enforces LC_NUMERIC=C -- thanks for spotting this). suggestions for improvement (or better/further benchmarks) are always welcome.

some sort of benchmarking (be it by shbench or some other tool) seems sensible in my view to avoid what happened in ksh2020: performance erosion getting unnoticed for too long (and than difficult to track down and fix etc.)

[jelmd]

@McDutchie

@jelmd: I'm not sure you actually understand what I am doing. Have you taken a look at the 93u+m git log and NEWS file?

No, unfortunately haven't had the time to dig in your repo, just read your comments here. However, took your hint to skim over it some minutes ago - looks promising (and like a lot of work)!

I am way, way ahead of something like testing BSD-related patches... there is some proper work going on: the regression test suite is in a working state now, a considerable number of bugs have already been fixed (including some that have never been fixed before in any fork/branch), and I've already got four pull requests merged as well.

Very good! As said, every contribution to ksh is welcome, it does not really matter, in which repo it happens. It makes perfect sense, if you do not wanna wait for any reviews or other things slowing you down, use your available time slots with max. efficiency. A little bit risky wrt. QA, but in your case probably not really a problem.

For more plans, see the TODO file.

Just skimmed over it. Looks also promising, but of course, there are some things, I would not agree with. E.g. Fix build system/Solaris - I'm using it for 20+ years and it still works (actually after 25+ years Solaris is still my primary work env). Or wrt. nmake: The project is about ksh, not about the tool chain used to build it. So I outsourced as many as I could (otherwise it would also mean, we would maintain nmake - there is no intention right now to do that). Nothing prevents anybody to continue to use nmake for building. But nmake should have its own repo like other tools (e.g. make, cmake, gcc, sed, ...). pty.sh is of course a point, which makes sense, if one wanna keep the current test system (right now, I'm undecided, but is perhaps the way with least resistance).

You're more than welcome to take everything when you're ready to get started, but I'm making too much progress to send pull requests for individual patches.

Yes, thanks a lot. Understand the "making too much progress" situation. Any PR+review would slow you down (and advantage for me: I can relax, do not get add. pressure wrt. reviewing it immediately ;-) ). When we finally pull them in, you get at least a second? review, even if it is a little bit late or may appear useless (because of your work in the Austin group and being a perfectionist I think, in contrast to krader, you know what you are doing and the outcome is much much more reliable than in the ksh2020 experiment).

So IMHO all good. :-)