Project Lifecycle: Reevaluate non-production goals of projects

[anvega]

The PQCA projects have achieved widespread adoption and reference implementation status for several of its projects, most notably liboqs and the oqs-provider. While this success is commendable, it calls for a reevaluation of our quality assurance processes and project goals to ensure certain expectations of security and reliability are met.

liboqs and oqs-provider have become "de facto" reference in the post-quantum cryptography space.

The current state of these libraries has not undergone the level of scrutiny typically required for high-security applications.
An audit is currently in progress for liboqs, and a community self-assessment is underway for oqs-provider.

It would be great start conversations on restating and potentially expanding the goals for liboqs and oqs-provider. This should include: Clearly defined security levels and use-case scenarios, long-term maintenance and update strategies, standardized testing requirements, public roadmaps.

In turn this would increase confidence and dependability for high-security use cases.

[baentsch]

I whole-heartedly support this suggestion -- but I think by now that no LF body like TAC or TSC is the right place to discuss this (see the lack of contribution or closure at open-quantum-safe/tsc#1). Here they even label this "enhancement" and "documentation"... In a project developing security software, this should be a priority, not an enhancement or a documentation issue, but one requiring concrete technical goals and contributions as per your suggestion.

This would require equally concrete technical and time commitment, though; and I'm not seeing much of that (hosting and/or participating in 1h meetings every now and then doesn't really count in my eyes :). Your concrete proposal & support with the self-assessment (next to the openssh upgrade, kudos to @brian-jarvis-aws 's team) has been the most glaring exception; cncf/tag-security#1333 was the first time I got the impression that LF membership can advance the software and not just make working and contributing more difficult.

My suggestion thus is to cooperate on the concrete issue open-quantum-safe/oqs-provider#483 , @anvega ; otherwise, there's also a discussion on roadmap that may benefit from your perspective.

[anvega]

With so many consumers relying on it, and anticipating the number will grow, if these repos were to go away, I’m sure there would be disruption to production environments. However, it’s important to remember that developers are not big babies and aware of the risks associated with using open source software—there’s no implied contract holding OQS accountable if something breaks.

My main point is that the current disclaimer might discourage trying to get buy in or advocate for investment from their organization in contributing to the project.

The balance we should aim for is in transparency around the engineering governance of the project, including the existing controls—or lack thereof. By clearly demonstrating whatever levels of testing, reliability, and security scrutiny the project undergoes, alongside a roadmap outlining future directions, prospective users can make informed decisions on whether or not to adopt the project.

Instead of rephrasing the disclaimer, it would be more beneficial to remove it alltogether and overtime replace it with badges or mentions that highlight the audit status, what aspects are tested, what isn’t, and any other relevant information. This would provide a clearer picture of the project’s reliability and security, helping users gauge their risk accordingly.

[baentsch]

At the core, I agree with your goal @anvega : Make this software reliable. What I'm not sure about is the order of action you're proposing: The below seems to imply you first want to remove the OQS disclaimer and then work on improving matters: Right or wrong?

If the former, I'm afraid I've got to dissect your arguments a bit:

if these repos were to go away, I’m sure there would be disruption to production environments

Two thoughts on that statement

1. Don't you think conscientious product owners work off their own forks (to make sure they use something that cannot go away -- or changes in ways they didn't anticipate)?
2. I have never been suggesting to remove these repos -- I only argued for honesty when publishing their status.

Either way, I see your point as moot.

it’s important to remember that developers are not big babies

Agreed. My concern is not related to developers, though, but to (end) users of the software: Those can (currently) easily hold accountable commercial entities incorporating OQS into their products in case of breakage due to bad software quality, bugs, etc. as there (currently -- with the disclaimer in place) is no plausible deniability for such companies as to the purpose of the software (evaluation at best).

If the OQS "fitness for purpose" disclaimer simply gets removed as IBM and LinuxFoundation now argue for, users have the same recourse to ruthless software corporations cutting corners to make a quick buck: None. IMO OSS security software should aim for something better.

Thus, also that argument of yours does not hold water: Developers indeed (mostly :) know what they do -- but this is typically driven by their employer's commercial interests; in this case, just possibly, to use OSS software re-labelled to "productive readiness" instead of investing to really achieve & earn that status?

prospective users can make informed decisions on whether or not to adopt the project.

Again, our understanding of the term "user" is different: You mean people/companies integrating the software PQCA creates; I mean "Average Joe", a person not reading software contracts. LF's @hartm once put it nicely: People only look at the LinuxFoundation lifecycle readiness statement to judge the reliability of the software -- and that definition exactly is so weak such as to nearly mean nothing/allowing practically all software to be considered "production ready" (I've been trying for months to get that improved, see many open issues on the topic).

As mentioned before, such "(re)lax(ed) definition of quality" is basically no problem for lots of software, e.g. stuff that people came to expect to be a hoax anyway (blockchain, anyone? :), but it is (in hopefully not just my eyes) different for security software, stuff meant to (eventually, hopefully, if my dream comes true to) secure "Average Joe's" online banking.

And then something personal as PQCA made this personal:

> ACTION Andres to work with Michael to get consensus

Please accept that I treat (particularly security) software I'm involved in to a high standard: I aim for honesty as to what software my name is attached to can/cannot or should/should not (be relied on to) do.

My main point is that the current disclaimer might discourage trying to get buy in or advocate for investment from their organization in contributing to the project.

This is a big "might" -- it indeed may be so for organizations subscribing to IBM's @planetf1 's adage of "OSS is basically scratch your own itch": Entities only willing to invest in self-serving OSS features/not willing to invest for the common good, e.g., making the software reliable for everyone, IMO let potential community support go to waste that'd also benefit themselves.

My main point is that if the project has (retains!) trust with honest statements about its status and works (also by corporate contributions) towards improving that status, it can become more and more interesting to invest in by companies otherwise investing in their in-house software, i.e., software they can put trust in, e.g., for the definition of reliability and utility not simply be changed from one day to the next...

Finally, allow me to ask why apparently our software is productively used? Might it be because the OSS community has invested in it? Agreed, many companies hold back (with notable exceptions like AWS, Cisco or Entrust), but isn't it more likely for this to be not because of this disclaimer but simply because of a lack of capabilities or even more simply, selfishness?

Instead of rephrasing the disclaimer, it would be more beneficial to remove it alltogether

That statement I've got to question most: Beneficial for whom? All "OQS leechers", their accountants and lawyers, most likely. The general public? Definitely No.

The general public simply would not understand

badges or mentions that highlight the audit status, what aspects are tested, what isn’t, and any other relevant information

The OQS disclaimer in turn is clearly understandable by everyone.

I am painfully aware that PQCA and OQS have put IBM in control (chair/co-chair positions in the project-controlling bodies) and thus can probably force this through (IBM's @maximilien already assigned this issue to himself), but I personally will keep arguing for first

• properly (that is, openly) documenting the status of the project and its governance
• fixing all problems thus detected (beyond whitewash "badges")
• getting it a serious third party review or in general, getting reputable, independent statements of support & use testifying to the "fitness for purpose" of the software

before considering changing the disclaimer. Allow me to even propose a litmus test: If the non-LF project OpenSSL adopts OQS as its (trusted) source for PQC I think we can consider this case closed.

Until then, finally, a little reminder: PQCA itself posits OQS as follows:

Its charter states clearly "software for evaluation and prototyping".

So what you're asking for in this issue thus seems like nothing more and nothing less than a "silent" change of purpose of PQCA.

In conclusion, as per the initial statement, I'd be all for such change -- if done openly, honestly, with the required work, investment and contributions by all parties first and not by quietly "removing a disclaimer":

I already invested a lot in this software, and I think it's fair to now expect others to do the same: The whole purpose of OQS joining LF was to bring the software to a new (also quality) level.

When that's done, we can also change disclaimers to be more in line with "standard software disclaimers".

[anvega]

I think the collective interest is the same. We agree on the goal.

It was just floating an idea so thanks for holding it to scrutinity. Good ideas stand up to challenges, while bad ideas should be discarded. As the saying goes “Truth springs from argument among friends".

I also understand your concerns about the risks of removing or relaxing the disclaimer without a solid plan. I appreciate Douglas’ proposed steps leading to 1.0 release and related work as a sequence to demonstrate more maturity and your points on building on documentation, problem resolution, third-party reviews, and securing independent statements of support, as you've advocated. We can then reassess the situation.

I’m particularly interested in evolving the disclaimer to reflect the importance of conscientious product owners using their own forks of the software to avoid relying on potentially unstable or deprecated repositories.

[baentsch]

It was just floating an idea so thanks for holding it to scrutinity. Good ideas stand up to challenges, while bad ideas should be discarded. As the saying goes “Truth springs from argument among friends".

Thanks very much for taking it that way, @anvega : This is definitely the first time I hear that saying within the context of LF and maybe also the first time a) someone from the LF universe discusses stuff in this openness (and doesn't push it to "later" or another meeting if my opinion is not to their liking) and b) (apparently :) agrees with some of my (surely debatable) points. I'd be grateful if you could impart that to the non-contributors (in GH code and GH discussion) in the TAC (if you're willing to take the time for such meetings).

Side note: For the very same reason (insufficient technical discussions in the TAC) I'm not subscribed to the project so don't get notified on messages, so please @mention me if you'd like input. If you don't, just write "Michael" (like here).

I’m particularly interested in evolving the disclaimer to reflect the importance of conscientious product owners using their own forks of the software to avoid relying on potentially unstable or deprecated repositories.

This (particularly the latter part) seems to me a bit "over the top" for two key reasons:

1. It seems to imply/communicate that you/we think/we consider OQS unstable or might deprecate it. That (I hope) is not the case. What is true is that --particularly liboqs-- will most likely have to go through some heavy refactoring to allow for a long-term maintainable "product/experimentation" split that Douglas has in mind -- and that surely justifies for the "invisible productive users" to have a separate fork.
2. Doing products off private forks seems common sense/practice and wouldn't (shouldn't?) need explicit mention as you yourself rightly said

it’s important to remember that developers are not big babies