(Re)Consider adding “pre-release” browser-version value for cases where we don’t yet have a BCD-known version number

[sideshowbarker]

Concrete proposal

Here’s a concrete proposal for marking up data for pre-release browser versions; let’s start doing one of the following:

1. Start using Safari Technology Preview release numbers in addition to Safari major-release numbers; or
2. Start using some generic value “pre-release version” value — for example, dev or pre-release or whatever; or
3. Start using values that correspond to how the browser projects actually label there particular pre-release versions; for example, nightly, canary, and TP (technology preview).

Problem description

The concrete proposal above is intended to address a problem that we’ve had discussion about before. So maybe this is a basically duplicate of an existing issue. If so, I can just take all of the following and add it as comment to that other issue later.

Here’s a detailed explanation of the actual problem I am hoping we can find a way to solve together:

We’re aware that we have a significant number of cases when we know support for a certain feature has been implemented in a particular browser, but only in a pre-release version. And in those cases, if we do not yet have that version number in any https://github.com/mdn/browser-compat-data/tree/master/browsers file, then we have no way of updating the related BCD data to indicate the browser in question has support for that feature.

I assert that’s a deficiency which we should try to find some way to correct — because the data point “this browser has implemented support for this feature” is a useful fact, regardless of what the actual browser version is.

In particular, I think that “this browser has implemented support for this feature” data is potentially useful to downstream consumers of the BCD data.

And more particularly, I can state that it’s absolutely useful to the downstream-consumer tooling that I maintain myself at https://github.com/w3c/mdn-spec-links and that’s used for adding MDN annotations to all W3C and WHATWG specs.

So I’d like to request that we re-consider coming up with some way to mark our data for such cases.

The special problem with Safari data

It’s worth noting here that this is a bigger problem for the Safari data than it is for the Chrome or Firefox data. The reason is that while major numbered versions of Firefox and Chrome are released every six or seven weeks (that is, 7 or 8 releases each year) — major numbered versions of Safari are released only every six months (that is, only twice a year; typically with one major release in March and then another in September).

What that ends up meaning for us is that we have to wait a very long time between Safari releases before we can update our data — even in cases where we know that something has already shipped in a Safari Technology Preview release.

Safari gets actually get updates often; let’s stop ignoring the existing feature data already available

We actually do have numbered versions of Safari shipping quite often; in fact, a new Safari Technology Preview release seems to be shipped on average every four weeks at least — even more often than Firefox or Chrome releases.

So we actually have the data for Safari — that is, we have data for numbered versions of Safari that are being released even more often than numbered versions of Chrome and Firefox.

However, what we’re basically doing is, we’re completely ignoring that data — the data we have for features on those numbered versions of Safari — because we are choosing arbitrarily to limit ourselves to only adding data for features in the two versions of Safari each year that Apple chooses to give major-release version numbers to.

Recognizing that BCD data has use cases beyond just tables in MDN and caniuse.com

In past discussions, the response I’ve received about why we choose to constrain our Safari feature data to only major Safari versions is, we do things that way because those two-releases-a-year Safari versions are the only Safari releases that ship to end users, and so those are the only versions that developers need to care about.

That point of view would make sense if the only imaginable use of the BCD data were for developers wanting to know which shipped-to-end-users versions of browsers have particular features.

But that is of course not the only imaginable use of BCD data. There are lots of other ways we can easily imagine BCD data being used that aren’t limited just to identifying shipped-to-end-users browser versions.

In fact I have an existence proof of such a usage of BCD data: https://github.com/w3c/mdn-spec-links — which is completely agnostic to whether a feature is implemented in a shipped-to-end-users browser version, or not. All that matters for that use of the BCD data is whether the browser has implemented a feature at all, in any version at all.

Let’s stop artificially constraining the possible use cases for BCD data

What we are storing in this repo is data. And we should not be making arbitrary assumptions about how that data is used or by whom. The Browser Compatibility tables in MDN and tables at caniuse.com are just one kind of possible use of the data.

So we rightly should not have policies that artificially constrain the data to only being useful for that one kind of case, and that arbitrarily obstruct the utility of the data for many other possible use cases.

[sideshowbarker]

FYI to anyone who already read the initial issue description I wrote for this (~8 hours before this comment was posted) — I subsequently updated the issue description quite extensively, so please re-read it in its current state

[ddbeck]

@sideshowbarker Thank you for opening this issue. I'm glad to rethink this (though I don't know we'll end up in a different place). I agree that our handing of Safari TP data is troublesome, at best. I plan to give this some more thought, but I thought I would put a comment and some questions in front of you early:

1. What's the story for collapsing pre-release versions in to actual release version numbers? For example, if we start logging TP versions in Safari entries, how will we be able to tell if features that shipped in TP X, Y, and Z made the cut for Safari W and translate those numbers accordingly?* How does this work for browsers like Firefox, which don't version pre-releases independently (i.e., when "nightly" is a moving target)?
2. Who do you foresee doing this work? What would be the process for converting version numbers?

* If there's not a good answer to this question, I have an idea I want to explore more: tracking Safari TP as its own browser. This wouldn't help with "Nightly"/"Canary" values, but it would help with Safari, at least.

What we are storing in this repo is data. And we should not be making arbitrary assumptions about how that data is used or by whom. The Browser Compatibility tables in MDN and tables at caniuse.com are just one kind of possible use of the data.

So we rightly should not have policies that artificially constrain the data to only being useful for that one kind of case, and that arbitrarily obstruct the utility of the data for many other possible use cases.

I appreciate you spelling this out too. I do like the idea of creating a neutral data set that can be used for lots of applications. That said, I don't think MDN and caniuse's use cases are catered to for arbitrary reasons. Rather, they're the consumers of BCD that reach the greatest number of Web developers. I am happy to work on things that help other consumers, but I think we need to have a strong case that we'll be serving new or expanded audiences in doing so. I've been bitten before on making schema changes that went unused and, as a consequence, I'd like to avoid highly speculative efforts (not that this is one of those efforts—just articulating a general principle here).

[sideshowbarker]

That said, I don't think MDN and caniuse's use cases are catered to for arbitrary reasons. Rather, they're the consumers of BCD that reach the greatest number of Web developers.

Agreed. That was just a poor choice of words on my part. I think anybody would agree it’s clear that the MDN/caniuse use case continues to be by far the most important use of the data.

What I had been trying for, in that already-long comment of mine, was just to use a kind of shorthand term in place of this much-longer explanation:

We don’t have an external constraints being imposed on us that limit us to only tracking data for features that have shipped in major browser releases. Instead, we are imposing a choice ourselves to limit the browser-version data to major browser releases.

So, what I’m really trying to say is that I hope we are not imposing that choice on ourselves just due to lack of awareness about use cases beyond the MDN/caniuse case — that is, not based an assumption that the only use case that matters in practice is the MDN/caniuse case.

[sideshowbarker]

I am happy to work on things that help other consumers, but I think we need to have a strong case that we'll be serving new or expanded audiences in doing so. I've been bitten before on making schema changes that went unused and, as a consequence, I'd like to avoid highly speculative efforts (not that this is one of those efforts—just articulating a general principle here).

OK, that’s a great point — it makes me realize that what would really help here is concrete example.

So https://streams.spec.whatwg.org/#ws-class is an example that I think is highly relevant here. That’s the section of the WHATWG Streams standard that defines WritableStream. And if you expand the MDN annotation there, you see this:

So, very noticeably that annotation says (in red), In only one current engine

In light of the information Mike Conca provided in #6873 about the fact that WritableStream is actually already implemented not only in Blink, but also in Gecko and WebKit too, that annotation is very misleadingly inaccurate. What it should show instead is something like what’s shown in the annotation at https://streams.spec.whatwg.org/#rs-tee, which looks like this:

…that is, it rightly should say (in green), In all current engines

But in order to have the https://github.com/w3c/mdn-spec-links tooling generate an annotation for WritableStream that says In all current engines, I need a source of data I can consume which actually records which browsers it’s in.

And BCD is currently the sole source of the data the https://github.com/w3c/mdn-spec-links tooling uses to generate those annotations — so if the BCD data claims it’s not in a particular browser, then that’s all I have to go on.

And so if instead we could refine BCD policies to provide a way to record data about features that have been implemented but not necessarily shipped in a major browser version yet, then the spec annotations generated by the mdn-spec-links tooling would be able to accurately reflect the implementation status for those cases (rather than providing the wrong information for those cases, as is the current state).

[sideshowbarker]

1. What's the story for collapsing pre-release versions in to actual release version numbers?

For Safari, I guess I had just imagined that if we had set a pre-release version number for some feature to TPNNN, and then that feature ends up later shipping in Safari major version ZZ, then at that time, we just replace the TPNNN version number with the ZZ version number.

So I guess in that sense the TP version numbers would be, for our purposes, somewhat ephemeral and disposable. I say that based on the assumption that once version ZZ of Safari ships, no web developers are going to care any longer what TP version the feature first landed in. (And assuming no other downstream consumers of the data are going to care. Certainly I can say that for the mdn-spec-links use case, what TP version it historically shipped in would no longer be useful data.)

For example, if we start logging TP versions in Safari entries, how will we be able to tell if features that shipped in TP X, Y, and Z made the cut for Safari W and translate those numbers accordingly?

I guess for now we’d need to do that by trusting what’s recorded in the Safari release notes. But I think a more robust solution that would help significantly would be if we can manage to get somebody from the Safari/WebKit project more actively involved here so that they could help with confirming feature support case-by-case in such circumstances.

How does this work for browsers like Firefox, which don't version pre-releases independently (i.e., when "nightly" is a moving target)?

That’s a good question. But I don’t have a great answer for that at this point.

2. Who do you foresee doing this work? What would be the process for converting version numbers?

For the Safari case, unless/until we manage to get somebody from the Safari/WebKit project more directly involved, I would be willing to do that work myself, for any features that submit patches for.

For the Firefox and Chrome cases, I think much less new work would be required than for the Safari case. I think that for the most part, we are already doing fine for keeping up to date with Firefox and Chrome feature data; the 6-week release cycle is short enough that we don’t end up with nearly the same scale of problem that the 6-month Safari major-release cycle creates for us.

[jpmedley]

The single biggest and most glaring problem I see with this proposal is that there is no web-standard definition for how features are handled in non-stable browsers. "Supported" means exactly the same thing in every release browser. In Chrome, a number of situations would lead to frustration if our pre-release data were available to web developers generally. (We distinguish between those who are build for production and those who are experimenting. That's a different discussion.)

Features in Canary don't always make it to beta: The problem is churn. Keeping up with what is and is not graduating to beta is difficult for me, and I get paid to track it. When we were preparing Chrome 86 for beta, there was one feature that was listed or unlisted for beta three times within a single week. Teams also have the ability, with special permission from management, to merge a feature in an origin trial backwards. For example, a feature slated to start an origin trial in 87 could get permission to be merged to 86 before 86 rolls to beta. For some reason this has been happening a lot lately.

Documentation may not exist: For pre-production features, there's unlikely to be any documentation about a pre-release feature. Chrome is trying to get better about documenting features it adds to the platform. To whatever extent we succeed, we don't currently add things to MDN before they're in production and we're unlikely to do so before beta.

Many of the features are not considered production-ready: Contributors, I am sure, would fail to notice the distinction between features that will be enabled by default in a particular version and features that are behind a flag or in an origin trial. One thing Chrome never does (and we strongly discourage anyone who wants to) is to telling users to flip a runtime flag to use a feature. Flagged features are intended for experimentation and testing, not for production use. This is why myself and others within Chrome have long opposed the inclusion of Chrome flags in BCD.

Origin trials aren't even for stable specs. They're part of our incubation process. API surfaces can change multiple times between initial implementation and production. We developed origin trials to prevent what our engineers called "burn-in": experimental features becoming so widely adopted that we are unable to change their designs or even remove them if they prove to be sufficiently bad. If an origin trial graduates to stable, we deliberately shut off the OT version two weeks before stable. If a feature design makes it to stable unchanged from its origin trial, that says more about the design of the feature than it does about the origin trial.

On thing that I think is easy to forget for those of us in the browser business is that web developers' interest in new features is often more academic than practical. If I want to use a feature that's only in one browser, knowing that it's in everyone else's pre-release may or may not be helpful. If I can't say for sure that Firefox's version will arrive before my app has to go into production, my only two choices are use a polyfill or don't use the new feature.

[sideshowbarker]

@jpmedley Thanks extremely much — your comment has a ton of great info that I find very useful to know (and I reckon others involved here do too) even beyond the context of this particular issue. So I’m glad we’ve got it recorded here.

As far as the context of this issue: after reflecting on the things you’ve pointed out, I am thinking about an alternate proposal:

We could add a new, optional boolean flag named implemented, with an accompanying optional notes field. I’d expect that the notes field would usually just contain a link to a changeset/commit URL.

That way, we could have a record in BCD that support for the feature had landed in particular browser’s codebase. And as I’ve attempted to point out in my other comments here, I think that’s good data for us to be including in BCD.

And along with the general utility of having that data recorded in BCD, it’d solve the problem I described in #6896 (comment)

More specifically, I can say that for the WHATWG/W3C MDN spec-annotations case (https://github.com/w3c/mdn-spec-links stuff), it’s not absolutely necessary to always have a version_added value; instead all that’s necessary is to know whether or not there is an implementation of the feature in a particular browser engine. So an implemented flag would provide that.

And to be clear: If we added an optional implemented flag, I wouldn’t expect other contributors to feel any obligation to go out of their way to add it for any features (for example, features in the states described in #6896 (comment)). It’s possible I might be the only one who ends up using it regularly.

But regardless, I wouldn’t anticipate that adding the flag would end up being a maintenance burden for other contributors.

Nor would I expect it to be problem for MDN or any downstream consumers — because any consumer of the data can just ignore the implemented flag and only use the existing version_added data; in other words, they could just keep doing what they’re already doing now — without needing to change any of their existing code or processes.

Anyway, I’m anxious to hear your thoughts on that idea.

[sideshowbarker]

We could add a new, optional boolean flag named implemented, with an accompanying optional notes field. I’d expect that the notes field would usually just contain a link to a changeset/commit URL.

Actually, since we’d really have no need to ever mark anything implemented:false, we’d not need a boolean, and so instead in place of the optional notes field, the value of the flag itself could just a string that provides the changeset/commit URL.

And given that, we could name the key/property implementation_url (or something_url) — similar to our existing mdn_url and spec_url properties — and the value would be a changeset/commit URL.

And that implementation_url data would have general value beyond just being an indicator of whether the browser had implemented the feature. It’d preserve, in BCD, the URL for original changeset/commit which landed the feature — preserve it in BCD as machine-readable data. I think it’s easy to imagine that being data that others might want to mine from BCD.

In practice for now, because of the special problem we have for the case of Safari, I can imagine that it’d still mostly end up being used for taking the URLs from rNNNNN links in, e.g., https://webkit.org/blog/11300/release-notes-for-safari-technology-preview-114/, and adding implementation_url properties to BCD for those.

But on top of that, I’m now thinking back on all the times in the past when I’ve raised BCD PRs, and — to show reviewers that a feature actually got implemented in a particular browser — I’ve cited a changeset/commit URL in the BCD PR description or BCD commit message. So, going forward, I could also include that as an implementation_url in the BCD PR/patch itself.

So this idea is seeming more and more to me like it could be a nice win for us — for BCD in general. But I’m really looking forward to hearing what others think of the idea.

[foolip]

I'm findings this issue only now via #7039, good stuff, thanks for filing @sideshowbarker!

I too have often wanted the ability to record that something is in Safari Technology Preview, for the sorts of things I highlighted in https://twitter.com/foolip/status/1319559899358089216. Like @sideshowbarker it's most important to know that something has been implemented in WebKit, not when. My use case is using BCD to identify features missing in only one of Chrome, Firefox and Safari, as input for prioritization. (We already do this for web-platform-tests, but BCD is at a higher level and I believe has some advantages over test results.)

Keeping track of exactly which STP version added something is going to be too much work, and I believe it's important that STP data doesn't ever automatically get promoted to mean Safari stable support. That really only rules out using "14.1", and using "TP" like caniuse would be my preference of the alternatives presented.

While I also often dig up the exact commits when something was added, and would definitely make use of implementation_url if it existed, I do slightly worry that we reviewers would come to ask contributors to update their PRs with this information, which even if we provide the URL is something that could scare away contributors. I also worry that I could never sleep again until I had identified precisely when thousands upon thousands of things were implemented :)

[sideshowbarker]

While I also often dig up the exact commits when something was added, and would definitely make use of implementation_url if it existed

OK, then I reckon I’ll write up a patch for a schema change to add it, and corresponding change for the schema docs. That’ll give us something concrete to discuss further.

I do slightly worry that we reviewers would come to ask contributors to update their PRs with this information, which even if we provide the URL is something that could scare away contributors.

I agree and because in general I think it’s better not to pile too many requested review changes on new contributors in their PRs — then as a reviewer in all the repos I contribute to, I personally try to keep requested changes for new-contributors down to just the minimum necessary to be able to merge the PR.

And then I just plan to do a follow-up PR with further changes, with a comment to the contributor in the original PR like “Ideally we’d like to have some implementation_url values added for these features, but we don’t need to block this PR on that — we can add those in a follow-up PR.” That way the contributor still learns something, but doesn’t end up getting disheartened or losing their enthusiasm for contributing due to too many review nits.

I also worry that I could never sleep again until I had identified precisely when thousands upon thousands of things were implemented :)

haha — yup, I know that anxiety.. On the one hand, it’s a nice feeling to realize, wow, here’s some data that we haven’t been recording yet and that we could be, and that maybe could end up being pretty useful — but other hand, it’s also intimidating to realize, wow, there’s really a huge amount of that data we’d need to add if our goal is to record it thoroughly.

So if we do decide to add it to the schema, I think we’d next just try to agree on some criteria for how to prioritize which features and browsers it would be most useful for — so that whatever effort we might make to start retroactively adding it, we try to help make sure that effort’s going into the places where it actually matters most.

[gsnedders]

We’re aware that we have a significant number of cases when we know support for a certain feature has been implemented in a particular browser, but only in a pre-release version. And in those cases, if we do not yet have that version number in any https://github.com/mdn/browser-compat-data/tree/master/browsers file, then we have no way of updating the related BCD data to indicate the browser in question has support for that feature.

If we want to make things even more complicated, there's also the matter of features pref'd off by default (especially in pre-release versions).

Recognizing that BCD data has use cases beyond just tables in MDN and caniuse.com

I'll point out that caniuse.com does show pre-release browsers in the bottommost row, so having the data would be useful even for those use cases.

What's the story for collapsing pre-release versions in to actual release version numbers? For example, if we start logging TP versions in Safari entries, how will we be able to tell if features that shipped in TP X, Y, and Z made the cut for Safari W and translate those numbers accordingly?* How does this work for browsers like Firefox, which don't version pre-releases independently (i.e., when "nightly" is a moving target)?

There's… two hard problems here. One is easier: figuring out the branch point from trunk of both STP and Safari; the other is much harder: compile-time enabled/disabled features and defaults for settings that enable/disable features.

In reality, I think focusing too much on the (closed-source) Safari is probably not worthwhile here. Other browsers have as much challenge here: Chrome and Firefox can both have features enabled on Nightly but not in Beta or Stable. This throws up many of the asme problems.

[ddbeck]

@sideshowbarker Thank you so much for working through these ideas publicly. It gives me a much better sense of what you're trying to achieve and how it might impact other BCD consumers.

Overall, I'm inclined to move forward on "implementation_url" (with some caveats below), because I think it answers some concerns that using "TP" version numbers didn't, such as how to cope with version churn (i.e., converting TP X, Y, and Z to "14.2") and unversioned future releases (e.g., features gated to Firefox Nightly). I also appreciate how this adds to our existing data, providing a machine-readable partial citation for the versions we claim. Finally, along with expanding "spec_url", it hints at additional possible uses for BCD (e.g., you could begin to build a multi-vendor web platform status tool around BCD's spec, implementation, and support data).

(I do have some specific questions and ideas that I mean to propose against the actual, uh, implementation PR—but I think they're details and unlikely to substantially change the overall proposal.)

All that said, I'm inclined to introduce this as an explicit experiment within BCD, without initially committing to not breaking it in the future. If we need to rework things (e.g., an array of strings instead of just one) or it doesn't pan out, I want to retain the option to change it quickly.

[sideshowbarker]

All that said, I'm inclined to introduce this as an explicit experiment within BCD, without initially committing to not breaking it in the future. If we need to rework things (e.g., an array of strings instead of just one) or it doesn't pan out, I want to retain the option to change it quickly.

That all sounds reasonable and would all be fine by me

[gsnedders]

Fixed by #10334.