Move to System.Reflection.Metadata? #12906
Replies: 19 comments
-
@jack-pappas I'm not in favour for two reasons
Specifically for (2), the Fable/Javascript port of F# includes the binary reader/writer for components on the .NET IL. Even though the compiler is run using Javascript, it still uses the .NET IL format as its binary format, both for input and output metadata. The same would apply whenever we cross-compile the F# compiler to other runtimes - e.g. there is an Erlang-runtime port of F#. (2) means there are a major long-term advantages to having the F# compiler be an all-F# program: it helps maintain the technical independence of F# from the .NET runtime. This is at the cost of actually implementing the binary reader/writer in the first place, but that cost is paid. (2) also implies that I'm generally not in favor of |
Beta Was this translation helpful? Give feedback.
-
The advantage of moving over to libraries shared with Roslyn are that we can leverage more of the Roslyn team for work. The disadvantage is that if we manage it ourselves we can leverage more of the F# Open Source community. We have gained enormous benefits from the work of the community and the flexibility that offers them in implementing new things. |
Beta Was this translation helpful? Give feedback.
-
@dsyme (and @alfonsogarciacaro)
Say that, prior to Fable being created, we deferred to S.R.M. for reading/writing IL. Would that have prevented Fable from being developed in the first place? |
Beta Was this translation helpful? Give feedback.
-
Yes. That is, it would have likely prevented the cross-compilation of the F# compiler with Fable to execute on Javascript, maintained in this branch. This form of the F# compiler is used as the basis for the Fable REPL and has been mooted to be used for part of the re-implementation of tryfsharp.org.. Moving to S.R.M would almost certainly terminate that effort or cause a lot of extra re-implementation (basically forcing those contributors to re-implement S.R.M in F# or extend Fable to consume C#), which is not something I feel we should do.
An all-F# implementation of F# is much preferred for long-term language-independence. Otherwise F#'s existence and manifestations just becomes fundamentally dependent on Roslyn. The Fable port of the F# compiler is one great example, but Fez is another (F# to Erlang) and it's quite possible there will be others. The ideal is the FSharp.Compiler.Service.dll has no dependencies besides FSharp.Core.dll. |
Beta Was this translation helpful? Give feedback.
-
@ncave has done all the job of porting the F# compiler to Fable so I'm not the biggest authority here ;) but just a few notes:
The only problem with this is the Fable REPL has to download around 2MB of gzipped dll files. I would like if there was way to have only the metada without the rest of the code, be it in a .dll or a different format. But it's not really a big issue. |
Beta Was this translation helpful? Give feedback.
-
IMO the easy availability of the F# IL reader/writer makes tinkering with assemblies (e.g. exporting the BCL metadata, porting to other platforms, etc.) trivial, thus lowering the bar for entry. @alfonsogarciacaro The whole netcore 2.0 BCL metadata is less than 0.8 MB (gzipped), and FSharp.Core metadata is another 0.5 MB (gzipped). |
Beta Was this translation helpful? Give feedback.
-
Please keep the whole compiler in pure F# (including the F# IL writer/reader), because it will allows F# to port F# to different runtimes. For example there is a port in the pipe (as I the compiler is implemented similarly to Fable) to Erlang BEAM virtual machine (https://github.com/kjnilsson/fez). I also have plans - but no time yet - to port a minimal subset to LLVM (probably on top of https://github.com/kp-tech/fshlvm). |
Beta Was this translation helpful? Give feedback.
-
@zpodlovics Potentially porting F# to the JVM would also be another example - no on has bitten that off but it is not impossible by any means (you would just lose some features, e.g. tailcalls not guaranteed, generics would box etc.) |
Beta Was this translation helpful? Give feedback.
-
I'll close this as it's clear that we won't be moving to S.R.M. any time soon (or ever). |
Beta Was this translation helpful? Give feedback.
-
Re-opening, as moving to SRM could actually be on the horizon. I've re-implemented ilread.fs using SRM, https://github.com/TIHan/visualfsharp/blob/new-metadata-reader/src/fsharp/FSharp.Compiler.Private/ilread_srm.fs The remaining unknowns are native resources, managed resources, and pdb info (such as sequence points) as I'm not sure what to do about them yet. |
Beta Was this translation helpful? Give feedback.
-
In summary, |
Beta Was this translation helpful? Give feedback.
-
@TIHan @cartermp I still think this is a good idea, but others had some concerns about it (above). What’s changed since then that you’re going ahead with the change now (in spite of those concerns)? |
Beta Was this translation helpful? Give feedback.
-
We have an approach that doesn't involve ripping things out. F# on .NET would use SRM, and there would be a configuration to not use it in other contexts, should someone decide to take on the (insanely complicated) task of doing something like porting F# to the JVM or LLVM. The primary motivations are perf and maintainability. Initial benchmarks show that it's considerably faster and uses significantly less memory. SRM is pretty highly-optimized, so that shouldn't be a surprise. Reading IL in particular is significant from a perf perspective for F# in an IDE, so there are very concrete benefits to doing this for the large majority of F# users. |
Beta Was this translation helpful? Give feedback.
-
Initial benchmarks show that it's better in perf, but this is with a lot of caching enabled so that may have something to do with it plus we still have more work to do that could impact perf later on; though we got most of the major stuff implemented already. The primary motivation of it was to improve the codebase and as a consequence, improves perf a bit. Thus, I believe it is worth it. Yes, the concerns presented above are real in regards to Fable. I don't have a clear picture on what we need to do that could help them in this context. This type of change is to be well communicated in advance. I don't know when we will actually start trying to get this in; it will be a while, but not never. We want to improve the F# compiler and tooling. We want to make it faster and more maintainable. There is no smoking bullet to make that happen; we need to try to improve everything. This is just another step into that direction. |
Beta Was this translation helpful? Give feedback.
-
@TIHan it sounds very interesting, do you happen to know if having this in place would help on the issue I was describing in #2345 The issue back then (not sure it still applies): memory was taken from referenced assemblies embedding large resources. Finding ways to trim resources when reading the IL into memory for compile / type checking purpose could save a great deal of space. When this code will get rearchitected, it may be worth to try to get a clear cut between the state and the querying over it that occurs later in the code, eventually those could be split for out of process cache of IL references in context of VS language service. |
Beta Was this translation helpful? Give feedback.
-
The issue you described should be resolved due to Don's work to integrate with roslyn metadata stuff. It shouldn't always load the resources into memory today. |
Beta Was this translation helpful? Give feedback.
-
So to be clear: What I read is that Microsoft will no longer maintain the F# version of the metadata reader and writer, but it will be handed over to the community, correct? If this is true, one additional thing to consider is that it might be hard for the community to make sure the quality of the F# version will be "on-par" with the SRM implementation. In practice, that means breaking some of the above use-cases until someone is willing to PR a bugfix and then waiting for the next release. Note: I'm not saying we shouldn't do this — just something we might need to keep in mind. Maybe we can do something about it (process wise or via some other way)? Perhaps it's a non-issue just something I wanted to bring up into the discussion. |
Beta Was this translation helpful? Give feedback.
-
I think it would become part of the Fable PR to cross-compile the compiler with Fable. That would make sense to me. Alternatively we could I don't expect any need for the ilread.fs binary reader in any other situation (well, there's a variant of it living on in ProvidedTypes.fs, but that's a separate thing, see fsprojects/FSharp.TypeProviders.SDK#298 |
Beta Was this translation helpful? Give feedback.
-
This was started by @TIHan here: #8081. It is nearly complete, however needs updating to main I'll convert this issue to a discussion |
Beta Was this translation helpful? Give feedback.
-
Is it worth simplifying/eliminating/consolidating a big chunk of the F# compiler code by replacing most of the
AbsIL
code with the System.Reflection.Metadata library? That’s the library used by Roslyn to read/write .NET assemblies; switching over to using that library would more closely align the F# compiler with the Roslyn compilers and reduce the amount of code we need to manage/maintain in the F# compiler. It’s also been well-optimized so may provide some performance benefits during compilation.If there’s interest in this from the F# compiler team (e.g. @dsyme or @KevinRansom), I’m willing to take a shot at prototyping it.
Beta Was this translation helpful? Give feedback.
All reactions