Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement most of the C++ special member function rules #843

Merged
merged 36 commits into from
Mar 5, 2022

Conversation

bsilver8192
Copy link
Contributor

@bsilver8192 bsilver8192 commented Feb 27, 2022

This now tracks most of the information about which C++ special member
functions are implicit/explicit/deleted/etc for most of the common
cases. This information was needed in several places, which were each
using different approximations that failed in different ways, so unify
those to get it all working. Also add a bunch of tests around the
various cases to keep this working.

This assumes that any non-analyzed types (except some built-in ones
which are handled specially) have no implicit special member functions,
instead of the previous behavior which assumed they all existed if the
analyzed type had explicit declarations. This should generate
functional code for more situations, but it will skip some optional
things (such as moveit traits and make_unique) for additional types. If
you run into issues with those things disappearing after this change,
make sure all dependencies of the type (superclasses and member types)
have a generate!/generate_pod!.

Added TODOs for the following unhandled parts:

Also added TODOs related to the followig issues, which limit what can be
tested but aren't made better or worse by this change:

Also fixed some existing bugs which are surfaced by generating more
trait implementations for types in the existing tests:

  • Use the correct C++ name for destructors of nested types
  • Handle trait methods for types that end up ignored
  • Tests pass
  • Appropriate changes to README are included in PR

@bsilver8192
Copy link
Contributor Author

@philipcraig could you help me understand the error in "Windows (msvc)-test" (https://github.com/google/autocxx/runs/5349131420?check_suite_focus=true)? I see that compilation failed, but I don't see any messages indicating why. I don't have any experience with MSVC, maybe I'm just overlooking it?

This PR should avoid generating UniquePtr instantiations for types without accessible destructors, and the failing test is for that behavior. Given that you had to fix a related issue in #841 that Linux GCC compiled anyways, I wonder if differences in how MSVC handles that part of C++ are involved.

@philipcraig
Copy link
Contributor

philipcraig commented Feb 27, 2022

@philipcraig could you help me understand the error in "Windows (msvc)-test" (https://github.com/google/autocxx/runs/5349131420?check_suite_focus=true)? I see that compilation failed, but I don't see any messages indicating why. I don't have any experience with MSVC, maybe I'm just overlooking it?

This PR should avoid generating UniquePtr instantiations for types without accessible destructors, and the failing test is for that behavior. Given that you had to fix a related issue in #841 that Linux GCC compiled anyways, I wonder if differences in how MSVC handles that part of C++ are involved.

Happy to help. The error is there in the CI if you search the CI output for the failing test name (test_tricky_destructors).

C:\Users\RUNNER~1\AppData\Local\Temp\.tmpIr1ld2\input.h(13): error C3861: '__builtin_trap': identifier not found
cl : Command line warning D9002 : ignoring unknown option '-std=c++14'
cl : Command line warning D9002 : ignoring unknown option '-std=c++14'

which comes from this code in the PR (in the test): https://github.com/google/autocxx/pull/843/files#diff-7d1a694c0fef109924a3e6d7c3462f13a4727cfa3b2a2f86620734ca3e1800fcR9381

Note you will have to get the Github UI to expand integration_test.rs as by default it hides it as the diff is large.

__builtin_trap is a gcc/clang specific builtin, not standard C++, so fails on msvc. I'm not sure if you really wanted it in your test -- was it left in for debugging purposes?

@bsilver8192
Copy link
Contributor Author

Happy to help. The error is there in the CI if you search the CI output for the failing test name (test_tricky_destructors).

C:\Users\RUNNER~1\AppData\Local\Temp\.tmpIr1ld2\input.h(13): error C3861: '__builtin_trap': identifier not found
cl : Command line warning D9002 : ignoring unknown option '-std=c++14'
cl : Command line warning D9002 : ignoring unknown option '-std=c++14'

Thanks, I hadn't realized that between GitHub and cargo test half the information is down at the bottom and half of it is up when the test was run...

__builtin_trap is a gcc/clang specific builtin, not standard C++, so fails on msvc. I'm not sure if you really wanted it in your test -- was it left in for debugging purposes?

I want abort() there, I did it the GCC-specific way without thinking. Thanks!

Although now I'm stuck again with https://github.com/google/autocxx/runs/5349490901?check_suite_focus=true. All I see is a red error in here:

test C:\Users\RUNNER~1\AppData\Local\Temp\.tmp5zpBBl\input.rs ... error: 'rustfmt.exe' is not installed for the toolchain 'nightly-x86_64-pc-windows-msvc'
To install, run `rustup component add rustfmt`
gen0.cxx
error
cl : Command line warning D9002 : ignoring unknown option '-std=c++14'
cl : Command line warning D9002 : ignoring unknown option '-std=c++14'
test integration_test::test_various_emplacement ... FAILED
Execution of the test case was unsuccessful but there was no output.

@philipcraig
Copy link
Contributor

philipcraig commented Feb 27, 2022

I'm stuck again with https://github.com/google/autocxx/runs/5349490901?check_suite_focus=true. All I see is a red error in here:

gen0.cxx
error
test integration_test::test_various_emplacement ... FAILED
Execution of the test case was unsuccessful but there was no output.

I don't know what to do next here either -- it seems some test output from failure is lost, maybe? Reading the source, seems like the Rust code has assertions. I wonder if when they fail we see this?

Could it be related to the s2 example failing to compile on all the triples, or you think that's separate?

In other tricky news, I've built your branch on a Windows host I have access to-- the test passes there, both when run individually or in the overall batch. That of course doesn't mean that everything is necessarily correct -- if there's undefined behaviour somewhere, it doesn't have to crash everywhere. It doesn't help in understanding what's going on though.

@adetaylor
Copy link
Collaborator

Hi @bsilver8192, thanks for this. I'm really excited that you've done such a systematic job here.

I'm afraid it's going to conflict slightly with a PR I've been working on for about a month - #766. I've split that into multiple PRs to get the conflicting bits in place so you can build on them. Specifically the FnPhase analysis phase is now split into two, so your FnStructAnalysis tweak will need to be adjusted a bit.

I also see unexplained problems in the Windows tests. Until we have completed #819 and have a full understanding of how to diagnose and resolve Windows-specific failures, I'm OK with applying attributes to disable extra tests on Windows (after a reasonable amount of investigation).

I also want to #844 to determine whether we have (some types of) UB.

@bsilver8192
Copy link
Contributor Author

I don't know what to do next here either -- it seems some test output from failure is lost, maybe? Reading the source, seems like the Rust code has assertions. I wonder if when they fail we see this?

On Linux, I always see a message when Rust assertions trigger, and a note about setting RUST_BACKTRACE to see the backtrace. I don't see either here.

Could it be related to the s2 example failing to compile on all the triples, or you think that's separate?

That's definitely separate. See the comment chain above for details. It's not generating Rust declarations that the test is testing the existence of, which means Rust compilation fails, which produces much more visible errors.

In other tricky news, I've built your branch on a Windows host I have access to-- the test passes there, both when run individually or in the overall batch. That of course doesn't mean that everything is necessarily correct -- if there's undefined behaviour somewhere, it doesn't have to crash everywhere. It doesn't help in understanding what's going on though.

I looked through that test and I don't see any undefined behavior. I verified on linux-gnu that the destructor is being run the correct number of times (which is more than before this PR; it used to just leak the A, including its std::string, from back_on_stack).

It passed in previous commits of this PR, and I've only made cosmetic changes since then. Any chance this is like #819 (comment), which talks about another test being flaky? The output looks the same, and both tests are moving things with moveit.

This now tracks most of the information about which C++ special member
functions are implicit/explicit/deleted/etc for most of the common
cases. This information was needed in several places, which were each
using different approximations that failed in different ways, so unify
those to get it all working. Also add a bunch of tests around the
various cases to keep this working.

This assumes that any non-analyzed types (except some built-in ones
which are handled specially) have no implicit special member functions,
instead of the previous behavior which assumed they all existed if the
analyzed type had explicit declarations. This should generate
functional code for more situations, but it will skip some optional
things (such as moveit traits and make_unique) for additional types. If
you run into issues with those things disappearing after this change,
make sure all dependencies of the type (superclasses and member types)
have a `generate!`/`generate_pod!`.

Added TODOs for the following unhandled parts:
* google#815 (this is a Clang warning anyways, TODOs show all
  the places to change to fix it)
* google#816 (this just means we ignore some implicit
  constructors which do exist)

Also added TODOs related to the followig issues, which limit what can be
tested but aren't made better or worse by this change:
* google#832 (this one affects lots of areas)
* google#829 (this one's pretty prone to unexpected behavior)

Also fixed some existing bugs which are surfaced by generating more
trait implementations for types in the existing tests:
* Use the correct C++ name for destructors of nested types
* Handle trait methods for types that end up ignored
@bsilver8192
Copy link
Contributor Author

In other tricky news, I've built your branch on a Windows host I have access to-- the test passes there, both when run individually or in the overall batch. That of course doesn't mean that everything is necessarily correct -- if there's undefined behaviour somewhere, it doesn't have to crash everywhere. It doesn't help in understanding what's going on though.

I looked through that test and I don't see any undefined behavior. I verified on linux-gnu that the destructor is being run the correct number of times (which is more than before this PR; it used to just leak the A, including its std::string, from back_on_stack).

It passed in previous commits of this PR, and I've only made cosmetic changes since then. Any chance this is like #819 (comment), which talks about another test being flaky? The output looks the same, and both tests are moving things with moveit.

Well, current main is calling a C++ destructor on a pointer which never had the constructor called in another related test, which actually crashes for me (on linux-gnu). I'm going to fix that and send out a separate (hopefully smaller) PR so other people stop running into it, and hopefully that fixes all of them. asan should catch this reliably too.

@bsilver8192
Copy link
Contributor Author

Well, current main is calling a C++ destructor on a pointer which never had the constructor called in another related test, which actually crashes for me (on linux-gnu). I'm going to fix that and send out a separate (hopefully smaller) PR so other people stop running into it, and hopefully that fixes all of them. asan should catch this reliably too.

Looks like it was a recent change (#766 (comment)), so doesn't explain flakiness. I pushed my rebased version here if anybody wants to review.

This is actually pretty easy, and it eliminates the main source of new
incompatibilities I can think of.

Also remove the note about typedefs, they actually work fine.
Copy link
Collaborator

@adetaylor adetaylor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this! I've reviewed everything except the actual logic for creating constructors within implicit_constructors.rs. I've suggested you merge that file with implicit_constructor_rules.rs and then I'll take a look.

I've made some other requests but this is looking pretty great.

integration-tests/tests/integration_test.rs Show resolved Hide resolved
engine/src/known_types.rs Outdated Show resolved Hide resolved
engine/src/known_types.rs Outdated Show resolved Hide resolved
engine/src/conversion/codegen_cpp/mod.rs Outdated Show resolved Hide resolved
engine/src/conversion/analysis/remove_ignored.rs Outdated Show resolved Hide resolved
engine/src/conversion/analysis/fun/mod.rs Outdated Show resolved Hide resolved
engine/src/conversion/analysis/fun/mod.rs Outdated Show resolved Hide resolved
engine/src/conversion/codegen_rs/impl_item_creator.rs Outdated Show resolved Hide resolved
@adetaylor adetaylor mentioned this pull request Mar 2, 2022
Copy link
Collaborator

@adetaylor adetaylor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few tiny nits/questions but this is basically ready to land.

engine/src/conversion/error_reporter.rs Show resolved Hide resolved
engine/src/conversion/analysis/fun/mod.rs Outdated Show resolved Hide resolved
@adetaylor
Copy link
Collaborator

OK, assuming this all passes the CI, I'll go ahead and merge this tomorrow. Thanks again for all this excellent work!

@bsilver8192
Copy link
Contributor Author

Awesome! Thanks for maintaining this, I'm impressed at how much C++ it handles already, and how much more it does every day.

@bsilver8192
Copy link
Contributor Author

With my full use case, I'm getting C++ wrappers for constructors etc for _bindgen_ty_54-style types which don't actually exist, so the generated C++ doesn't compile. Looks like they're getting past the GC stage because many of the generated constructors in a given namespace share the QualifiedName(Namespace([]), "new_synthetic_move_ctor") name, so they all get lumped together.

I suspect it's something that can happen before this PR, but this PR makes it happen in many more situations. Although I'm not actually sure what situations create these _bindgen_ty_ types, so maybe they can never have explicit special members that would trigger code generation before this PR. That also means I don't know how common of an occurrence this problem is, so I don't know whether it makes sense to hold up this PR for it.

I can keep digging into this over the next few days if there isn't an easy answer.

if let Some(Api::Struct {
analysis: PodAndConstructorAnalysis { constructors, .. },
..
}) = apis.iter_mut().find(|api| api.name() == self_ty)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm concerned this may O(n^2) over the number of APIs. I think I've avoided that elsewhere so far...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like you fixed this one in #889, LGTM.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Never mind, you just cut the square in half, but the triangle is still O(n^2). This one is pretty easy to move into find_constructors_present instead, done.

#[derive(Debug, Default, Copy, Clone)]
pub(crate) struct PublicConstructors {
pub(crate) default_constructor: bool,
pub(crate) copy_constructor: bool,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I can see, a couple of these fields are never read.

When I merge this with #879 this gives me compile-time warnings. I'm not sure why it doesn't here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's just this one, which ended up not being referenced by other stages. Done

let initial_name = name.clone();
// Now go find the Api::Struct for our type, and verify it has an accessible
// destructor.
for api in apis.iter() {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also seems to be O(n^2)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@adetaylor
Copy link
Collaborator

adetaylor commented Mar 5, 2022

OK, state of play:

Using the implicit ones (like this PR does for non-nested types)
requires some additional refactoring, so let's leave that for now
(google#884).
@bsilver8192
Copy link
Contributor Author

  • Thanks for reporting the issues with non-unique API names. I have fixed them in Ensure every API has a unique name #879.
  • That's a substantial change, and of course courses merge conflicts. I have taken the liberty of resolving them with your changes here, and put the result in Implicit constructors, with unique API names #889. Feel free to just merge that branch into this PR if you like. (There are a lot of merge commits - it was a bit painstaking).

Thanks, that's an annoying merge.

I got partway through fixing #884, and then realized that it's not actually necessary. I might have it finished later tonight, definitely sometime this weekend. Treating nested types the same way as types with unknown bases/members lets explicit special member functions work like before, so all the existing tests pass. WDYT about leaving that for a separate PR?

@adetaylor
Copy link
Collaborator

Leaving it till later sounds fine to me.

@@ -89,6 +89,10 @@ impl<P: AnalysisPhase> ApiVec<P> {
self.apis.iter()
}

pub(crate) fn iter_mut(&mut self) -> impl Iterator<Item = &mut Api<P>> {
self.apis.iter_mut()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, we can't do this. In fact I worked really hard to remove this in bcfeb6f! The problem is that it might invalidate the invariants which ApiVec promises to maintain.

I struggled for a while with a custom iterator which then re-checked the invariants on its Drop, but borrow checker fights caused me to give up, and rewrite calling code so it didn't need iter_mut. You're welcome to try that again.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WDYT about fn mutate_fun_analysis<F: Fn(&mut P::FunAnalysis)>(&mut self, f: F) (and another one for structs)? Kind of ugly, but those don't allow changing the names, so the invariants can't be invalidated.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or, a bit less ugly but with a runtime assertion, fn filter_inplace<F: Fn(&mut Api)>(&mut self, f: F) which assert!s that the name doesn't change after each function call.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either would be fine with me, yes. I think on the whole I prefer the former, but I don't mind!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@adetaylor
Copy link
Collaborator

OK, I'm going to go ahead and merge this, but I would like one or other of us to try making a follow-up change to remove a little of the added complexity.

I'm not keen to add an extra field to MethodKind unless it's strictly necessary. I think in this case we can avoid it.

This is the sort of pattern that autocxx does in other places:
d213e4e
The power of hash tables to avoid O(n^2)...

That commit isn't quite complete because (a) it doesn't remove the newly-added field, (b) I haven't checked that it correctly respects all implicit + explicit destructors, but hopefully we can make this work.

Unfortunately I won't have any time at all to work on autocxx this weekend, hence going ahead and merging this to unblock you. Feel free to raise a PR completing this approach if you like, or I'm happy to take a look later. It might turn out not to be possible anyway.

Apart from that, this PR remains terrific, thanks a lot!

@adetaylor adetaylor merged commit 7bd351c into google:main Mar 5, 2022
@bsilver8192 bsilver8192 deleted the special-member-functions branch March 5, 2022 20:46
bsilver8192 added a commit to bsilver8192/autocxx that referenced this pull request Mar 5, 2022
This should be the state after google#843 and google#891. I think documenting this
will be helpful for people running this on a new codebase (or part of a
codebase).
@bsilver8192
Copy link
Contributor Author

Bisecting by hand, it's a method that takes cpu_set_t by value which causes the problem. I'll dig more into what about that type is unique tomorrow.

A followup on this: I no longer see the bad destructors, not even pre-GC. I think the apivec-driven fixes addressed the root cause. Yay!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants