Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ruby 2.7: Refinements take place at Object#method and Module#instance_method #2177

Merged
merged 16 commits into from
Jan 31, 2021

Conversation

ssnickolay
Copy link
Collaborator

Related to #2004

A few notes:

  1. Refinements for #method is similar to #respond_to? (see: https://github.com/oracle/truffleruby/pull/2120/files) where we used ReadCallerFrameNode. In this PR I don't use ReadCallerFrameNode, but instead directly call:
getContext().getCallStack().getCallerFrameIgnoringSend(FrameInstance.FrameAccess.READ_ONLY);

to avoid frame materialization. However, we discussed the possible optimization of ReadCallerFrameNode so maybe we should repeat the RespondToNode approach (with @Child private ReadCallerFrameNode readCallerFrame) to be more consistent.

I didn't find #method or #instance_method uses via Primitive (so there are no needs in : d84da65).

There were also options with AST-inlining, but we haven't done that yet.

  1. The following behavior seems a little confusing
# $ ruby -v
# => ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-darwin19]

klass = Class.new

refinement = Module.new do
  refine klass do
    def foo; end
  end
end

result = nil
result2 = nil

Module.new do
  using refinement

  result = klass.instance_method(:foo)
  result2 = klass.method_defined?(:foo)
end

puts result
# => #<UnboundMethod: #<Class:0x00007fa2df849688>(#<refinement:#<Class:0x00007fa2df849688>@#<Module:0x00007fa2df8494d0>>)#foo() test.rb:5>
puts result2
# => false

How do you think this is expected?

spec/ruby/core/module/refine_spec.rb Show resolved Hide resolved
@@ -1352,6 +1354,15 @@ protected RubyMethod method(VirtualFrame frame, Object self, Object name,
@Cached ConditionProfile notFoundProfile,
@Cached ConditionProfile respondToMissingProfile,
@Cached LogicalClassNode logicalClassNode) {
final Frame callerFrame = getContext()
.getCallStack()
.getCallerFrameIgnoringSend(FrameInstance.FrameAccess.READ_ONLY);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kernel#method should be fast and basically just be an inline cache on the instance class + name.
This causes to iterate the stack every time.

Let's use the same approach as in RespondToNode.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, but ReadCallerFrameNode (used in RespondToNode) also iterate the stack, isn't?

public Object execute(VirtualFrame frame) {
final Object data = getData(frame);
if (callerDataProfile.profile(data != null)) {
return data;
} else {
return getCallerData();
}
}

It just has a condition profiler to "skip" an iteration sometimes (not sure how often)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first branch is basically always taken after the first call for a given call site, so it only iterates once per call site, and then all later calls just read the caller frame from the arguments of the current frame (except edge cases like a non-Ruby caller, but we can detect those by being passed a null frame or so).

src/main/java/org/truffleruby/core/kernel/KernelNodes.java Outdated Show resolved Hide resolved
@eregon eregon self-assigned this Dec 7, 2020
@ssnickolay
Copy link
Collaborator Author

Here is a quick summary of the work & progress of this issue.

The whole idea of supporting refinements for #method is to get somehow caller frame and use its DeclarationContext (which has activated refinements);

  • The first idea was to get callerFrame with getContext().getCallStack().getCallerFrameIgnoringSend, but this way is suboptimal (1) each time we have to go over the whole stack; 2) it could be slow for non-Ruby calls).
  • To fix it we have a proven way: ReadCallerFrameNode. It works OK when Node does not have foreign calls (e.g. C-API level calls). In the current case GetMethodObjectNode has C-API level execution (see:
    https://github.com/oracle/truffleruby/blob/master/src/main/c/cext/exception.c#L136-L139), so using it leads to exception:
<no message> (java.lang.NullPointerException)
	from org.truffleruby.core.CoreLibrary.isSend(CoreLibrary.java:967)
	from org.truffleruby.language.CallStackManager.lambda$callerIsSend$7(CallStackManager.java:108)
	from org.truffleruby.language.CallStackManager$FilterApplyVisitor.visitFrame(CallStackManager.java:173)
	from com.oracle.truffle.api.impl.DefaultTruffleRuntime.iterateFrames(DefaultTruffleRuntime.java:174)
	from org.truffleruby.language.CallStackManager.iterateFrames(CallStackManager.java:150)
	from org.truffleruby.language.CallStackManager.callerIsSend(CallStackManager.java:103)
	from org.truffleruby.language.FrameAndVariablesSendingNode.startSendingOwnFrame(FrameAndVariablesSendingNode.java:90)
	from org.truffleruby.language.arguments.ReadCallerFrameNode.startSending(ReadCallerFrameNode.java:35)
	from org.truffleruby.language.arguments.ReadCallerDataNode.notifyCallerToSendData(ReadCallerDataNode.java:65)
	from org.truffleruby.language.arguments.ReadCallerDataNode.getCallerData(ReadCallerDataNode.java:45)
	from org.truffleruby.language.arguments.ReadCallerDataNode.execute(ReadCallerDataNode.java:37)
	from org.truffleruby.language.arguments.ReadCallerFrameNode.execute(ReadCallerFrameNode.java:25)
	from org.truffleruby.core.kernel.KernelNodes$MethodNode.method(KernelNodes.java:1334)
	from org.truffleruby.core.kernel.KernelNodesFactory$MethodNodeFactory$MethodNodeGen.execute(KernelNodesFactory.java:4461)
	from org.truffleruby.language.arguments.CheckArityNode.execute(CheckArityNode.java:41)
	from org.truffleruby.language.methods.ExceptionTranslatingNode.execute(ExceptionTranslatingNode.java:33)
	from org.truffleruby.language.RubyRootNode.execute(RubyRootNode.java:61)
	from com.oracle.truffle.api.impl.DefaultCallTarget.callDirectOrIndirect(DefaultCallTarget.java:84)
/Users/ssnickolay/Projects/oss/graal/truffleruby-ws-2/truffleruby/src/main/c/cext/exception.c:136:in `method'
	from /Users/ssnickolay/Projects/oss/graal/truffleruby-ws-2/truffleruby/src/main/c/cext/exception.c:136:in `rb_tr_init_exception'
	from /Users/ssnickolay/Projects/oss/graal/truffleruby-ws-2/truffleruby/src/main/c/cext/ruby.c:32:in `rb_tr_init'
	from /Users/ssnickolay/Projects/oss/graal/truffleruby-ws-2/truffleruby/spec/mspec/lib/mspec/guards/platform.rb:55:in `require'
	from /Users/ssnickolay/Projects/oss/graal/truffleruby-ws-2/truffleruby/spec/mspec/lib/mspec/guards/platform.rb:55:in `<class:PlatformGuard>'
	from /Users/ssnickolay/Projects/oss/graal/truffleruby-ws-2/truffleruby/spec/mspec/lib/mspec/guards/platform.rb:3:in `<top (required)>'
	from <internal:core> core/kernel.rb:260:in `require'
	from /Users/ssnickolay/Projects/oss/graal/truffleruby-ws-2/truffleruby/spec/mspec/lib/mspec/guards.rb:7:in `<top (required)>'
	from <internal:core> core/kernel.rb:260:in `require'
	from /Users/ssnickolay/Projects/oss/graal/truffleruby-ws-2/truffleruby/spec/mspec/lib/mspec.rb:6:in `<top (required)>'
	from <internal:core> core/kernel.rb:260:in `require'
	from /Users/ssnickolay/Projects/oss/graal/truffleruby-ws-2/truffleruby/spec/mspec/lib/mspec/utils/script.rb:280:in `main'
	from spec/mspec/bin/mspec:7:in `<main>'

We need to tweek org.truffleruby.language.RubyDynamicObject#readMember to pass null frame and avoid ReadCallerFrameNode execution if frame == null.

  • @eregon proposed to avoid DispatchNode lookup and:

"We should avoid, in general, to use ReadCallerFrameNode when the caller is foreign, though. In this case, I'm not sure how we could avoid it. So using GetMethodObjectNode directly in RubyDynamicObject#readMember seems best." (c) Benoit Daloze

  • Using GetMethodObjectNode directly in RubyDynamicObject#readMember requires support @Cached for GetMethodObjectNode. The first attempt was to implement Cached API manually as we have for DispatchNode. It almost worked: refinements spec were fixed, but one test from jt test fast was failed:
jt test spec/truffle/interop/matrix_spec.rbtruffleruby: an internal exception escaped out of the interpreter,
please report it to https://github.com/oracle/truffleruby/issues.```
Node must be adopted before a reference can be looked up. (java.lang.IllegalStateException)
	from com.oracle.truffle.api.nodes.Node.checkAdoptable(Node.java:744)
	from com.oracle.truffle.api.nodes.Node.getExecutableNode(Node.java:733)
	from com.oracle.truffle.api.nodes.Node.lookupContextReference(Node.java:799)
	from org.truffleruby.language.methods.LookupMethodNodeGen.executeAndSpecialize(LookupMethodNodeGen.java:110)

Then we had a conversation in Slack about the solution and the final thought is:

@GenerateUncached should be used for nodes that need an uncached version, DispatchNode should be about the only exception not using it (would need some new Truffle DSL feature)" (c) Benoit Daloze

  • The second attempt was about using @GenerateUncached, but it is a bit harder then I expected:
  1. I have an exception java.lang.IllegalStateException: Node must be adopted before a reference can be looked up. at the beginning of ruby initialization
  2. @Cached ReadCallerFrameNode readCallerFrame does not work because this Node does not support @GenerateUncached. Did temporary workaround with manually creating node ReadCallerFrameNode.create()
  3. doIt works with Frame frame, but two methods inside require VirtualFrame. I had to use typecasting, but I believe this is not a good solution.

As a result, I would be happy to get a review of the last solution and a hint of what I am doing wrong 🙏

@eregon
Copy link
Member

eregon commented Jan 11, 2021

My apologies for the super late reply.
I rebased the PR.

The error is from ReadCallerFrameNode.create(), which is done at runtime, and that node is never inserted in the AST.
And making ReadCallerFrameNode uncachaeable seems difficult.
So we should get the caller frame in the caller and have the ReadCallerFrameNode only there (e.g., Kernel#method / KernelNodes.MethodNode), if such semantics are needed (e.g., not for interop), and pass it to GetMethodObjectNode.
We probably need to pass an extra DeclarationContext originalDeclarationContext argument in that case to restore it for respond_to_missing?.

Regarding type casting, I think we can exploit the fact that MaterializedFrame subclasses VirtualFrame, but we might not even need it.

@eregon eregon force-pushed the feat/2.7-refinements-for-method branch from 286ad8d to 237d2a5 Compare January 11, 2021 15:59
@ssnickolay ssnickolay force-pushed the feat/2.7-refinements-for-method branch 3 times, most recently from 8aea66c to 783c3cc Compare January 25, 2021 16:11
@ssnickolay
Copy link
Collaborator Author

@eregon we are ready for another review-round 🧑‍🍳

@ssnickolay ssnickolay force-pushed the feat/2.7-refinements-for-method branch from 849d51a to 01ef61a Compare January 29, 2021 08:12
@ssnickolay ssnickolay force-pushed the feat/2.7-refinements-for-method branch from 01ef61a to 2d95cfe Compare January 29, 2021 08:18
Copy link
Member

@eregon eregon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great PR, thank you for your work and persistence on this, it was much harder than expected!

* Now that we found out that having a public execute(Frame, ...) is fine.
@eregon eregon added the in-ci The PR is being tested in CI. Do not push new commits. label Jan 29, 2021
}
final RubyMethod instance = new RubyMethod(
context.getCoreLibrary().methodClass,
RubyLanguage.getCurrentLanguage().methodShape,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RubyLanguage.getCurrentLanguage() is slow, it's essentially a ThreadLocal lookup.
So I switched to @CachedLanguage instead.

…sed a reason

* neverPartOfCompilation() is much safer to use now that it is checked
  statically when building a Native Image.
…erPartOfCompilation

* neverPartOfCompilation() is much safer to use now that it is checked
  statically when building a Native Image.
@eregon
Copy link
Member

eregon commented Jan 29, 2021

I added a check that RubyLanguage.getCurrentLanguage/getCurrentContext are never used in PE code, so the gate would nicely catch issues like #2177 (comment).

@eregon eregon added this to the 21.1.0 milestone Jan 29, 2021
@eregon
Copy link
Member

eregon commented Jan 30, 2021

It looks like TCK tests fail, I'll take a look:

There were 3 failures:
1) com.oracle.truffle.tck.tests.IdentityFunctionTest#testIdentityFunction[java-host::identity(ruby::nil)]
java.lang.AssertionError: Running snippet 'identity' retrieved from 'java-host' provider (java class com.oracle.truffle.tck.tests.JavaHostLanguageProvider) with parameters:
'nil' from 'ruby' provider, value: nil (Meta Object: NilClass)
failed:
Unexpected Exception: java.lang.NullPointerException
Snippet: null
Parameter 0 Snippet: -> { nil }

	at com.oracle.truffle.tck.tests.IdentityFunctionTest.testIdentityFunction(IdentityFunctionTest.java:119)
	at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at com.oracle.mxtool.junit.MxJUnitWrapper.runRequest(MxJUnitWrapper.java:357)
	at com.oracle.mxtool.junit.MxJUnitWrapper.main(MxJUnitWrapper.java:222)
Caused by: org.graalvm.polyglot.PolyglotException: java.lang.NullPointerException
	at org.truffleruby.core.CoreLibrary.isSend(CoreLibrary.java:956)
	at org.truffleruby.language.CallStackManager.lambda$callerIsSend$7(CallStackManager.java:108)
	at org.truffleruby.language.CallStackManager$FilterApplyVisitor.visitFrame(CallStackManager.java:187)
	at com.oracle.truffle.api.impl.DefaultTruffleRuntime.iterateFrames(DefaultTruffleRuntime.java:174)
	at org.truffleruby.language.CallStackManager.iterateFrames(CallStackManager.java:164)
	at org.truffleruby.language.CallStackManager.callerIsSend(CallStackManager.java:103)
	at org.truffleruby.language.FrameAndVariablesSendingNode.startSendingOwnFrame(FrameAndVariablesSendingNode.java:89)
	at org.truffleruby.language.arguments.ReadCallerFrameNode.startSending(ReadCallerFrameNode.java:35)
	at org.truffleruby.language.arguments.ReadCallerDataNode.notifyCallerToSendData(ReadCallerDataNode.java:65)
	at org.truffleruby.language.arguments.ReadCallerDataNode.getCallerData(ReadCallerDataNode.java:45)
	at org.truffleruby.language.arguments.ReadCallerDataNode.execute(ReadCallerDataNode.java:37)
	at org.truffleruby.language.arguments.ReadCallerFrameNode.execute(ReadCallerFrameNode.java:25)
	at org.truffleruby.core.kernel.KernelNodes$MethodNode.method(KernelNodes.java:1313)
	at org.truffleruby.core.kernel.KernelNodesFactory$MethodNodeFactory$MethodNodeGen.execute(KernelNodesFactory.java:4496)
	at org.truffleruby.language.arguments.CheckArityNode.execute(CheckArityNode.java:45)
	at org.truffleruby.language.methods.ExceptionTranslatingNode.execute(ExceptionTranslatingNode.java:33)
	at org.truffleruby.language.RubyNodeWrapper.execute(RubyNodeWrapper.java:45)
	at org.truffleruby.language.RubyRootNode.execute(RubyRootNode.java:61)
	at org.graalvm.polyglot.Value.getMember(Value.java:371)
	at com.oracle.truffle.tck.tests.ValueAssert.assertValueImpl(ValueAssert.java:523)
	at com.oracle.truffle.tck.tests.ValueAssert.assertValue(ValueAssert.java:129)
	at com.oracle.truffle.tck.tests.ValueAssert.assertValue(ValueAssert.java:124)
	at com.oracle.truffle.tck.tests.TestUtil.assertValue(TestUtil.java:177)
	at com.oracle.truffle.tck.tests.TestUtil.validateResult(TestUtil.java:163)
	at com.oracle.truffle.tck.tests.TestUtil.validateResult(TestUtil.java:152)
	at com.oracle.truffle.tck.tests.IdentityFunctionTest.testIdentityFunction(IdentityFunctionTest.java:111)
	... 37 more
...

@graalvmbot graalvmbot merged commit 22161c4 into oracle:master Jan 31, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
in-ci The PR is being tested in CI. Do not push new commits. oca-signed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants