-
Notifications
You must be signed in to change notification settings - Fork 179
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Metaclasses cannot have tp_new in Python 3.14 #1216
Comments
This likely won't work for us. This is part of their continued effort to make all of the machinery behind the scenes private. I tried their official way of doing things. The problem is that types are closed and there is no way to add the memory required for our meta class storage. If they want us to have any hope to proceeding forward, they will need to look at our code, understand our use case, and recommend a solution. |
The function we are doing with
The second from meta is more drastic as it is replicating a heap type allocation entirely. It was added because Python does not support FromSpecWithBases which is a specialization. If we get them to accept a patch for that concept then it would reduce our burdens. In theory the calls could be moved elsewhere though I suspect the type system has similar issues with those places because we have irregular class structures. Unfortunately every hour of my time is accounted for until the end of the fiscal year and then I have other real life concerns like finding food (account numbers). In the worst case scenario, I will have to accept a project that prohibits public contributions to free software. My employer uses JPype and in theory they should provide me an account to deal with these issues so that I can be responsive. My recommendation would be to start a PEP to try to get the FromSepcWithBases accepted or direction how to put it in place. I will need to find a new path for the tp_new slot. But this will take me several weekends of time to get you an answer. I am still working the Python 3.13 issue. |
Okay I investigated this one. The problem is not with the PyType_FromMetaclass. It takes the meta class, module, spec, and bases so everything we need. The problem is that I can't find any way of passing kwargs to the places that need them when creating new types. I always get
The error message doesn't make much sense as I am creating JInterface class so of course it wouldn't be called. I tried adding |
I have found a way to make the kwargs pass through. The problem appears to be that Edit: Reading the code it appears that they should be sticky. I am guessing bugs in our calling path are sending Python down into dead code or other bad, bad places. |
I gave it a shot, but after 14 hours at fixing Python issues I am beat. See #1217 |
I think I just found the cause of this specific issue, at least for jpype/native/python/pyjp_number.cpp Line 390 in 66f8c6c
It should be |
Excellent. I will be out on a hike for the day and try to get a few more hours on this later. I recommend rebasing against the Py3.13 branch latest. There were a number of instabilities in the prior attempts.
|
I probably should have said this in the beginning, but there really is no rush here. I was just trying to make noise to prevent the same situation we currently have with 3.13. I do see the same issue with I had an idea to eliminate the need to have extra storage for the I spent a few hours this morning setting this up to build with cmake and ninja so I can build and test in place without having to wait for everything to rebuild everytime and copy the output pyd. I also "cleaned" up the headers so now I can use modern tooling without having 12354684651383 errors thrown at me. I will hopefully be more helpful sooner rather than later. Edit: Nevermind on not being able to find the javaslot on |
Ok. So from what I can gather, the hassle of having to allocate additional storage for all the primitive types can go away. The only 2 which require a "java slot" are Each The benefit of doing it this way would allow jpype to work for older versions of Python and newer versions in the "same" way while allowing it to be done correctly with The downside of this, which I'm sure is quite obvious, is that it will require a lot of refactoring. I've already started and I think I can make this work, it's just going to take some time. It may be a month or two before I have anything done. Worst case scenerio, I'm completely wrong and I end up becoming much more familiar with jpype's internals which is beneficial anyways. tl;dr we can take advantage of every Python object having a type and being able to store specific data in a type, to hold the |
The issue for the Java slot is for the boxed types as well as the overload. We do keep the JClass instance on JInt, JLong, etc for type matching when overloading. Thus all types require that extra information. Remember the main reason for those types it to resolve overloading the deciding the correct box type when box casting. In theory you could have a lookup table when performing the overload rather that a direct box lookup but that was punishing when performing overload matching. I am going to try to get my son to write a PEP for moving user data ahead of the object into the opaque space. |
If the sole goal is to avoid the memory model issues, then the solution is to allocate two types for every JClass, the abstract version obeys the class diagram tree of the Java and the concrete tree adds the memory on top of each specific type as needed (type, exception, long, float, object, exception). There would then be a pointer in the JClass which denotes the offset to the JValue. All of the new methods for the abstract would construct the concrete with the correct type. No extra memory is required though boxed types and primitives become a real pain as both of them have very hard coded layouts making meaning you will just end up fighting the next breakage. It just requires the extra association on the type hints and a bunch of extra layers to make sure the user can accidently create one of the abstract classes. The issue that I had when I tried that solution was that isa tests which seem trivial were broken. Everything is a derived class and not the type that people expected. So I abandoned it in favor of the "extra memory" solution. After all what is the point of having an allocator slot if you can't add to the memory profile? The issue here is that the user level GC API in Python is an after thought and because it is off the beaten path, it breaks so regularly that it might as well not exist. I spend weekend after weekend tracing through Python code with debuggers deciding that the methods they gave me don't work and are corrupting method, and then I just abandon it in favor of a hack which works at least until they start adding opaque memory everywhere. For now my polymorph solution works for every known version of Python and hopefully we can negotiate a workable API going forward. It is a three line change to add a preheader size slot to the type structure, add the new opaque data to preheader size when the type is allocated, and then return that pointer when the opaque data is request. I tested it without issues on the release candidate, and it makes it so every object can have an inline dictionary if they want. So we should just press them for that solution in 3.14. Once they go to a grand unified object model we will use the user data slot. No need to redo jpype in the meantime. (It took me months to perform the testing and research to create this model in the first place.) Also don't dismiss the overhead of the cost of name binding. As it is currently we do a few if statements and we immediately get a type slot we can test. The previous solution was the "dictionary option" in which we forced a managed dictionary on all of the types we created and then inserted a capsule in the dictionary with the value. The lookups for binding were 20+ times slower as though it seems trivial to access the dictionary and find the slot then ask for the capsule information, every one of those required many steps each of which had many guard conditions. (where is the dict pointer located in the object, is the dict null, is the key slot located, is the capsule null, is the type a capsule, does the capsule type match the requested key.) We are currently running nearly at the limit of the Python/JNI transfer rate which took a long time to achieve. PS: It isn't so simple even if we did everything above board to work on many versions of Python. Every version has bugs/idiosyncrasies like the Init functions that forgot to increment the ref count on the parent type, so you won't escape testing every version. If there is a corruption problem in one version it can be a man month of time tracing to the source. Hence I am very loath to performing a rewrite. |
At this point I'm going to continue on anyway at least until I understand the root of the problem. I don't see how an additional dereference into the type to get the |
Good news. I managed to get a version that works on all available versions of Python which does not use |
Getting a JClass is fast, but remember there is no guarantee that it is a JClass. I used to test using IsInstance but that turned out to be order O(n) which given the rather large inheretance tree of Java was problematic. (I implemented fast isinstance and tried to get it added to python but nevet got traction.) Internally Python cheats by using a bit flag. The closest way to make a fast check was to set something in the slots that can't be defeated. Method testing was the easiest solution. The finalizer slot which we must have to clean up the jvalue reference is thus the best fast check we can make for a JClass. |
What I was thinking about doing, since all the jpype types would be moved into c++, is creating a large char[] of all the jpype type names aligned to a boundary. Ten pointers into the table would be used for Even if it doesn't work out with |
Doesn't that mean the data must grow with every new type that is imported from java. That seems rather expensive. If you are going that way it is better to look at the
Notice that all java types have |
Nope. It would only be the jpype types and not instances of them. It would just be a static table in read-only memory. For instances of JClass it would at most require an additional dereference to get the JClass type. |
Then I am not quite sure I understand the solution.
If you want to test a String instance then the
though I don't see how that is any different than
|
This is just an example, it is incomplete. I only put enough to present the idea. alignas(0x100) static const char NAME_TABLE[] = {
/* 0x00 */ '_', 'j', 'p', 'y', 'p', 'e', '.', '_', 'J', 'C', 'l', 'a', 's', 's', 0x0, 0x0,
/* 0x10 */ '_', 'j', 'p', 'y', 'p', 'e', '.', 'J', 'B', 'o', 'o', 'l', 'e', 'a', 'n', 0x0,
/* 0x20 */ '_', 'j', 'p', 'y', 'p', 'e', '.', 'J', 'B', 'y', 't', 'e', 0x0, 0x0, 0x0, 0x0,
/* 0x30 */ '_', 'j', 'p', 'y', 'p', 'e', '.', 'J', 'S', 'h', 'o', 'r', 't', 0x0, 0x0, 0x0,
/* 0x40 */ '_', 'j', 'p', 'y', 'p', 'e', '.', 'J', 'I', 'n', 't', 0x0, 0x0, 0x0, 0x0, 0x0,
/* 0x50 */ '_', 'j', 'p', 'y', 'p', 'e', '.', 'J', 'L', 'o', 'n', 'g', 0x0, 0x0, 0x0, 0x0,
/* 0x60 */ '_', 'j', 'p', 'y', 'p', 'e', '.', '_', 'J', 'A', 'r', 'r', 'a', 'y', 0x0, 0x0,
/* 0x70 */ '_', 'j', 'p', 'y', 'p', 'e', '.', 'J', 'F', 'l', 'o', 'a', 't', 0x0, 0x0, 0x0,
/* 0x80 */ '_', 'j', 'p', 'y', 'p', 'e', '.', 'J', 'D', 'o', 'u', 'b', 'l', 'e', 0x0, 0x0,
};
const char *const JClassName = NAME_TABLE + 0x00;
const char *const JBooleanName = NAME_TABLE + 0x10;
const char *const JByteName = NAME_TABLE + 0x20;
const char *const JShortName = NAME_TABLE + 0x30;
const char *const JIntName = NAME_TABLE + 0x40;
const char *const JLongName = NAME_TABLE + 0x50;
const char *const JArrayName = NAME_TABLE + 0x60;
const char *const JFloatName = NAME_TABLE + 0x70;
const char *const JDoubleName = NAME_TABLE + 0x80;
const uintptr_t NAME_TABLE_START = (uintptr_t) NAME_TABLE;
const uintptr_t NAME_TABLE_END = NAME_TABLE_START + sizeof(NAME_TABLE);
PyTypeObject *PyJPJint_Type = nullptr;
PyType_Spec numberJintSpec = {
JIntName,
0,
0,
Py_TPFLAGS_DEFAULT,
0
};
void sample(PyObject* module) {
PyObject *bases;
bases = PyTuple_Pack(1, &PyJPNumberLong_Type);
PyJPJint_Type = (PyTypeObject*) PyJPClass_FromSpecWithBases(&numberJintSpec, bases);
Py_DECREF(bases);
JP_PY_CHECK(); // GCOVR_EXCL_LINE
PyJPJint_Type->tp_name = JIntName;
// alias this in types.py
PyModule_AddObject(module, "JInt", (PyObject*) PyJPJint_Type);
JP_PY_CHECK(); // GCOVR_EXCL_LINE
}
enum class PyJPTypeKind {
CLASS,
BOOLEAN,
BYTE,
SHORT,
INT,
LONG,
ARRAY,
FLOAT,
DOUBLE,
INVALID
};
static inline bool isJavaType(PyTypeObject *type) {
if (type == nullptr) {
return false;
}
uintptr_t addr = (uintptr_t) type->tp_name;
return addr >= NAME_TABLE_START && addr < NAME_TABLE_END;
}
// place somewhere it can be inlined
// this gives you something you can switch over or use as an array index
inline PyJPTypeKind getJavaTypeKind(PyObject *self) {
PyTypeObject *type = Py_TYPE(self);
if (!isJavaType(type)) {
type = Py_TYPE(type);
}
if (!isJavaType(type)) {
return PyJPTypeKind::INVALID;
}
return PyJPTypeKind((((uintptr_t)type->tp_name) >> 4) & 0xf);
} |
I am still not sure that this solves a problem. All calls to get a slot are on a derived class so tp_name will never be one of those checked for. When the derive class is created you can set the offset of in a regular place like the m_Doc field. So sorting it out isnt the issue. The problem is that some types are varible size and others have incompatible basicsizes. This makes it such the describing the slot location depends on the object if after the var section. And challenging to build in the first place if in basicsize. Java insists that Exception and Boxed types come from Object. Python insists those classes can't change the basicsize. I recommend starting the same way I did by creating a dummy Python module that inherits each of the base types in the order that Java requires. After a few months of playing I gave up and went with the allocator solution. It was the only solution that maintained the isinstance checks. Perhaps you will find differently. |
I did consider the solution where the object tree was flat such the classes did not actually derive from each other and all gave fake isinstance and subclass checks. Though technically that passes, it was complex on the C backed. |
python/cpython#123947
Of course it is not mentioned in the documentation for
PyType_FromMetaclass
where the deprecation is documented, but I think you are supposed to use__call__
. I'm not 100% sure but I think simply switching to using__call__
inPyJPClass_hook
instead of directly usingPyJPClass_Type->tp_new
should prevent this from being a problem for jpype. IfPyJPClass_Type
still has thetp_new
slot it will raise aTypeError
when callingPyType_FromSpecWithBases
. I can't actually tell for sure if this will be a problem or if the current hacks just completely circumvent it.jpype/native/python/pyjp_class.cpp
Line 1204 in 66f8c6c
The text was updated successfully, but these errors were encountered: