Attempt to add 64 bit support to sdktools

[bottiger1]

No description provided.

[bottiger1]

@dvander @psychonic @asherkin

pinging the sourcemod core devs here so they can say if they are already working on this or that they don't like this solution before I spend any more time on this although I think it is pretty much done.

I'm introducing a new sdktools type called SDKType_Pointer which will takes 1 int on 32 bits and 2 int on 64 bits. Since sourcepawn can't return a 64 bit value as far as I know, this is why I introduce a new type instead of just fixing the POD type.

When you set it as a return value you need to put the return value as a parameter since it could be 64 bits.

Usage example with tf2attributes gamedata:

    Handle hGameConf = LoadGameConfigFile("tf2.attributes");
    
    Handle hSDKSchema;
    Handle hSDKGetAttributeDef;

    StartPrepSDKCall(SDKCall_Static);
    PrepSDKCall_SetFromConf(hGameConf, SDKConf_Signature, "GEconItemSchema");
    PrepSDKCall_SetReturnInfo(SDKType_Pointer, SDKPass_Plain); //Returns address of CEconItemSchema
    hSDKSchema = EndPrepSDKCall();
    if (!hSDKSchema) {
        SetFailState("Could not initialize call to GEconItemSchema");
    }

    StartPrepSDKCall(SDKCall_Raw);
    PrepSDKCall_SetFromConf(hGameConf, SDKConf_Signature, "CEconItemSchema::GetAttributeDefinition");
    PrepSDKCall_AddParameter(SDKType_PlainOldData, SDKPass_Plain);
    PrepSDKCall_SetReturnInfo(SDKType_Pointer, SDKPass_Plain); //Returns address of a CEconItemAttributeDefinition
    hSDKGetAttributeDef = EndPrepSDKCall();
    if (!hSDKGetAttributeDef) {
        SetFailState("Could not initialize call to CEconItemSchema::GetAttributeDefinition");
    }    

    int pSchema[2];
    SDKCall(hSDKSchema, pSchema);
    PrintToServer("pSchema %x %x", pSchema[0], pSchema[1]);

    int pAttr[2];
    SDKCall(hSDKGetAttributeDef, pSchema, pAttr, 1);
    PrintToServer("pAttr %x %x", pAttr[0], pAttr[1]);

[KyleSanderson]

Have you reviewed #705 ? It's nearly 7 years old but should help avoid what's being proposed with high and low bits.

[bottiger1]

Have you reviewed #705 ? It's nearly 7 years old but should help avoid what's being proposed with high and low bits.

I didn't review it. How will it help if it is already merged? Is there something specific you want to point out? I don't want to look through 90 files worth of changes.

[KyleSanderson]

Have you reviewed #705 ? It's nearly 7 years old but should help avoid what's being proposed with high and low bits.

I didn't review it. How will it help if it is already merged? Is there something specific you want to point out? I don't want to look through 90 files worth of changes.

hmmm...

The other notable change here is using what I call "pseudo addresses" on 64-bit platforms in any place that a native returns or is passed a memory address. A pseudo address consists of a 6-bit table index and a 26-bit offset. The table consists of base addresses to memory allocations. Using this kind of scheme allows some pointer arithmetic to be performed by a plugin. See core/logic/PseudoAddrManager.cpp for further details. I'll note that using dladdr was not an option here because GetEntityAddress for example returns addresses to things on the heap. The choice of 6 and 26 bits is kind of arbitrary, so we could adjust this as we see fit.

Is there something missing?

[bottiger1]

pseudoaddress is already being used and supposedly does not work on addresses far away from the code (ie heap)

[KyleSanderson]

pseudoaddress is already being used and supposedly does not work on addresses far away from the code (ie heap)

so ... do we need to allocate a new pseudoaddress type, based on your findings? Not trying to be cryptic here, but if we don't have to reinvent anything, and patch 16~ years of plugins that's the better solution, no?

[bottiger1]

It is not possible to invent another pseudoaddress that will fix the problem. You just can't fit 64 bits into 32 bits. If you just use the full 32 bits as the index to a hash table, you can't do pointer arithmetic on it anymore. No matter what combination of bits you use for the index or the offset, it's going to result in problems.

With my changes, you can port plugins to 64 bit and they will work on 32 bits, so it's fully backwards compatible. It just doesn't magically make unchanged plugins work on 64 bits.

Dvander himself said here not to wait for him to add 64 bit ints and to use arrays. So that is what I'm doing. If people want to wait until sourcemod fixes Address so it can be 64 bits transparently, they can wait. Or if they don't want their server to be down for a long time, they can use this. I don't think there are many plugins that use sdktools to get addresses.

Given that dvander said not to wait, I get the feeling that a stopgap measure like this is needed.

[bottiger1]

Can I get a decision on whether this will be approved or not or are we waiting for dvander to add 64 bit ints to sourcepawn? This is 1 of the last 3 things needed for full 64 bit support.

[Kenzzer]

Not here to comment on the decision. But I'd suggest you fix the diff change on the PR, as it's currently impossible to review what you've added or removed. And you should delete that duplicate change in sdktools, as you already have another PR #2152 adding that.

[bottiger1]

But I'd suggest you fix the diff change on the PR

I'm not sure what you mean. It looks pretty clean to me.

https://github.com/alliedmodders/sourcemod/pull/2159/files

Don't click on the force-push I did, because that was just me rebasing the branch to the recently merged changes.

I just want to hear a simple yes or no whether this solution is acceptable or we are waiting until dvander to make major changes to sourcepawn can return 64 bit values (even though he said not to). If it isn't, there's no point in doing more work on this like making the commits perfect.

I need the fixes I made from the other pull request so my server can run without crashing.

[Kenzzer]

I'm not sure what you mean. It looks pretty clean to me.

It's really not though
Edit: Yeah it's fine if you hide whitespace change, but still I think it's best if it's fixed. But anyways that just a suggestion to make things cleaner looking

[bottiger1]

yeah as you found out you didn't have whitespace changes hidden. I am not sure how that happened, but there are quite a few files in the project already with different line ending styles.

[KaelaSavia]

any ETA on merging? as int64 type is not coming to sourcepawn anytime soon and pseudoaddresses are gonna cause more trouble than not with heap, it seems like there's no better solution at the hand

[KaelaSavia]

Apologies for double post but we decided to compile SourceMod branch with this PR merged and run on our servers. So far it works great (tf2attributes, tf2 taunt'em, etc.) although was kinda confusing when to use SDKType_Pointer and when not.

But that's more of skill issue on our side

[dedimark]

any plans to merging?