Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed the deletion of ConVar/ConCommand not managed by Source.Python. #426

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

CookStar
Copy link
Contributor

The function ConVar_Unregister unregisters all ConVar/ConCommand and should not be used.

Output:

plugin_print
Loaded plugins:
---------------------
0:      "Source.Python, (C) 2012-2021, Source.Python Team."
1:      "Metamod:Source 1.11.0-dev+1145"
---------------------

meta version
 Metamod:Source Version Information
    Metamod:Source version 1.11.0-dev+1145
    Plugin interface version: 16:14
    SourceHook version: 5:5
    Loaded As: Valve Server Plugin
    Compiled on: Jul 11 2021 22:32:15
    Built from: https://github.com/alliedmodders/metamod-source/commit/0bb53f2
    Build ID: 1145:0bb53f2
    http://www.metamodsource.net/

plugin_unload sourcepython
[Source.Python] Unloading...
[Source.Python] Unloaded successfully.
Unloaded plugin "sourcepython"

plugin_print
Loaded plugins:
---------------------
0:      "Metamod:Source 1.11.0-dev+1145"
---------------------

meta version
Unknown command "meta"

to

plugin_unload sourcepython
[Source.Python] Unloading...
[Source.Python] Unloaded successfully.
Unloaded plugin "sourcepython"

plugin_print
Loaded plugins:
---------------------
0:      "Metamod:Source 1.11.0-dev+1145"
---------------------

meta version
 Metamod:Source Version Information
    Metamod:Source version 1.11.0-dev+1145
    Plugin interface version: 16:14
    SourceHook version: 5:5
    Loaded As: Valve Server Plugin
    Compiled on: Jul 11 2021 22:32:15
    Built from: https://github.com/alliedmodders/metamod-source/commit/0bb53f2
    Build ID: 1145:0bb53f2
    http://www.metamodsource.net/

@jordanbriere
Copy link
Contributor

jordanbriere commented Oct 10, 2021

ConVar_Unregister only unregisters ConCommandBase that matches our DLL identifier. The one that was generated when we initialized our accessor there:

void InitServerCommands()
{
ConVar_Register(0, &g_ConVarAccessor);
}

In fact, it works just as it should on my side:

plugin_print
Loaded plugins:
---------------------
0:      "Source.Python, (C) 2012-2021, Source.Python Team."
1:      "Metamod:Source 1.11.0-dev+1145"
---------------------
meta
Metamod:Source Menu
usage: meta <command> [arguments]
  alias        - List or set an alias
  clear        - Unload all plugins forcefully
  cmds         - Show plugin commands
  cvars        - Show plugin cvars
  credits      - About Metamod:Source
  force_unload - Forcefully unload a plugin
  game         - Information about GameDLL
  info         - Information about a plugin
  list         - List plugins
  load         - Load a plugin
  pause        - Pause a running plugin
  refresh      - Reparse plugin files
  retry        - Attempt to reload a plugin
  unload       - Unload a loaded plugin
  unpause      - Unpause a paused plugin
  version      - Version information
sp
A sub-command is required:
  sp auth <sub-command>                     Authorization specific commands.
  sp credits                                List all credits for Source.Python.
  sp delay <delay:float> <command> [*args]  Execute a command after a given delay.
  sp docs <sub-command>                     Documentation specific commands.
  sp dump <sub-command>                     Dump various data to files.
  sp help [command=None]                    Print all sp sub-commands or help for a
    [*server_sub_commands]                    specific command.
  sp info                                   Print information about OS, SP and
                                              installed plugins.
  sp plugin <sub-command>                   Plugin specific commands.
  sp update                                 Update Source.Python to the latest
                                              version. A restart of the server is
                                              required.
plugin_unload 0
[Source.Python] Unloading...
[Source.Python] Restoring old output function...
[Source.Python] Resetting cache notifier...
[Source.Python] Shutting down python...
[Source.Python] Unloading main module...
Unloading plugins...
Removing entities listener...
Unloading auth...
[Source.Python] Clearing convar changed listener...
[Source.Python] Unhooking all functions...
[Source.Python] Clearing all commands...
[Source.Python] Unregistering ConVar...
[Source.Python] Disconnecting tier2 libraries...
[Source.Python] Disconnecting tier1 libraries...
[Source.Python] Unloaded successfully.
Unloaded plugin "0"
meta
Metamod:Source Menu
usage: meta <command> [arguments]
  alias        - List or set an alias
  clear        - Unload all plugins forcefully
  cmds         - Show plugin commands
  cvars        - Show plugin cvars
  credits      - About Metamod:Source
  force_unload - Forcefully unload a plugin
  game         - Information about GameDLL
  info         - Information about a plugin
  list         - List plugins
  load         - Load a plugin
  pause        - Pause a running plugin
  refresh      - Reparse plugin files
  retry        - Attempt to reload a plugin
  unload       - Unload a loaded plugin
  unpause      - Unpause a paused plugin
  version      - Version information
sp
Unknown command "sp"

@CookStar
Copy link
Contributor Author

It only works because your s_nDLLIdentifier happens to be the same as Source.Python's DLL identifier. There is no guarantee that it will work every time.

ConVar_Unregister is a very dumb function. Even if it works correctly, it will only work once.
https://github.com/alliedmodders/hl2sdk/blob/c33f7155e9aff9573c20d86e10c8158425a3d67a/tier1/convar.cpp#L86-L95

@jordanbriere
Copy link
Contributor

It only works because your s_nDLLIdentifier happens to be the same as Source.Python's DLL identifier. There is no guarantee that it will work every time.

But it's not a coincidence. We assign our s_nDLLIdentifier when we call ConVar_Register. That ID is ours and ours only. Other binaries that register their accessor will generate their own and set their symbol accordingly. The only issue currently with the registration is that the following calls should be reverted:

void InitCommands()
{
// Register the say and say_team commands
DevMsg(1, MSG_PREFIX "Registering say and say_team commands...\n");
RegisterSayCommands();
// Register the ConVar accessor.
DevMsg(1, MSG_PREFIX "Registering ConVar accessor...\n");
InitServerCommands();
}
Our accessor should be registered before we register say and say_team.

Other than that, It's possible there is a linking issue on Linux and that we ends up sharing the same symbol. If the following assert for you:

from cvars import *

assert cvar.find_base('sm').dll_identifier is not SP_CVAR_DLL_IDENTIFIER
assert cvar.find_base('meta').dll_identifier is not SP_CVAR_DLL_IDENTIFIER
assert cvar.find_base('sp').dll_identifier is SP_CVAR_DLL_IDENTIFIER

Then that's likely it. Try to add tier1.a to the excluded libraries:

Set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -Wl,--exclude-libs,libprotobuf.a")

ConVar_Unregister is a very dumb function. Even if it works correctly, it will only work once.

We don't call it multiple times, only on unload, so that isn't really an issue, is it?

Ayuto referenced this pull request Oct 25, 2021
@CookStar
Copy link
Contributor Author

CookStar commented Nov 5, 2021

ConVar and Source SDK is so broken that I feel discouraged.

But it's not a coincidence. We assign our s_nDLLIdentifier when we call ConVar_Register. That ID is ours and ours only. Other binaries that register their accessor will generate their own and set their symbol accordingly.

I understand that, but it's not working in practice.

Then that's likely it. Try to add tier1.a to the excluded libraries:

I tried that too, but it doesn't work.

Test Code:

#   Commands
from commands import ConCommandBase
#   Cvars
from cvars import cvar
from cvars import SP_CVAR_DLL_IDENTIFIER
#   Memory
from memory import get_virtual_function
from memory import make_object
from memory.hooks import PreHook

@PreHook(get_virtual_function(cvar, "RegisterConCommand"))
def pre_register_con_command(args):
    base = make_object(ConCommandBase, args[1])
    if not base.name:
        return

    print("RegisterConCommand")
    print("Name: ", base.name)
    print("Help Text: ", base.help_text)
    print("Is Registered: ", base.is_registered())
    print("Is Command: ", base.is_command())
    print("Address: ", base._ptr().address)
    print("dll_identifier: ", base.dll_identifier)

    print("Is dll_identifier == sp_cvar_dll_identifier: ", base.dll_identifier == SP_CVAR_DLL_IDENTIFIER)

For clarity.
commands/commands_server.cpp

class CPluginConVarAccessor : public IConCommandBaseAccessor
{
public:
	virtual bool RegisterConCommandBase(ConCommandBase* pCommand)
	{
		printf("[Source.Python] RegisterConCommandBase\n");
		if (!g_pCVar->FindCommandBase(pCommand->GetName())) {
			g_pCVar->RegisterConCommand(pCommand);
			printf("[Source.Python] Registered\n");
			return true;
		}
		printf("[Source.Python] Failed\n");

		return false;
	}
};
Output (CS:GO Linux tier1.a excluded with clarity code)
plugin_load addons/source-python
[Source.Python] Loading...
[Source.Python] RegisterConCommandBase
[Source.Python] Registered
[Source.Python] RegisterConCommandBase
[Source.Python] Registered
[Source.Python] RegisterConCommandBase
[Source.Python] Registered
[Source.Python] RegisterConCommandBase
[Source.Python] Registered
[Source.Python] RegisterConCommandBase
[Source.Python] Registered
[Source.Python] RegisterConCommandBase
[Source.Python] Registered
[Source.Python] RegisterConCommandBase
[Source.Python] Registered
[Source.Python] RegisterConCommandBase
[Source.Python] Registered
[Source.Python] RegisterConCommandBase
[Source.Python] Registered
[Source.Python] Loaded successfully.
Loaded plugin "addons/source-python"

sp plugin load test
[SP] Loading plugin 'test'...
[Source.Python] RegisterConCommandBase
[Source.Python] Registered
[SP] Successfully loaded plugin 'test'.

plugin_print
Loaded plugins:
---------------------
0:      "Source.Python, (C) 2012-2021, Source.Python Team."
---------------------

plugin_load ../csgo/addons/metamod/bin/server
[Source.Python] RegisterConCommandBase
RegisterConCommand
Name:  meta
Help Text:  Metamod:Source control options
Is Registered:  False
Is Command:  True
Address:  3813366172
dll_identifier:  11
Is dll_identifier == sp_cvar_dll_identifier:  True
[Source.Python] Registered
[Source.Python] RegisterConCommandBase
[Source.Python] Registered
RegisterConCommand
Name:  meta
Help Text:  Metamod:Source control options
Is Registered:  True
Is Command:  True
Address:  3813366172
dll_identifier:  11
Is dll_identifier == sp_cvar_dll_identifier:  True
[Source.Python] RegisterConCommandBase
RegisterConCommand
Name:  metamod_version
Help Text:  Metamod:Source Version
Is Registered:  False
Is Command:  False
Address:  237287776
dll_identifier:  11
Is dll_identifier == sp_cvar_dll_identifier:  True
[Source.Python] Registered
RegisterConCommand
Name:  metamod_version
Help Text:  Metamod:Source Version
Is Registered:  True
Is Command:  False
Address:  237287776
dll_identifier:  11
Is dll_identifier == sp_cvar_dll_identifier:  True
[Source.Python] RegisterConCommandBase
RegisterConCommand
Name:  mm_pluginsfile
Help Text:  Metamod:Source Plugins File
Is Registered:  False
Is Command:  False
Address:  234871728
dll_identifier:  11
Is dll_identifier == sp_cvar_dll_identifier:  True
[Source.Python] Registered
RegisterConCommand
Name:  mm_pluginsfile
Help Text:  Metamod:Source Plugins File
Is Registered:  True
Is Command:  False
Address:  234871728
dll_identifier:  11
Is dll_identifier == sp_cvar_dll_identifier:  True
[Source.Python] RegisterConCommandBase
RegisterConCommand
Name:  mm_basedir
Help Text:  Metamod:Source Base Folder
Is Registered:  False
Is Command:  False
Address:  235182816
dll_identifier:  11
Is dll_identifier == sp_cvar_dll_identifier:  True
[Source.Python] Registered
RegisterConCommand
Name:  mm_basedir
Help Text:  Metamod:Source Base Folder
Is Registered:  True
Is Command:  False
Address:  235182816
dll_identifier:  11
Is dll_identifier == sp_cvar_dll_identifier:  True
Loaded plugin "../csgo/addons/metamod/bin/server"

plugin_print
Loaded plugins:
---------------------
0:      "Source.Python, (C) 2012-2021, Source.Python Team."
1:      "Metamod:Source 1.11.0-dev+1145"
---------------------

meta
You must change the map to activate Metamod:Source.

plugin_unload 0
[Source.Python] Unloading...
[Source.Python] Unloaded successfully.
Unloaded plugin "0"

sp
Unknown command "sp"

meta
Unknown command "meta"

plugin_print
Loaded plugins:
---------------------
0:      "Metamod:Source 1.11.0-dev+1145"
---------------------
With distributed binaries (CS:GO Linux SP version: 710)
plugin_load addons/source-python
[Source.Python] Loading...
[Source.Python] Loaded successfully.
Loaded plugin "addons/source-python"

sp info

IMPORTANT: Please copy the full output.
--------------------------------------------------------
Checksum      : 6a9c0c874424594cd8a6db604c2701a1
Date          : 2021-11-05 05:51:22.111626
OS            : Linux-4.15.0-112-generic-x86_64-with-debian-buster-sid
Game          : csgo
SP version    : 710
Github commit : 620d6c39e652a7ab9c928663c094a1d23a38d85e
Server plugins:
   00: Source.Python, (C) 2012-2021, Source.Python Team.
SP plugins:
--------------------------------------------------------


sp plugin load test
[SP] Loading plugin 'test'...
[SP] Successfully loaded plugin 'test'.

plugin_load ../csgo/addons/metamod/bin/server
RegisterConCommand
Name:  meta
Help Text:  Metamod:Source control options
Is Registered:  False
Is Command:  True
Address:  3813808540
dll_identifier:  11
Is dll_identifier == sp_cvar_dll_identifier:  True
RegisterConCommand
Name:  meta
Help Text:  Metamod:Source control options
Is Registered:  True
Is Command:  True
Address:  3813808540
dll_identifier:  11
Is dll_identifier == sp_cvar_dll_identifier:  True
RegisterConCommand
Name:  metamod_version
Help Text:  Metamod:Source Version
Is Registered:  False
Is Command:  False
Address:  251280608
dll_identifier:  11
Is dll_identifier == sp_cvar_dll_identifier:  True
RegisterConCommand
Name:  metamod_version
Help Text:  Metamod:Source Version
Is Registered:  True
Is Command:  False
Address:  251280608
dll_identifier:  11
Is dll_identifier == sp_cvar_dll_identifier:  True
RegisterConCommand
Name:  mm_pluginsfile
Help Text:  Metamod:Source Plugins File
Is Registered:  False
Is Command:  False
Address:  250035296
dll_identifier:  11
Is dll_identifier == sp_cvar_dll_identifier:  True
RegisterConCommand
Name:  mm_pluginsfile
Help Text:  Metamod:Source Plugins File
Is Registered:  True
Is Command:  False
Address:  250035296
dll_identifier:  11
Is dll_identifier == sp_cvar_dll_identifier:  True
RegisterConCommand
Name:  mm_basedir
Help Text:  Metamod:Source Base Folder
Is Registered:  False
Is Command:  False
Address:  250119536
dll_identifier:  11
Is dll_identifier == sp_cvar_dll_identifier:  True
RegisterConCommand
Name:  mm_basedir
Help Text:  Metamod:Source Base Folder
Is Registered:  True
Is Command:  False
Address:  250119536
dll_identifier:  11
Is dll_identifier == sp_cvar_dll_identifier:  True
Loaded plugin "../csgo/addons/metamod/bin/server"

plugin_print
Loaded plugins:
---------------------
0:      "Source.Python, (C) 2012-2021, Source.Python Team."
1:      "Metamod:Source 1.11.0-dev+1145"
---------------------

meta
You must change the map to activate Metamod:Source.

plugin_unload 0
[Source.Python] Unloading...
[Source.Python] Unloaded successfully.
Unloaded plugin "0"

sp
Unknown command "sp"

meta
Unknown command "meta"

plugin_print
Loaded plugins:
---------------------
0:      "Metamod:Source 1.11.0-dev+1145"
---------------------
Switch the plugin load order (CS:GO Linux tier1.a excluded)
plugin_print
Loaded plugins:
---------------------
0:      "Metamod:Source 1.11.0-dev+1145"
1:      "Source.Python, (C) 2012-2021, Source.Python Team."
---------------------

sp info

IMPORTANT: Please copy the full output.
--------------------------------------------------------
Checksum      : 41b2b930d883739c112f418398fa631e
Date          : 2021-11-05 06:22:14.197332
OS            : Linux-4.15.0-112-generic-x86_64-with-debian-buster-sid
Game          : csgo
SP version    : None
Github commit : None
Server plugins:
   00: Metamod:Source 1.11.0-dev+1145
   01: Source.Python, (C) 2012-2021, Source.Python Team.
SP plugins:
   00: test
--------------------------------------------------------

meta load addons/stripper/bin/stripper_mm
RegisterConCommand
Name:  stripper_cfg_path
Help Text:  Stripper Config Path
Is Registered:  False
Is Command:  False
Address:  3958717748
dll_identifier:  11
Is dll_identifier == sp_cvar_dll_identifier:  True
RegisterConCommand
Name:  stripper_current_file
Help Text:  Stripper for current map
Is Registered:  False
Is Command:  False
Address:  3958717836
dll_identifier:  11
Is dll_identifier == sp_cvar_dll_identifier:  True
RegisterConCommand
Name:  stripper_next_file
Help Text:  Stripper for next map
Is Registered:  False
Is Command:  False
Address:  3958717924
dll_identifier:  11
Is dll_identifier == sp_cvar_dll_identifier:  True
RegisterConCommand
Name:  stripper_file_lowercase
Help Text:  Load stripper configs in lowercase
Is Registered:  False
Is Command:  False
Address:  3958718012
dll_identifier:  11
Is dll_identifier == sp_cvar_dll_identifier:  True
RegisterConCommand
Name:  stripper_version
Help Text:  Stripper Version
Is Registered:  False
Is Command:  False
Address:  3958718900
dll_identifier:  11
Is dll_identifier == sp_cvar_dll_identifier:  True
RegisterConCommand
Name:  stripper_dump
Help Text:  Dumps the map entity list to a file
Is Registered:  False
Is Command:  True
Address:  3958718988
dll_identifier:  11
Is dll_identifier == sp_cvar_dll_identifier:  True
InstallChangeCallback ignoring duplicate change callback!!!
Plugin "Stripper" loaded with id 1.

meta list
Listing 1 plugin:
  [01] Stripper (1.2.2) by BAILOPAN

plugin_unload 1
[Source.Python] Unloading...
[Source.Python] Unloaded successfully.
Unloaded plugin "1"

sp
Unknown command "sp"

stripper_cfg_path
Unknown command "stripper_cfg_path"

Since this problem affects all ConVars, there is no choice other than to fix it, but #421 and #430 are covered by this problem, which makes things even more complicated.

@@ -409,9 +409,6 @@ void CSourcePython::Unload( void )
DevMsg(1, MSG_PREFIX "Clearing all commands...\n");
ClearAllCommands();

DevMsg(1, MSG_PREFIX "Unregistering ConVar...\n");
ConVar_Unregister( );

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please replace with:

	// See issue #430.
	// DevMsg(1, MSG_PREFIX "Unregistering ConVar...\n");
	// ConVar_Unregister( );

@CookStar
Copy link
Contributor Author

CookStar commented Dec 3, 2021

@jordanbriere Can you confirm this issue on your side?

Please replace with:

	// See issue #430.
	// DevMsg(1, MSG_PREFIX "Unregistering ConVar...\n");
	// ConVar_Unregister( );

Since the root cause of the problem could not be determined, it was neglected, but it's actually not that simple.

from cvars import SP_CVAR_DLL_IDENTIFIER
from cvars import cvar

def get_bases():
    bases = list()

    base = cvar.commands
    while base is not None:
        bases.append((base.dll_identifier, base.name, base.is_command()))
        base = base.next

    return bases

bases = get_bases()
bases.sort(key=lambda x:x[0])
for base in bases:
    identifier, name, is_command = base
    if identifier < 0 or identifier >= SP_CVAR_DLL_IDENTIFIER:
        print(identifier, name, is_command)

In this code, you can see if SP_CVAR_DLL_IDENTIFIER overlaps with the dll_identifier of other metamod plugins. If you unload a metamod plugin with overlapping dll_identifier, the ConCommand will be released without being unregistered, causing a segmentation fault in the code as shown earlier(OR find <string> console command!).

If this is not a Source.Python problem, but a MetaMod problem, I'll just have to apply the earlier fix and see what happens, but causing a clear crash is a problem.

@CookStar
Copy link
Contributor Author

CookStar commented Dec 3, 2021

In my environment, the only way to get around this reliably is to load Source.Python after metamod has finished loading.
Loading with autoexec.cfg is not a workaround for this.

This is just my guess, but it looks like there is a delay in the loading of MetaMod, and Source.Python is catching up and hijacking the ConCommand while MetaMod is loading.

@jordanbriere
Copy link
Contributor

To be honest, I don't think we should care. There is no point into trying to make a non-issue an issue. I mean, so long as loading SM or SP plugins at run-time works, we should not bother if there are issues unloading MM/SM/SP themselves because they can all be fixed with a server reboot and the fact nobody ever reported issues related to this means nobody really does unload and load them on the fly. I'd say make the changes so we know why it was done, and leave it at that. If this becomes a recurrent issue for users then let's address it then.

@CookStar
Copy link
Contributor Author

CookStar commented Dec 3, 2021

To be honest, I don't think we should care. There is no point into trying to make a non-issue an issue. I mean, so long as loading SM or SP plugins at run-time works, we should not bother if there are issues unloading MM/SM/SP themselves because they can all be fixed with a server reboot and the fact nobody ever reported issues related to this means nobody really does unload and load them on the fly. I'd say make the changes so we know why it was done, and leave it at that. If this becomes a recurrent issue for users then let's address it then.

It's not just a matter of loading/unloading MetaMod/SourceMod/Source.Python itself.
MetaMod supports reloading of MetaMod plugins, and so does SourceMod's Extension system.

The reason this issue is not more obvious is that problems caused by reloading the MetaMod plugin or SourceMod Extension do not lead to an immediate crash, and even if they do, it is not clear how they are related to Source.Python.

@jordanbriere
Copy link
Contributor

It's not just a matter of loading/unloading MetaMod/SourceMod/Source.Python itself.
MetaMod supports reloading of MetaMod plugins, and so does SourceMod's Extension system.

Estimated population that does so: 0.0000005%
Estimated scenarios where we share the same custom ConVar/ConCommand: 0.00000002%

I know you like to break stuff... 😄 But this is a non-issue in my opinion. At least, not until it becomes one. THere is really no point into focusing on stuff that is unlikely to happens or that never has been reported. If many users come forward and report issues related to this, then it can be addressed at that time but for now, trying to profile and debug this is a waste of time to me,

@CookStar
Copy link
Contributor Author

CookStar commented Dec 3, 2021

At least, not until it becomes one. THere is really no point into focusing on stuff that is unlikely to happens or that never has been reported.

Edited:
This crash is already occurring on other users, it's just not reported here.

then it can be addressed at that time but for now, trying to profile and debug this is a waste of time to me,

I'm working on a fix for the problem in ConVar/ConCommand, I don't know the cause of this problem, but please wait to close this issue as I'm figuring out the workaround.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants