Attempt to unmarshal headerless bytecode

[deliciouslytyped]

I ran into a situation where I needed to decompile serialized code objects (marshal.dumps), however the uncompyle cli operates on pyc files, and I was doing something wrong when I tried to dump the code as if it was a pyc (I suppose I didn't try hard enough, but it wasn't needed anyway).

The error:

Traceback (most recent call last):
  File "/nix/store/9vs1idrhqysdc961rd2w3qw3pfjzh8bf-python3.7-xdis-4.0.3/lib/python3.7/site-packages/xdis/load.py", line 208, in load_module_from_file_object
    co = marshal.loads(bytecode)
ValueError: bad marshal data (unknown type code)
Traceback (most recent call last):
  File "/nix/store/9vs1idrhqysdc961rd2w3qw3pfjzh8bf-python3.7-xdis-4.0.3/lib/python3.7/site-packages/xdis/load.py", line 208, in load_module_from_file_object
    co = marshal.loads(bytecode)
ValueError: bad marshal data (unknown type code)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nix/store/qhavgb609vv920aqk4y897va35nz6fsd-python3.7-uncompyle6-3.3.3/bin/.uncompyle6-wrapped", line 11, in <module>
    sys.exit(main_bin())
  File "/nix/store/qhavgb609vv920aqk4y897va35nz6fsd-python3.7-uncompyle6-3.3.3/lib/python3.7/site-packages/uncompyle6/bin/uncompile.py", line 179, in main_bin
    **options)
  File "/nix/store/qhavgb609vv920aqk4y897va35nz6fsd-python3.7-uncompyle6-3.3.3/lib/python3.7/site-packages/uncompyle6/main.py", line 248, in main
    linemap_stream, do_fragments)
  File "/nix/store/qhavgb609vv920aqk4y897va35nz6fsd-python3.7-uncompyle6-3.3.3/lib/python3.7/site-packages/uncompyle6/main.py", line 148, in decompile_file
    source_size) = load_module(filename, code_objects)
  File "/nix/store/9vs1idrhqysdc961rd2w3qw3pfjzh8bf-python3.7-xdis-4.0.3/lib/python3.7/site-packages/xdis/load.py", line 116, in load_module
    get_code=get_code,
  File "/nix/store/9vs1idrhqysdc961rd2w3qw3pfjzh8bf-python3.7-xdis-4.0.3/lib/python3.7/site-packages/xdis/load.py", line 222, in load_module_from_file_object
    "Ill-formed bytecode file %s\n%s; %s" % (filename, kind, msg)
ImportError: Ill-formed bytecode file webview.pyc
<class 'ValueError'>; bad marshal data (unknown type code)

My code:

#https://stackoverflow.com/questions/47967298/compile-python-code-to-pycodeobject-and-marshal-to-pyc-file
def c2b(code, mtime=0, source_size=0):
    """Compile a code object into bytecode for writing out to a byte-compiled
    file."""
    import importlib.util as u
    import importlib as i
    data = bytearray(u.MAGIC_NUMBER)
    data.extend(i._w_long(mtime))
    data.extend(i._w_long(source_size))
    data.extend(marshal.dumps(code))
    return data

It would have been great if uncompyle can ingest a raw bytecode file with a specified version;
what I ended up doing was importing uncompyle.main and calling decompile(...) manually with the code object as an argument, and it worked great.

It could be useful if uncompyle can brute force / autodetect an unknown bytecode version, but I don't know how commonly this is necessary in practice.

if __name__ == "__main__":
  from uncompyle6.main import decompile
  module = ...
  loader = Loader(module, sys.argv[1])
  code = loader.get_code(module)
  decompile(None, code, sys.stdout)

This way, code files can be dumped to disk and uncompyle can be run with the usual flags and interface recursively.

[deliciouslytyped]

FWIW I'm pretty sure the error was from the somewhat outdated host python version and/or an outdated xdis.
After using the right interpreter version and a fresh venv with latest versions a few files that were erroring for me now work don't have this error.

[rocky]

Hi - Glad you figured out a way around your problem. I always appreciate feedback on how uncompyle is working.
Thanks for the suggestion about autodetecting unknown bycode versions. I've thought about this and mentioned it to other people, and it would be a good thing to have. If you want put in a pull request and work something up, that would be great. Otherwise, please close the issue.

uncompyle is largely a one-person project, done out of interest and in my spare time. There are many issues that come up and the order that I can address them in is somewhat arbitrary. Also, I'm working on multiple projects, so it might take years before I get to any one particular issue. I'm in the process of putting together a system so that people who want to support me can get priority.

Again, thanks for the feedback.

[deliciouslytyped]

I think the simplest method to start with would just be to add a flag that will try every xdis version and see what doesn't error? This way no cleverness is needed.

[rocky]

It is true that the place to address this is in xdis and adjust this to make use of the additional feature.

And there really isn't any no cleverness needed. I am happy to wait until someone wants to do this properly or there is funding for the endeavor.

[rocky]

Closing here since we are not fixing here.