Merge pull request #887 from volatilityfoundation/release/v2.4.0

Prepare for the 2.4.0 release, the major version has jumped a few numbers for compatibility, but this is the next release including the following:

    New plugins
        linux.mountinfo
        linux.psaux
        windows.devicetree
        windows.joblinks
        windows.ldrmodules
        windows.mbrscan
        windows.mftscan
        windows.sessions
    Introduced the concept of modules and module requirements
    Unified symbol handling and ISF file caching between OS versions
    Better QEVM support (fixed the QEMU PCI hole)
    Exposed an API for automatic PDB symbol table use
    Improved contributed documentation
    Various bug fixes and changes across the codebase

.github/workflows/build-pypi.yml

@@ -15,14 +15,16 @@ on:
 jobs:
 
   build:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-20.04
+    strategy:
+      matrix:
+        python-version: ["3.6"]
     steps:
-    - uses: actions/checkout@v2
-
-    - name: Set up Python 3.x
-      uses: actions/setup-python@v2
+    - uses: actions/checkout@v3
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v4
       with:
-        python-version: '3.x'
+        python-version: ${{ matrix.python-version }}
 
     - name: Install dependencies
       run: |

.github/workflows/test.yaml

@@ -0,0 +1,55 @@
+name: Test Volatility3
+on: [push, pull_request]
+jobs:
+
+  build:
+    runs-on: ubuntu-20.04
+    strategy:
+      matrix:
+        python-version: ["3.6"]
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v4
+      with:
+        python-version: ${{ matrix.python-version }}
+
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install Cmake
+        pip install setuptools wheel
+        pip install -r ./test/requirements-testing.txt
+
+    - name: Build PyPi packages
+      run: |
+        python setup.py sdist --formats=gztar,zip
+        python setup.py bdist_wheel
+
+    - name: Download images
+      run: |
+        curl -sLO "https://downloads.volatilityfoundation.org/volatility3/images/linux-sample-1.bin.gz"
+        gunzip linux-sample-1.bin.gz
+        curl -sLO "https://downloads.volatilityfoundation.org/volatility3/images/win-xp-laptop-2005-06-25.img.gz"
+        gunzip win-xp-laptop-2005-06-25.img.gz
+
+    - name: Download and Extract symbols
+      run: |
+        cd ./volatility3/symbols
+        curl -sLO https://downloads.volatilityfoundation.org/volatility3/symbols/linux.zip
+        unzip linux.zip
+        cd -
+
+    - name: Testing...
+      run: |
+        py.test ./test/test_volatility.py --volatility=vol.py --image win-xp-laptop-2005-06-25.img -k test_windows -v
+        py.test ./test/test_volatility.py --volatility=vol.py --image linux-sample-1.bin -k test_linux -v
+
+    - name: Clean up post-test
+      run: |
+        rm -rf *.lime
+        rm -rf *.img
+        cd volatility3/symbols
+        rm -rf linux
+        rm -rf linux.zip
+        cd -

.gitignore

@@ -27,3 +27,18 @@ config*.json
 # Pyinstaller files
 build
 dist
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+
+# Memory dump files
+*.dmp
+*.vmem
+*.img
+
+# PyTest cache files
+.pytest_cache/

.style.yapf

@@ -107,7 +107,7 @@ each_dict_entry_on_separate_line=True
 i18n_comment=
 
 # The i18n function call names. The presence of this function stops
-# reformattting on that line, because the string it has cannot be moved
+# reformatting on that line, because the string it has cannot be moved
 # away from the i18n comment.
 i18n_function_call=
 

API_CHANGES.md

@@ -4,6 +4,41 @@ API Changes
 When an addition to the existing API is made, the minor version is bumped.
 When an API feature or function is removed or changed, the major version is bumped.
 
+2.4.0
+=====
+Add a `get_size()` method to Windows VAD structures and fix several off-by-one issues when calculating VAD sizes.
+
+2.3.1
+=====
+Update in the windows `_EPROCESS.owning_process` method to support Windows Vista and later versions.
+
+2.3.0
+=====
+Add in `child_template` to template class
+
+2.2.0
+=====
+Changes to linux core calls
+
+2.1.0
+=====
+Add in the linux `task.get_threads` method to the API.
+
+2.0.3
+=====
+Add in the windows `DEVICE_OBJECT.get_attached_devices` and `DRIVER_OBJECT.get_devices` methods to the API.
+
+2.0.2
+=====
+Fix the behaviour of the offsets returned by the PDB scanner.
+
+2.0.0
+=====
+Remove the `symbol_shift` mechanism, where symbol tables could alter their own symbols.
+Symbols from a symbol table are now always the offset values.  They can be added to a Module
+and when symbols are requested from a Module they are shifted by the module's offset to get
+an absolute offset.  This can be done with `Module.get_absolute_symbol_address` or as part of
+`Module.object_from_symbol(absolute = False, ...)`.
 
 1.2.0
 =====

LICENSE.txt

@@ -31,7 +31,7 @@ If you make any Additions available to others, such as by providing copies of th
 - You are responsible to ensure you have rights in Additions necessary to comply with this section.
 
 Contributing
-If you contribute (or offer to contribute) any materials to Volatility Foundation for the software, such as by submitting a pull request to the repository for the software or related content run by Volatility Foundation, you agree to contribute them under the under the BSD 2-Clause Plus Patent License (in the case of software) or the Creative Commons Zero Public Domain Dedication (in the case of content), unless you clearly mark them "Not a Contribution."
+If you contribute (or offer to contribute) any materials to Volatility Foundation for the software, such as by submitting a pull request to the repository for the software or related content run by Volatility Foundation, you agree to contribute them under the BSD 2-Clause Plus Patent License (in the case of software) or the Creative Commons Zero Public Domain Dedication (in the case of content), unless you clearly mark them "Not a Contribution."
 
 Trademarks
 This license grants you no rights to any trademarks or service marks.

MANIFEST.in

@@ -1,6 +1,6 @@
 prune development
 include * .*
-include doc/make.bat doc/Makefile
+include doc/make.bat doc/Makefile doc/requirements.txt
 recursive-include doc/source *
 recursive-include volatility3 *.json
 recursive-exclude doc/source volatility3.*.rst

README.md

@@ -94,6 +94,9 @@ Symbol tables zip files must be placed, as named, into the `volatility3/symbols`
 
 Windows symbols that cannot be found will be queried, downloaded, generated and cached.  Mac and Linux symbol tables must be manually produced by a tool such as [dwarf2json](https://github.com/volatilityfoundation/dwarf2json).
 
+Important: The first run of volatility with new symbol files will require the cache to be updated.  The symbol packs contain a large number of symbol files and so may take some time to update!
+However, this process only needs to be run once on each new symbol file, so assuming the pack stays in the same location will not need to be done again.  Please also note it can be interrupted and next run will restart itself.
+
 Please note: These are representative and are complete up to the point of creation for Windows and Mac.  Due to the ease of compiling Linux kernels and the inability to uniquely distinguish them, an exhaustive set of Linux symbol tables cannot easily be supplied.
 
 ## Documentation

development/stock-linux-json.py

@@ -87,7 +87,7 @@ def process_files(self, named_files: Dict[str, str]):
         output_filename = 'unknown-kernel.json'
         for named_file in named_files:
             prefix = '--system-map'
-            if not 'System' in named_files[named_file]:
+            if 'System' not in named_files[named_file]:
                 prefix = '--elf'
                 output_filename = './' + '-'.join((named_file.split('/')[-1]).split('-')[2:])[:-4] + '.json.xz'
             args += [prefix, named_files[named_file]]

doc/requirements.txt

@@ -1,4 +1,4 @@
 # These packages are required for building the documentation.
-sphinx>=1.8.2
+sphinx>=4.0.0
 sphinx_autodoc_typehints>=1.4.0
-sphinx-rtd-theme>=0.4.3
+sphinx-rtd-theme>=0.4.3

doc/source/complex-plugin.rst

@@ -300,7 +300,7 @@ This will mean that when a specific structure is loaded from the symbol_space, i
 `StructType`, but instead is instantiated using the NewStructureClass, meaning new methods can be called directly on it.
 
 If the situation really calls for an entirely new object, that isn't covered by one of the existing
-:py:class:`~volatility3.framework.objects.PrimativeObject` objects (such as
+:py:class:`~volatility3.framework.objects.PrimitiveObject` objects (such as
 :py:class:`~volatility3.framework.objects.Integer`,
 :py:class:`~volatility3.framework.objects.Boolean`,
 :py:class:`~volatility3.framework.objects.Float`,

doc/source/conf.py

@@ -84,6 +84,15 @@ def setup(app):
                     for line in submodule_lines:
                         contents.write(line.replace(b'volatility3.framework.plugins', b'volatility3.plugins'))
 
+    # Clear up the framework.plugins page
+    with open(os.path.join(os.path.dirname(__file__), 'volatility3.framework.plugins.rst'), "rb") as contents:
+        real_lines = contents.readlines()
+
+    with open(os.path.join(os.path.dirname(__file__), 'volatility3.framework.plugins.rst'), "wb") as contents:
+        for line in real_lines:
+            if b'volatility3.framework.plugins.' not in line:
+                contents.write(line)
+
 
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
@@ -102,9 +111,11 @@ def setup(app):
 # ones.
 extensions = [
     'sphinx.ext.autodoc', 'sphinx.ext.doctest', 'sphinx.ext.napoleon', 'sphinx.ext.intersphinx', 'sphinx.ext.todo',
-    'sphinx.ext.coverage', 'sphinx.ext.viewcode'
+    'sphinx.ext.coverage', 'sphinx.ext.viewcode', 'sphinx.ext.autosectionlabel'
 ]
 
+autosectionlabel_prefix_document = True
+
 try:
     import sphinx_autodoc_typehints
 

doc/source/development.rst

@@ -0,0 +1,8 @@
+Writing Plugins
+===============
+
+.. toctree::
+
+    simple-plugin
+    complex-plugin
+    using-as-a-library