Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add +LINUX and +WINDOWS doctest options #2507

Merged
merged 11 commits into from
Jan 21, 2025
Merged

Add +LINUX and +WINDOWS doctest options #2507

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
de18d8c
Add `+LINUX` and `+WINDOWS` doctest options
peace-maker Dec 18, 2024
062b5b9
Limit Sphinx version to secure platform patches
peace-maker Dec 18, 2024
4c657b1
CI: Run doctests with coverage on Windows
peace-maker Dec 18, 2024
3f22d81
Only apply platform patch on Python 3
peace-maker Dec 18, 2024
5ad4723
Disable uploading coverage on Windows
peace-maker Dec 18, 2024
d80b096
Use threading.Timer for doctest timeout
peace-maker Dec 18, 2024
c92fe20
Add POSIX optionflag
peace-maker Dec 22, 2024
fa1f3bd
Enable tube and tube/sockets tests on Windows
peace-maker Dec 23, 2024
cf4f7be
Use `signal.alarm` for timeouts if it's available
peace-maker Dec 23, 2024
787ac54
Merge branch 'dev' into windows_doctests
peace-maker Jan 15, 2025
a82f6ac
Update CHANGELOG
peace-maker Jan 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -284,10 +284,25 @@ jobs:
pip install --upgrade pip
pip install --upgrade --editable .

- name: Install documentation dependencies
run: pip install -r docs/requirements.txt

- name: Sanity checks
run: |
python -bb -c 'from pwn import *'
python -bb examples/text.py

- name: Coverage doctests
run: |
python -bb -m coverage run -m sphinx -b doctest docs/source docs/build/doctest

# FIXME: Paths are broken when uploading coverage on ubuntu
# coverage.exceptions.NoSource: No source for code: '/home/runner/work/pwntools/pwntools/D:\a\pwntools\pwntools\pwn\__init__.py'.
# - uses: actions/upload-artifact@v4
# with:
# name: coverage-windows
# path: .coverage*
# include-hidden-files: true

upload-coverage:
runs-on: ubuntu-latest
Expand Down
2 changes: 1 addition & 1 deletion docs/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,6 @@ psutil
requests>=2.5.1
ropgadget>=5.3
sphinx==1.8.6; python_version<'3'
sphinx>=7.0.0; python_version>='3'
sphinx>=8.1.3, <9; python_version>='3'
sphinx_rtd_theme
sphinxcontrib-autoprogram<=0.1.5
3 changes: 3 additions & 0 deletions docs/source/adb.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@
from pwn import *
adb = pwnlib.adb

import doctest
doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['LINUX']

:mod:`pwnlib.adb` --- Android Debug Bridge
=====================================================

Expand Down
4 changes: 4 additions & 0 deletions docs/source/asm.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@
import subprocess
from pwn import *

# TODO: Remove global POSIX flag
import doctest
doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']

:mod:`pwnlib.asm` --- Assembler functions
=========================================

Expand Down
77 changes: 69 additions & 8 deletions docs/source/conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -398,6 +398,15 @@ def dont_skip_any_doctests(app, what, name, obj, skip, options):

class _DummyClass(object): pass

# doctest optionflags for platform-specific tests
# they are skipped on other platforms
WINDOWS = doctest.register_optionflag('WINDOWS')
LINUX = doctest.register_optionflag('LINUX')
POSIX = doctest.register_optionflag('POSIX')

# doctest optionflag for tests that haven't been looked at yet
TODO = doctest.register_optionflag('TODO')

class Py2OutputChecker(_DummyClass, doctest.OutputChecker):
def check_output(self, want, got, optionflags):
sup = super(Py2OutputChecker, self).check_output
Expand Down Expand Up @@ -425,27 +434,79 @@ def check_output(self, want, got, optionflags):
return False
return True

import sphinx.ext.doctest

class PlatformDocTestRunner(sphinx.ext.doctest.SphinxDocTestRunner):
def run(self, test, compileflags=None, out=None, clear_globs=True):
original_optionflags = self.optionflags | test.globs.get('doctest_additional_flags', 0)
def filter_platform(example):
optionflags = original_optionflags
if example.options:
for (optionflag, val) in example.options.items():
if val:
optionflags |= optionflag
else:
optionflags &= ~optionflag

if (optionflags & WINDOWS) == WINDOWS and sys.platform != 'win32':
return False
if (optionflags & LINUX) == LINUX and sys.platform != 'linux':
peace-maker marked this conversation as resolved.
Show resolved Hide resolved
return False
if (optionflags & POSIX) == POSIX and os.name != 'posix':
return False
return True

test.examples[:] = [example for example in test.examples if filter_platform(example)]

return super(PlatformDocTestRunner, self).run(test, compileflags, out, clear_globs)

class PlatformDocTestBuilder(sphinx.ext.doctest.DocTestBuilder):
_test_runner = None

@property
def test_runner(self):
return self._test_runner

@test_runner.setter
def test_runner(self, value):
self._test_runner = PlatformDocTestRunner(value._checker, value._verbose, value.optionflags)

def py2_doctest_init(self, checker=None, verbose=None, optionflags=0):
if checker is None:
checker = Py2OutputChecker()
doctest.DocTestRunner.__init__(self, checker, verbose, optionflags)

if 'doctest' in sys.argv:
def setup(app):
pass # app.connect('autodoc-skip-member', dont_skip_any_doctests)

if sys.version_info[:1] < (3,):
import sphinx.ext.doctest
sphinx.ext.doctest.SphinxDocTestRunner.__init__ = py2_doctest_init
else:
def setup(app):
app.add_builder(PlatformDocTestBuilder, override=True)
# app.connect('autodoc-skip-member', dont_skip_any_doctests)
# monkey patching paramiko due to https://github.com/paramiko/paramiko/pull/1661
import paramiko.client
import binascii
paramiko.client.hexlify = lambda x: binascii.hexlify(x).decode()
paramiko.util.safe_string = lambda x: '' # function result never *actually used*
class EndlessLoop(Exception): pass
def alrm_handler(sig, frame):
signal.alarm(180) # three minutes
class EndlessLoop(BaseException): pass
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BaseException? so it will now interrupt doctest flow and not just the one test, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it's still just one test, but the exception won't be caught by other exception handlers in our own code.

File "..\..\pwnlib\context\__init__.py", line ?, in default
Failed example:
    r=remote('google.com', 80)
Expected:
    Traceback (most recent call last):
    ...
    ProxyConnectionError: Error connecting to SOCKS5 proxy localhost:1080: [Errno 111] Connection refused
Got:
    Traceback (most recent call last):
      File "C:\Users\Jannik\Documents\GitHub\pwntools\venv\Lib\site-packages\socks.py", line 787, in connect
        super(socksocket, self).connect(proxy_addr)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
    ConnectionRefusedError: [WinError 10061] Es konnte keine Verbindung hergestellt werden, da der Zielcomputer die Verbindung verweigerte
    <BLANKLINE>
    During handling of the above exception, another exception occurred:
    <BLANKLINE>
    Traceback (most recent call last):
      File "C:\Users\Jannik\AppData\Local\Programs\Python\Python313\Lib\doctest.py", line 1395, in __run
        exec(compile(example.source, filename, "single",
        ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                     compileflags, True), test.globs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "<doctest default[1]>", line 1, in <module>
        r=remote('google.com', 80)
      File "C:\Users\Jannik\Documents\GitHub\pwntools\pwnlib\tubes\remote.py", line 85, in __init__
        self.sock   = self._connect(fam, typ)
                      ~~~~~~~~~~~~~^^^^^^^^^^
      File "C:\Users\Jannik\Documents\GitHub\pwntools\pwnlib\tubes\remote.py", line 130, in _connect
        sock.connect(sockaddr)
        ~~~~~~~~~~~~^^^^^^^^^^
      File "C:\Users\Jannik\Documents\GitHub\pwntools\venv\Lib\site-packages\socks.py", line 47, in wrapper
        return function(*args, **kwargs)
      File "C:\Users\Jannik\Documents\GitHub\pwntools\venv\Lib\site-packages\socks.py", line 791, in connect
        self.close()
        ~~~~~~~~~~^^
      File "C:\Users\Jannik\Documents\GitHub\pwntools\venv\Lib\site-packages\socks.py", line 413, in close
        def close(self):
    <BLANKLINE>
      File "C:\Users\Jannik\Documents\GitHub\pwntools\docs\source\conf.py", line 494, in sigabrt_handler
        raise EndlessLoop()
    EndlessLoop

def sigabrt_handler(signum, frame):
raise EndlessLoop()
signal.signal(signal.SIGALRM, alrm_handler)
signal.alarm(600) # ten minutes
# thread.interrupt_main received the signum parameter in Python 3.10
if sys.version_info >= (3, 10):
signal.signal(signal.SIGABRT, sigabrt_handler)
def alrm_handler():
try:
import thread
except ImportError:
import _thread as thread
# pre Python 3.10 this raises a KeyboardInterrupt in the main thread.
# it might not show a traceback in that case, but it will stop the endless loop.
thread.interrupt_main(signal.SIGABRT)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this work in 3.9? I think the if should be here rather than there. Or should we os.kill the main thread directly (works on win32 supposedly)? Does this work with tests hanging on IO (read from a socket/pipe)? Maybe we can just support this development feature on platforms with alarm(2) and give up trying to achieve better portability? Not sure if hanging tests are bound to happen frequently on win32 and not on Linux for example.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SIGINT raised by calling thread.interrupt_main without an argument causes the doctest to stop and print "Interrupted!" without a traceback. os.kill just stops the process without any further output.

Which Python version do we want to support and test on? 3.8+ or 3.10+? CI only tests 3.10 and 3.12 right now. I think we can keep the old alarm based interrupt handling for host systems that support it and use this one for others? I think it's better to have some timeout in place even if it potentially doesn't handle all cases. I don't know about IO, but it's an exception from a signal handler similar to the alarm way before.

timer = threading.Timer(interval=180, function=alrm_handler) # three minutes
timer.daemon = True
timer.start()
import threading
timer = threading.Timer(interval=600, function=alrm_handler) # ten minutes
timer.daemon = True
timer.start()
3 changes: 3 additions & 0 deletions docs/source/elf/corefile.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@
# Set the environment here so it's not in the middle of our tests.
os.environ.setdefault('SHELL', '/bin/sh')

import doctest
doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']


:mod:`pwnlib.elf.corefile` --- Core Files
===========================================================
Expand Down
4 changes: 4 additions & 0 deletions docs/source/elf/elf.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@
from pwnlib.elf.maps import CAT_PROC_MAPS_EXIT
import shutil

# TODO: Remove global POSIX flag
import doctest
doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']

:mod:`pwnlib.elf.elf` --- ELF Files
===========================================================

Expand Down
4 changes: 4 additions & 0 deletions docs/source/encoders.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
.. testsetup:: *

from pwn import *

# TODO: Remove global POSIX flag
import doctest
doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']

:mod:`pwnlib.encoders` --- Encoding Shellcode
===============================================
Expand Down
4 changes: 4 additions & 0 deletions docs/source/filesystem.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@
from pwnlib.tubes.ssh import ssh
from pwnlib.filesystem import *

# TODO: Remove global POSIX flag
import doctest
doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']

:mod:`pwnlib.filesystem` --- Manipulating Files Locally and Over SSH
====================================================================

Expand Down
4 changes: 4 additions & 0 deletions docs/source/gdb.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@
context.arch = 'amd64'
context.terminal = [os.path.join(os.path.dirname(pwnlib.__file__), 'gdb_faketerminal.py')]

# TODO: Test on cygwin too
import doctest
doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']

:mod:`pwnlib.gdb` --- Working with GDB
======================================

Expand Down
3 changes: 3 additions & 0 deletions docs/source/intro.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@

from pwn import *

import doctest
doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']

Getting Started
========================

Expand Down
4 changes: 4 additions & 0 deletions docs/source/libcdb.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@
from pwn import *
from pwnlib.libcdb import *

# TODO: Remove global POSIX flag
import doctest
doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']

:mod:`pwnlib.libcdb` --- Libc Database
===========================================

Expand Down
4 changes: 4 additions & 0 deletions docs/source/qemu.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

from pwn import *

# TODO: Remove global POSIX flag
import doctest
doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']


:mod:`pwnlib.qemu` --- QEMU Utilities
==========================================
Expand Down
4 changes: 4 additions & 0 deletions docs/source/rop/rop.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,10 @@

context.clear()

# TODO: Remove global LINUX flag
import doctest
doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['LINUX']


:mod:`pwnlib.rop.rop` --- Return Oriented Programming
==========================================================
Expand Down
3 changes: 3 additions & 0 deletions docs/source/rop/srop.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@
from pwnlib.elf import ELF
from pwnlib.tubes.process import process

import doctest
doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['LINUX']

:mod:`pwnlib.rop.srop` --- Sigreturn Oriented Programming
==========================================================

Expand Down
4 changes: 4 additions & 0 deletions docs/source/runner.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@
from pwnlib.runner import *
from pwnlib.asm import asm

# TODO: Remove global POSIX flag
import doctest
doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']

:mod:`pwnlib.runner` --- Running Shellcode
===========================================

Expand Down
4 changes: 4 additions & 0 deletions docs/source/shellcraft.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

from pwnlib import shellcraft

# TODO: Remove global POSIX flag
import doctest
doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']

:mod:`pwnlib.shellcraft` --- Shellcode generation
=================================================

Expand Down
3 changes: 3 additions & 0 deletions docs/source/shellcraft/aarch64.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@
from pwn import *
context.clear(arch='aarch64')

import doctest
doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['LINUX']

:mod:`pwnlib.shellcraft.aarch64` --- Shellcode for AArch64
===========================================================

Expand Down
4 changes: 4 additions & 0 deletions docs/source/shellcraft/amd64.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@
from pwn import *
context.clear(arch='amd64')

# TODO: POSIX/WINDOWS shellcode test
import doctest
doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['LINUX']

:mod:`pwnlib.shellcraft.amd64` --- Shellcode for AMD64
===========================================================

Expand Down
3 changes: 3 additions & 0 deletions docs/source/shellcraft/arm.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@
from pwn import *
context.clear(arch='arm')

import doctest
doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['LINUX']

:mod:`pwnlib.shellcraft.arm` --- Shellcode for ARM
===========================================================

Expand Down
3 changes: 3 additions & 0 deletions docs/source/shellcraft/i386.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@
from pwn import *
context.clear(arch='i386')

import doctest
doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']

:mod:`pwnlib.shellcraft.i386` --- Shellcode for Intel 80386
===========================================================

Expand Down
3 changes: 3 additions & 0 deletions docs/source/shellcraft/mips.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@

context.clear(arch='mips')

import doctest
doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['LINUX']

:mod:`pwnlib.shellcraft.mips` --- Shellcode for MIPS
===========================================================

Expand Down
3 changes: 3 additions & 0 deletions docs/source/shellcraft/riscv64.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@
from pwn import *
context.clear(arch='riscv64')

import doctest
doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['LINUX']

:mod:`pwnlib.shellcraft.riscv64` --- Shellcode for RISCV64
==========================================================

Expand Down
3 changes: 3 additions & 0 deletions docs/source/shellcraft/thumb.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@
from pwn import *
context.clear(arch='thumb')

import doctest
doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['LINUX']

:mod:`pwnlib.shellcraft.thumb` --- Shellcode for Thumb Mode
===========================================================

Expand Down
4 changes: 4 additions & 0 deletions docs/source/tubes.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

from pwn import *

# TODO: Remove global POSIX flag
import doctest
doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']

:mod:`pwnlib.tubes` --- Talking to the World!
=============================================

Expand Down
4 changes: 4 additions & 0 deletions docs/source/tubes/buffer.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

from pwnlib.tubes.buffer import *

# TODO: Remove global POSIX flag
import doctest
doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']

:mod:`pwnlib.tubes.buffer` --- buffer implementation for tubes
==============================================================

Expand Down
4 changes: 4 additions & 0 deletions docs/source/tubes/processes.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

from pwn import *

# TODO: Remove global POSIX flag
import doctest
doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']

:mod:`pwnlib.tubes.process` --- Processes
===========================================================

Expand Down
Loading
Loading