Monkey-patching Python Core Functionality

      Comments Off on Monkey-patching Python Core Functionality

“Monkey patching” is an often dubious practice whereby you make changes to a running program, usually modifying the core capabilities to add some feature or fix some problem.  There are many pitfalls, especially with languages like Python and Ruby which it easy to do by nature of their flexible, accessible data models.

In general, I would never recommend people do it, always asking the question: “Isn’t there some other way?”  Recently, we ran into a Python asyncio bug that left us no better choice.  This post focuses on the technique I developed to do so safely and without risk.

The Problem

Our new Chaperone process manager relies heavily on Python’s new asyncio library.   While asyncio is really a marvelous new addition to the Python standard library, we’ve discovered that it has been revving rapidly from Python 3.4.0 (which is bundled with Ubuntu 14.04LTS) and more recent versions.

We were getting an InvalidStateError exception whenever Chaperone would exit, and finally traced it down to Python Issue #23140, which unfortunately indicated that although the problem was fixed in later versions, there was no workaround!  This was bad.  It meant that we would need to require a newer version of Python for Chaperone installs, which made Chaperone almost useless to many mainstream distributions.  After all, who wants to upgrade Python just to use a minor little utility program?

The Solution

We wanted to be sure that Chaperone would “just work” no matter what distribution was used, and would fail safe, so we had to assure that:

  • The code patch would never break Python.
  • The patch would be automatically omitted for later versions which fix the bug.
  • No additional software was required.

We decided the best way was to inspect the source code itself using importlib, and only apply the patch to old code which matched the flawed code exactly.   So, we created a module exclusively for patches (you can view it here on github), including a little function called PATCH_CLASS:

# Patch routine for patching classes.  Ignore ALL exceptions, since there 
# could be any number of reasons why a distribution may not allow such
# patching (though most do). Exact code is compared, so there is little chance 
# of an error in deciding if the patch is relevant.

def PATCH_CLASS(module, clsname, member, oldstr, newfunc):
    try:
        cls = getattr(importlib.import_module(module), clsname)
        should_be = ''.join(inspect.getsourcelines(getattr(cls, member))[0])
        if should_be == oldstr:
            setattr(cls, member, newfunc)
    except Exception:
        pass

The above code does rely on Python source code being installed and available, but for mainstream distributions, this is almost always the case.

Then, we included the patch for Issue #23140.  First, the old code (pre-patched):

# PATCH  for Issue23140: https://bugs.python.org/issue23140
# WHERE  asyncio
# IMPACT Eliminates exceptions during process termination
# WHY    There is no workround except upgrading to Python 3.4.3,
#        which dramatically affects distro compatibility.  Mostly,
#        this benefits Ubuntu 14.04LTS.

OLD_process_exited = """    def process_exited(self):
        # wake up futures waiting for wait()
        returncode = self._transport.get_returncode()
        while self._waiters:
            waiter = self._waiters.popleft()
            waiter.set_result(returncode)
"""

Then, the replacement code:

def NEW_process_exited(self):
    # wake up futures waiting for wait()
    returncode = self._transport.get_returncode()
    while self._waiters:
        waiter = self._waiters.popleft()
        if not waiter.cancelled():
            waiter.set_result(returncode)

Finally, at the bottom of our patches module, we applied the patch:

PATCH_CLASS('asyncio.subprocess', 'SubprocessStreamProtocol',
    'process_exited', OLD_process_exited, NEW_process_exited)

Conclusion

There are many other use-cases for monkey-patching, but we found the above created a simple framework so we could be sure that any time we discovered distribution problems, we could easily patch them to be sure Chaperone wasn’t too distribution-sensitive

If you’re interested, here are some other useful links to people who have found good use-cases and implementations:

However, I would like to add my own final bit of advice which I’ve managed for follow for years, until this recent relapse…

Avoid it!

Comments

comments