brennivin.threadutils module

Things to make working with threading in Python easier. Note, you should generally avoid using threads, but sometimes you need them! Check out brennivin.uthread for a tasklet based solution.

Contains useful Thread subclasses:

  • ExceptionalThread, which brings proper exceptions to threads.
  • NotAThread, which is useful for mocking threads because it runs synchronously when start() is called.
  • TimerExt: A cancellable/restartable threading.Timer.

Some useful threading-related utilities:

  • ChunkIter, useful for chunking work on a background thread and reporting it to another thread in chunks (useful for UI).
  • memoize(), a caching decorator that can be threadsafe (vital if you want a singleton that has some expensive global state to construct, for example). There is also expiring_memoize for a time-based solution.
  • token, a simple threading token that can be set/queried, useful for inter-thread communication.
  • Signal, used for registering and signaling events in a process.
  • join_timeout(), raises an error if a thread is alive after a join.

Members

class brennivin.threadutils.ChunkIter(iterable_, callback, chunksize=50)

Object that can be used to iterate over a collection on a background thread and report progress in a callback. This is useful when iteration of items is slow (such as if it is an expensive map or filter) and can be asynchronous.

Iteration starts as soon as the object is created.

Parameters:
  • iterable – An iterable object, such as a list or generator. If the iterable is to be mapped and filtered, use itertools.imap and itertools.ifilter to pass in generators that perform the mapping and filtering, so it too can be done on the background thread.
  • callback – A callable that takes a list of items as yielded by iterable_.
  • chunksize – Chunks will be reported back to callable with lists of chunksize items (the last chunk will be leftovers).

If you do not want to use threading, override or patch the start_thread class method to use whatever library.

Cancel()

Call to cancel the iteration. Not be instantaneous.

IsFinished()

Returns True if the iteration is finished.

WaitChunks(chunks=1, sleep_interval=1)

Waits for chunks amount of chunks to be reported. Useful directly after initialization, to wait for some seed of items to be iterated.

Parameters:
  • chunks – Number of chunks to wait for.
  • sleep_interval – Amount of time to sleep before checking to see if new chunks are reported.
cancel()

Call to cancel the iteration. Not be instantaneous.

is_finished()

Returns True if the iteration is finished.

wait_chunks(chunks=1, sleep_interval=1)

Waits for chunks amount of chunks to be reported. Useful directly after initialization, to wait for some seed of items to be iterated.

Parameters:
  • chunks – Number of chunks to wait for.
  • sleep_interval – Amount of time to sleep before checking to see if new chunks are reported.
wait_for_completion(timeout=None)

threading.Thread.join(timeout)() on the background thread.

class brennivin.threadutils.ExceptionalThread(*args, **kwargs)

Drop-in subclass for a regular threading.Thread.

If an error occurs during run():

  • Sets self.exc_info = sys.exc_info()
  • Calls self.excepted.Fire(self.exc_info)
  • If sys.excepthook is not the default, invoke it with self.exc_info. The default just writes to stderr, so no point using it.
  • If self.reraise is True, reraise the original error when all this handling is complete.

If an error occured, it will also be raised on join().

Parameters:kwargs

Same as Thread, except with a reraise key (default None). If reraise is True, invoke the Thread exception handling after ExceptionalThread’s exception handling. If False, do not invoke Thread’s exception handling. If None, only invoke Thread’s exception handling if excepted has no delegates and sys.excepthook is the default.

If you are joining on the thread at any point, you should always set reraise to False, since join will reraise any exceptions on the calling thread.

There’s usually little point using True because Thread’s exception handling because it just writes to stderr.

class brennivin.threadutils.NotAThread(group=None, target=None, name=None, args=(), kwargs=None, verbose=None)

A thread whose start() method runs synchronously. Useful to say ExceptionalThread = NotAThread if you want to debug a program without threading.

class brennivin.threadutils.Signal(eventdoc=None, onerror=traceback.print_exception)

Maintains a collection of delegates that can be easily fired.

Listeners can add and remove callbacks through the connect() and disconnect() methods. Owners can emit the event through emit().

Parameters:
  • eventdoc – Clients can provide info about the event signature and what it represents. It serves no functional purpose but is useful for readability.
  • onerror – Callable that takes (etype, evalue, tb) and is fired when any delegate errors.
class brennivin.threadutils.TimerExt(interval, function, args=(), kwargs=None)

Extends the interface of threading.Timer to allow for a restart() method, which will restart the timer. May be extended in other ways in the future.

Parameters:
  • interval – Number > 0.
  • functionfunction(*args, **kwargs) that is called when the timer elapses.
restart()

Resets the timer. Will raise if the timer has finished.

class brennivin.threadutils.Token

Defines a simple object that can be used for callbacks and cancellations.

class brennivin.threadutils.expiring_memoize(expiry=0, gettime=None)

Decorator used to cache method responses evaluated only once within an expiry period (seconds).

Usage:

import random
class MyClass(object):
    # Create method whose value is cached for ten minutes
    @ExpiringMemoize(expiry=600)
    def randint(self):
        return random.randint(0, 100)
brennivin.threadutils.join_timeout(thread, timeout=8, errtype=<type 'exceptions.RuntimeError'>)

threading.Thread.join(timeout)() and raises errtype if threading.Thread.is_alive() after join.

brennivin.threadutils.memoize(func=None, uselock=False, _lockcls=<IGNORE>)

Decorator for parameterless functions, to cache their return value. This is functionally the same as using a Lazy instance but allows the use of true functions instead of attributes.

Consider allowing memoization of parameterful functions, but that is far more complex so we don’t need to do it right now.

Parameters:
  • func – Filled when used parameter-less, which will use a non-locking memoize (so there is a potential for the function to be called several times). If func is passed, useLock must be False.
  • uselock – If True, acquire a lock when evaluating func, so it can only be evaluated once.