Python Decorators and Faux-OP

I recently found a feature of Python that I quite like (although think the interface for using it is poorly designed): decorators.

Decorators are Python functions that alter what happens when a function is called. In the example:

@do_some_magic
def foo
  pass

the function reference by do_some_magic will be called before foo to perform some magic. This opens up some interesting metaprogramming opportunities. I’d like to use it to my tiny MVC framework for Google App Engine for adding additional behavior to actions:

  @login_required
  @authorize
  @only_adminstrators
  def show
    pass

For what I want to do it’s not as powerful as Rails’ before_filter, but more pythonic (I hope) and will require less custom code. Unfortunately, to use decorators I need login_required, authorize, and only_adminstrators to be scoped correctly to the instance object.

Self, like many OOP aspects of Python, is accomplished by some internal trickery that is exposed to the programmer. Internal trickery I can handle; having it exposed to me, I cannot.
In Ruby, self is handled by the language and always behaves in a predictable way (referencing the receiver of a method call). In Python, self for a function is accomplished by some hoop-jumping. The first (secret and hidden) argument to a method in Python represents the object receiving the message.

So, you get methods definitions like

  def full_name(self)
    self.first_name + " " + self.last_name

This is called with

  person.full_name()

and self will be set to the object represented by person. “But wait!” you say, “you just called full_name with no arguments. How did it handle having called a method that clearly is defined with one argument being called with none?”

~~Trickery~~Explicitness. Despite appearing like this method requires you to provide an argument, it doesn’t. The called object is automatically inserted for you. In fact, calling this method with an argument (even one you think would be meaningful like, say, the instance object) will cause Python to throw an error:

  person.full_name(person)

will return

TypeError: full_name() takes exactly 1 argument (2 given)

Yeah, you read that right. I passed one argument and I’m told that, in fact, I passed two and it wants one. Python handles passing the first argument for me. Why it couldn’t handle keeping track of self internally and keep argument counts more obvious is something a more experience Python programmer would have to answer. To demonstrate just how faked self is, we can in fact call this first (magic and hidden) argument to a method anything we want:

  def full_name(identity_crisis)
    identity_crisis.first_name + " " + identity_crisis.last_name

has the same effect. Python programmers just use the word self by convention.

This loose concept of identity in Python instance objects was tripping me up when using decorators. With the sample code below

class Foo(object):
  def __init__(self, arg):
    self.name = arg

  def dec(self):
    self.name = "something else!"

  @dec
  def bar(self):
    return self.name

I expected calling bar would show me the name attribute of the object, now altered to “something else!”.

  x = Foo('x')
  x.name # returns 'x'
  x.bar()  # expecting "something else!", but got TypeError: 'NoneType' object is not callable

Huh? What in my code is NoneType? Well it turns out that inside my decorator dec, the object represented by self doesn’t have a name because self doesn’t refer to the instance object. If you’re Python person, stop chucking. If you’re a Ruby person, read that again. Why wouldn’t self refer to the instance object?

Well, even though the decorator was defined in the same way as every other instance method in the class, the first (secret and hidden) argument of a decorator function isn’t the instance object. It’s the function that it decorates. So, self here refers to the function bar. I read elsewhere that by convention the first (secret and hidden) argument to a decorator is named fn. So, I clean up the code a bit to make it more obvious that I don’t have access to my instance object:

class Foo(object):
  def __init__(self, arg):
    self.name = arg

  def dec(fn):
    # self.name = "something else!"

  @dec
  def bar(self):
    return self.name

but am still left with the question of “how to I get the instance object?”. I know that functions have a im_self attribute (don’t ask me what the im_ part means, I have no idea; Python seems obsessed with short names instead of meaningful ones). So, maybe I can call that:

class Foo(object):
  def __init__(self,name):
    self.name = name

  def dec(fn):
    fn.im_self.name = 'not what you think!'

  @dec
  def bar(self):
    return self.name

alas, I receive a Traceback:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 16, in Foo
  File "<stdin>", line 7, in dec
AttributeError: 'function' object has no attribute 'im_self'

Aha! Only certain functions have access to im_self. Not this one, where it would be useful for accessing the instance object, but other ones. So, I ask again, how do I get access to the instance object inside of a decorator?

I’m sure an experienced Python programmer has already seen my problem: I keep thinking of self as some sort of implicit aspect of OOP where any object knows how to refer to itself using self. I’m not thinking about self as an argument to function floating out in space, connected to a class by the thinnest of lifelines. The real question isn’t “how to get self” it’s “how to get the the arguments of decorated function, of which in instance object is the 0th (secret and hidden) argument.”

I stumbled upon the answer immediately when I started thinking about it correctly: another, nested function. If I put another nested function inside my decorator and return that at the end of the decorator I’ll be able to access the arguments (of which the instance objet is the 0th).

class Foo(object):
  def __init__(self,name):
    self.name = name

  def dec(fn):
    def inner(*args):
      args[0].name = 'not what you thought, huh?'
      return fn(*args)
    return inner

  @dec
  def bar(self):
    return self.name

and now, as expected

 x = Foo('x')
 x.name  # 'x'
 x.bar() #'not what you thought, huh?'
 x.name  # 'not what you thought, huh?'

I know Python programmers defend this with a cry of “explicit is better than implicit,” but this is really a mix of explicit and implicit. In my method definition I explicitly refer to the instance object which is then implicitly known when I call the method; explicitly passing it as argument will fail. And, because some functions in a class (decorators) take a function as a first argument, I cannot guarantee that this implicit rule is consistently applied.

Of course the instance object actually is passed when you understand that classes in Python behave closer to a namespacing mechanism than class objects and

  x = Foo('x')
  # call the bar method of our new Foo object
  x.bar()

is nothing more than implicit shorthand for explicit syntax:

  x = Foo('x')
  # call the bar method of Foo with an instance of Foo as an argument
  Foo.bar(x)

To create what other languages would treat as a class method, where self would refer to the class object you decorate the function with the @classmethod decorator

  @classmethod
  def classy(self):
    return self

which is really just shorthand for the old-style syntax of creating a class method:

  def classy(self):
    return self
  classy = classmethod(classy)

Comparing this to languages like Ruby, Smalltalk, or Objective-C I hesitate before telling people that Python is an object oriented language. Python clearly allows a programmer to support the object orientation pattern in their own code, but much of pattern implementation smells of functional programming tricks.

Still, apart from all of that, decorators are a cool idea.

About this entry

You’re currently reading “Python Decorators and Faux-OP,” an entry on Trek

Published:: October 6, 2008 / 2:58 pm

Category:: code

Tags:: decorators, programming, python

About

I’m a web developer, consultant, and sometimes instructor in Ann Arbor, MI. I’ve been fortunate enough to have done some cool things like get my masters degree (in HCI), become an accredited librarian, work for a crazy web startup, and win an apple design award.

You can get a hold of me at trek.glowacki (at) gmail.com
Follow me on twitter or github, and make friends on facebook

trek

Dan | thesamovar 10.6.08 / 8pm

Ugh, neither the code nor pre tags preserve formatting in comments. OK I’ll use ~~ to indicate tabs.

Python can be a little surprising at first but it’s kind of logical once you get the hang of it. In your case, I think what’s getting you is that a class definition is just a regular series of statements followed by some extra stuff, converting functions to methods. So for instance I can do this:

class C(object): ~~def f(self): ~~~~print self ~~f(2) print 'Start?' c = C() print 'Now what?' c.f()

and this will show:

2
Start?
Now what?
__main__.C object at 0×01BA24B0

The bit inside the class definition is first just run at the time the class is defined as if it were just a standard bit of code, with nothing special going on at all. Specifically, the class is a namespace, so in the above f is now an object in C’s namespace. Once the code in the class definition has finished running, all the function objects in that namespace are converted to bound methods (i.e. ones that call their instance as the first argument). So for example I can do this:

def g(x): ~~print x.name class D(object): ~~def __init__(self, name): ~~~~self.name = name ~~f = g d = D('bob') g(d) d.f()

and it prints:

bob
bob

Anyway, so in the first example where you use @dec and you have defined dec like def dec(self):… and then use dec in the class definition, it is just a function. Nothing special about it being a decorator, it’s just to do with the way class definitions work. The reason x.bar() raises the error is that your dec function doesn’t return anything. The following are equivalent:

@dec def f(): ~~...

and

def f(): ~~... f = dec(f)

A function that doesn’t return anything is a function that returns None in Python, so in your code x.bar is None. The decorator can’t have access to the instance itself, because when the decorator function is called (during the class definition), there are no instances of the class (in fact the class doesn’t even exist yet). Most of the time, defining decorators as methods of a class as you are doing is probably not a good idea (although it works here).

Adam Byrtek 10.7.08 / 8am

Sorry to say this but it looks like you got Python decorators *completely* wrong. In your example you somehow assume that a decorator is invoked on every function call, takes the same arguments as the decorated function and automatically passes control to the decorated function. All those assumptions are incorrect.

In reality the decorator function is invoked only once, takes the decorated function as the only argument (remember that functions are first-class objects in Python) and has to return a function object that will be used in place of the decorated one.

It all makes perfect sense, you just have to read the documentation before you blog. Try decorating plain functions first and after you get the point, move to instance methods.

wonderfullyflawed 10.7.08 / 8am

@Dan Huh, I definitely didn’t realize that the class pattern in Python was just a series of statements. That is much different than the OOP languages I’ve used. I’d almost think it was clearer if Python used a decortator instead of a keyword (although I realize decorators are fairly new):
@class def MyClass(ParentClass) def class_function_one pass def class_function_two pass
I can also see why it’s probably not a wise decision to define you decorators inside of a class. They don’t appear to “inherit” in the way I’d expect inheritance to work for methods.

class Bar(Foo): @dec def another_bar(self): return self.name

Will return an error that name 'dec' is not defined for the Bar class. To be useful to me, I need them in children classes, so I'll be storing them elsewhere.

Dan | thesamovar 10.7.08 / 10am

I would put dec outside the class definition, but if you really want it to be stored inside the class you could make it a static method by decorating dec with @staticmethod, and then in Bar you could decorate another_bar with @Foo.dec. I didn’t test it but it should work.

wonderfullyflawed 10.7.08 / 11am

@Adam oh, I understand that the Python decoration of a function only happens once at definition time. I read the documentation on them and several good tutorials on the web. I had decorators working fine outside of class before trying to make them work with classes.

I just (incorrectly) assumed that, being inside of a class and defined like instance methods, this decorator would behave in an object oriented manner and somehow maintain access to an implicit concept of a properly scoped self, which Python cannot do.

That’s when I moved over to trying to access the decorator function’s associated instance object with im_self, which again doesn’t work since the decorator isn’t associated with any instances at class definition time since, as @Dan points out, there are no instances (and no class yet).

Coming from different OOP languages I assumed that when referring to im_self of my decorator dec Python would understand I meant the instance object in play when I called the decorated function at runtime, not the non-existent-at-definition-time reference to the instance object.

I’ve since discovered that decorators aren’t really useful inside of a class. This seems to be implicit in the documentation and tutorials I read (which include no examples of using decorators in a class). I’ll be keeping them outside of classes from now on, which should resolve some of unexpected behavior I was experiencing.

Saying decorators return a function object that will be used in place of the decorated one is surely untrue. Take the slightly altered example below (where I return instance object as the result of bar() ).

class Foo(object): def __init__(self,name): self.name = name self.age = 21 def dec(fn): def inner(*args): args[0].name = 'not what you thought, huh?' return fn(*args) return inner @dec def bar(self): self.age = '0' return self

x = Foo('x') # runs the data manipulation x.bar # the name set in dec/inner x.name # returns the age set in bar x.age # 0

clearly the decortator isn’t just being used in place of the decorated function (the age attribute would never be altered in that case). x.bar returns the inner function inside of the decorator, but calling bar() executes both, altering name, age, and returning the instance object.

wonderfullyflawed 10.7.08 / 12pm

x.bar in the example above should be x.bar(). Still clearly not enjoying mandatory parens to execute a function despite years of javascript.

Adam Byrtek 10.10.08 / 11am

I don’t really understand what is the problem with the example you provided above. Inside the inner function you invoke the decorated function yourself, so you shouldn’t be surprised that both name and age are changed.

This fragment of code is responsible for this:

return fn(*args)

In other words the inner function (returned by the decorator) is used instead of the decorated function and if you want you can call the decorated function from it (but you don’t have to).

If you want to decorate methods instead of functions, just follow the convention and label the first argument of the inner function as “self” and you have access to an object instance. The decorator itself can be a static method, as somebody already mentioned.

Hope this helps.

Trek

Python Decorators and Faux-OP

About this entry

7 Comments

Have your say

About

Recently

Categories