How I Made Porn 20x More Efficient with Python Video Streaming

Intro

Porn is a big industry. There aren’t many sites on the Internet that can rival the traffic of its biggest players.

And juggling this immense traffic is tough. To make things even harder, much of the content served from porn sites is made up of low latency live video streams rather than simple static video content. But for all of the challenges involved, rarely have I read about the python developers who take them on. So I decided to write about my own experience on the job.

What’s the problem?

A few years ago, I was working for the 26th (at the time) most visited website in the world—not just the porn industry: the world.

At the time, the site served up porn video streaming requests with the Real Time Messaging protocol (RTMP). More specifically, it used a Flash Media Server (FMS) solution, built by Adobe, to provide users with live streams. The basic process was as follows:

  1. The user requests access to some live stream
  2. The server replies with an RTMP session playing the desired footage

For a couple reasons, FMS wasn’t a good choice for us, starting with its costs, which included the purchasing of both:

  1. Windows licenses for every machine on which we ran FMS.
  2. ~$4k FMS-specific licenses, of which we had to purchase several hundred (and more every day) due to our scale.

All of these fees began to rack up. And costs aside, FMS was a lacking product, especially in its functionality (more on this in a bit). So I decided to scrap FMS and write my own Python RTMP parser from scratch.

In the end, I managed to make our service roughly 20x more efficient.

Getting started

There were two core problems involved: firstly, RTMP and other Adobe protocols and formats were not open (i.e., publically available), which made them hard to work with. How can you reverse or parse files in a format about which you know nothing? Luckily, there were some reversing efforts available in the public sphere (not produced by Adobe, but rather by a group called OS Flash, now defunct) on which we based our work.

Note: Adobe later released “specifications” which contained no more information than what was already disclosed in the non-Adobe-produced reversing wiki and documents. Their (Adobe’s) specifications were of an absurdly low quality and made it near impossible to actually use their libraries. Moreover, the protocol itself seemed intentionally misleading at times. For example:

  1. They used 29-bit integers.
  2. They included protocol headers with big endian formatting everywhere—except for a specific (yet unmarked) field, which was little endian.
  3. They squeezed data into less space at the cost of computational power when transporting 9k video frames, which made little to no sense, because they were earning back bits or bytes at a time—insignificant gains for such a file size.

And secondly: RTMP is highly session oriented, which made it virtually impossible to multicast an incoming stream. Ideally, if multiple users wanted to watch the same live stream, we could just pass them back pointers to a single session in which that stream is being aired (this would be multicast video streaming). But with RTMP, we had to create an entirely new instance of the stream for every user that wanted access. This was a complete waste.

Three users demonstrating the difference between a multicast video streaming solution and a FMS streaming problem.

 

Read the full article in Toptal Engineering blog 

Django, Flask, and Redis Tutorial: Web Application Session Management Between Python Frameworks

Django Versus Flask: When Django is the Wrong Choice

I love and use Django in lots of my personal and client projects, mostly for more classical web applications and those involving relational databases. However, Django is no silver bullet.

By design, Django is very tightly coupled with its ORM, Template Engine System, and Settings object. Plus, it’s not a new project: it carries a lot of baggage to remain backwards compatible.

Some Python developers see this as a major problem. They say that Django isn’t flexible enough and avoid it if possible and, instead, use a Python microframework like Flask.

I don’t share that opinion. Django is great when used in the appropriate place and time, even if it doesn’t fit into every project spec. As the mantra goes: “Use the right tool for the job”.

(Even when it is not the right place and time, sometimes programming with Django can have unique benefits.)

In some cases, it can indeed be nice to use a more lightweight framework (like Flask). Often, these microframeworks start to shine when you realize how easy they are to hack on.

Microframeworks to the Rescue

In a few of my client projects, we’ve discussed giving up on Django and moving to a microframework, typically when the clients want to do some interesting stuff (in one case, for example, embedding ZeroMQ in the application object) and the project goals seem more difficult to achieve with Django.

More generally, I find Flask useful for:

  • Simple REST API backends
  • Applications that don’t require database access
  • NoSQL-based web apps
  • Web apps with very specific requirements, like custom URL configurations

At the same time, our app required user registration and other common tasks that Django solved years ago. Given its light weight, Flask doesn’t come with the same toolkit.

The question emerged: is Django an all-or-nothing deal?

The question emerged: is Django an all-or-nothing deal? Should we drop it completely from the project, or can we learn to combine it with the flexibility of other microframeworks or traditional frameworks? Can we pick and choose the pieces we want to use and eschew others?

Can we have the best of both worlds? I say yes, especially when it comes to session management.

(Not to mention, there are a lot of projects out there for Django freelancers.)

Now the Python Tutorial: Sharing Django Sessions

The goal of this post is to delegate the tasks of user authentication and registration to Django, yet use Redis to share user sessions with other frameworks. I can think of a few scenarios in which something like this would be useful:

  • You need to develop a REST API separately from your Django app but want to share session data.
  • You have a specific component that may need to be replaced later on or scaled out for some reason and still need session data.

For this tutorial, I’ll use Redis to share sessions between two frameworks (in this case, Django and Flask). In the current setup, I’ll use SQLite to store user information, but you can have your back-end tied to a NoSQL database (or a SQL-based alternative) if need be.

Understanding Sessions

To share sessions between Django and Flask, we need to know a bit about how Django stores its session information. The Django docs are pretty good, but I’ll provide some background for completeness.

Session Management Varieties

Generally, you can choose to manage your Python app’s session data in one of two ways:

  • Cookie-based sessions: In this scenario, the session data is not stored in a data store on the back-end. Instead, it’s serialized, signed (with a SECRET_KEY), and sent to the client. When the client sends that data back, its integrity is checked for tampering and it is deserialized again on the server.

  • Storage-based sessions: In this scenario, the session data itself is not sent to the client. Instead, only a small portion is sent (a key) to indicate the identity of the current user, stored on the session store.

In our example, we’re more interested in the latter scenario: we want our session data to be stored on the back-end and then checked in Flask. The same thing could be done in the former, but as the Django documentation mentions, there are some concerns about the security of the first method.

The General Workflow

The general workflow of session handling and management will be similar to this diagram:

A diagram showing the management of user sessions between Flask and Django using Redis.

Let’s walk through session sharing in a little more detail:

  1. When a new request comes in, the first step is to send it through the registered middleware in the Django stack. We’re interested here in the SessionMiddleware class which, as you might expect, is related to session management and handling:

    	class SessionMiddleware(object):
    	def process_request(self, request):
    	engine = import_module(settings.SESSION_ENGINE)
    	session_key = request.COOKIES.get(settings.SESSION_COOKIE_NAME, None)
    	request.session = engine.SessionStore(session_key)
    	
    	

    In this snippet, Django grabs the registered SessionEngine (we’ll get to that soon), extracts the SESSION_COOKIE_NAME from request (sessionid, by default) and creates a new instance of the selected SessionEngine to handle session storage.

  • Later on (after the user view is processed, but still in the middleware stack), the session engine calls its save method to save any changes to the data store. (During view handling, the user may have changed a few things within the session, e.g., by adding a new value to session object with request.session.) Then, the SESSION_COOKIE_NAME is sent to the client. Here’s the simplified version:

    	def process_response(self, request, response):
    	....
    	if response.status_code != 500:
    	request.session.save()
    	response.set_cookie(settings.SESSION_COOKIE_NAME,
    	request.session.session_key, max_age=max_age,
    	expires=expires, domain=settings.SESSION_COOKIE_DOMAIN,
    	path=settings.SESSION_COOKIE_PATH,
    	secure=settings.SESSION_COOKIE_SECURE or None,
    	httponly=settings.SESSION_COOKIE_HTTPONLY or None)
    	return response
    	
    	

We’re particularly interested in the SessionEngine class, which we’ll replace with something to store and load data to and from a Redis back-end.

Fortunately, there are a few projects that already handle this for us. Here’s an example from redis_sessions_fork. Pay close attention to the  save and load methods, which are written so as to (respectively) store and load the session into and from Redis:

class SessionStore(SessionBase):
"""
Redis session back-end for Django
"""
def __init__(self, session_key=None):
super(SessionStore, self).__init__(session_key)
def _get_or_create_session_key(self):
if self._session_key is None:
self._session_key = self._get_new_session_key()
return self._session_key
def load(self):
session_data = backend.get(self.session_key)
if not session_data is None:
return self.decode(session_data)
else:
self.create()
return {}
def exists(self, session_key):
return backend.exists(session_key)
def create(self):
while True:
self._session_key = self._get_new_session_key()
try:
self.save(must_create=True)
except CreateError:
continue
self.modified = True
self._session_cache = {}
return
def save(self, must_create=False):
session_key = self._get_or_create_session_key()
expire_in = self.get_expiry_age()
session_data = self.encode(self._get_session(no_load=must_create))
backend.save(session_key, expire_in, session_data, must_create)
def delete(self, session_key=None):
if session_key is None:
if self.session_key is None:
return
session_key = self.session_key
backend.delete(session_key)

It’s important to understand how this class is operating as we’ll need to implement something similar on Flask to load session data. Let’s take a closer look with a REPL example:

>>> from django.conf import settings
>>> from django.utils.importlib import import_module
>>> engine = import_module(settings.SESSION_ENGINE)
>>> engine.SessionStore()
<redis_sessions_fork.session.SessionStore object at 0x3761cd0>
>>> store["count"] = 1
>>> store.save()
>>> store.load()
{u'count': 1}

The session store’s interface is pretty easy to understand, but there’s a lot going on under the hood. We should dig a little deeper so that we can implement something similar on Flask.

Note: You might ask, “Why not just copy the SessionEngine into Flask?” Easier said than done. As we discussed in the beginning, Django is tightly coupled with its Settings object, so you can’t just import some Django module and use it without any additional work.

Django Session (De-)Serialization

As I said, Django does a lot of work to mask the complexity of its session storage. Let’s check the Redis key that’s stored in the above snippets:

>>> store.session_key
u"ery3j462ezmmgebbpwjajlxjxmvt5adu"

Now, lets query that key on the redis-cli:

redis 127.0.0.1:6379> get "django_sessions:ery3j462ezmmgebbpwjajlxjxmvt5adu"
"ZmUxOTY0ZTFkMmNmODA2OWQ5ZjE4MjNhZmQxNDM0MDBiNmQzNzM2Zjp7ImNvdW50IjoxfQ=="

What we see here is a very long, Base64-encoded string. To understand its purpose, we need to look at Django’s SessionBase class to see how it’s handled:

class SessionBase(object):
"""
Base class for all Session classes.
"""
def encode(self, session_dict):
"Returns the given session dictionary serialized and encoded as a string."
serialized = self.serializer().dumps(session_dict)
hash = self._hash(serialized)
return base64.b64encode(hash.encode() + b":" + serialized).decode('ascii')
def decode(self, session_data):
encoded_data = base64.b64decode(force_bytes(session_data))
try:
hash, serialized = encoded_data.split(b':', 1)
expected_hash = self._hash(serialized)
if not constant_time_compare(hash.decode(), expected_hash):
raise SuspiciousSession("Session data corrupted")
else:
return self.serializer().loads(serialized)
except Exception as e:
# ValueError, SuspiciousOperation, unpickling exceptions
if isinstance(e, SuspiciousOperation):
logger = logging.getLogger('django.security.%s' %
e.__class__.__name__)
logger.warning(force_text(e))
return {}

The encode method first serializes the data with the current registered serializer. In other words, it converts the session into a string, which it can later convert back into a session (look at the SESSION_SERIALIZER documentation for more). Then, it hashes the serialized data and uses this hash later on as a signature to check the integrity of the session data. Finally, it returns that data pair to the user as a Base64-encoded string.

By the way: before version 1.6, Django defaulted to using pickle for serialization of session data. Due to security concerns, the default serialization method is now django.contrib.sessions.serializers.JSONSerializer.

Encoding an Example Session

Let’s see the session management process in action. Here, our session dictionary will simply be a count and some integer, but you can imagine how this would generalize to more complicated user sessions.

>>> store.encode({'count': 1})
u'ZmUxOTY0ZTFkMmNmODA2OWQ5ZjE4MjNhZmQxNDM0MDBiNmQzNzM2Zjp7ImNvdW50IjoxfQ=='
>>> base64.b64decode(encoded)
'fe1964e1d2cf8069d9f1823afd143400b6d3736f:{"count":1}'

The result of the store method (u’ZmUxOTY…==’) is an encoded string containing the serialized user session and its hash. When we decode it, we indeed get back both the hash (‘fe1964e…’) and the session ({"count":1}).

Note that the decode method checks to ensure that the hash is correct for that session, guaranteeing integrity of the data when we go to use it in Flask. In our case, we’re not too worried about our session being tampered with on the client side because:

  • We aren’t using cookie-based sessions, i.e., we’re not sending all user data to the client.

  • On Flask, we’ll need a read-only SessionStore which will tell us if given key exists or not and return the stored data.

Extending to Flask

Next, let’s create a simplified version of the Redis session engine (database) to work with Flask. We’ll use the same SessionStore (defined above) as a base class, but we’ll need to remove some of its functionality, e.g., checking for bad signatures or modifying sessions. We’re more interested in a read-only SessionStore that will load the session data saved from Django. Let’s see how it comes together:

class SessionStore(object):
# The default serializer, for now
def __init__(self, conn, session_key, secret, serializer=None):
self._conn = conn
self.session_key = session_key
self._secret = secret
self.serializer = serializer or JSONSerializer
def load(self):
session_data = self._conn.get(self.session_key)
if not session_data is None:
return self._decode(session_data)
else:
return {}
def exists(self, session_key):
return self._conn.exists(session_key)
def _decode(self, session_data):
"""
Decodes the Django session
:param session_data:
:return: decoded data
"""
encoded_data = base64.b64decode(force_bytes(session_data))
try:
# Could produce ValueError if there is no ':'
hash, serialized = encoded_data.split(b':', 1)
# In the Django version of that they check for corrupted data
# I don't find it useful, so I'm removing it
return self.serializer().loads(serialized)
except Exception as e:
# ValueError, SuspiciousOperation, unpickling exceptions. If any of
# these happen, return an empty dictionary (i.e., empty session).
return {}

We only need the load method because it’s a read-only implementation of the storage. That means you can’t logout directly from Flask; instead, you might want to redirect this task to Django. Remember, the goal here is to manage sessions between these two Python frameworks to give you more flexibility.

Flask Sessions

The Flask microframework supports cookie-based sessions, which means all of the session data is sent to the client, Base64-encoded and cryptographically signed. But actually, we’re not very interested in Flask’s session support.

What we need is to get the session ID created by Django and check it against the Redis back-end so that we can be sure the request belongs to a pre-signed user. In summary, the ideal process would be (this syncs up with the diagram above):

  • We grab the Django session ID from the user’s cookie.
  • If the session ID is found in Redis, we return the session matching that ID.
  • If not, we redirect them to a login page.

It’ll be handy to have a decorator to check for that information and set the current user_id into the gvariable in Flask:

from functools import wraps
from flask import g, request, redirect, url_for
def login_required(f):
@wraps(f)
def decorated_function(*args, **kwargs):
djsession_id = request.cookies.get("sessionid")
if djsession_id is None:
return redirect("/")
key = get_session_prefixed(djsession_id)
session_store = SessionStore(redis_conn, key)
auth = session_store.load()
if not auth:
return redirect("/")
g.user_id = str(auth.get("_auth_user_id"))
return f(*args, **kwargs)
return decorated_function

In the example above, we’re still using the SessionStore we defined previously to fetch the Django data from Redis. If the session has an _auth_user_id, we return the content from the view function; otherwise, the user is redirected to a login page, just like we wanted.

Gluing Things Together

In order to share cookies, I find it convenient to start Django and Flask via a WSGI server and glue them together. In this example, I’ve used CherryPy:

from app import app
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()
d = wsgiserver.WSGIPathInfoDispatcher({
"/":application,
"/backend":app
})
server = wsgiserver.CherryPyWSGIServer(("127.0.0.1", 8080), d)

With that, Django will serve on “/” and Flask will serve on “/backend” endpoints.

In Conclusion

Rather than examining Django versus Flask or encouraging you only to learn the Flask microframework, I’ve welded together Django and Flask, getting them to share the same session data for authentication by delegating the task to Django. As Django ships with plenty of modules to solve user registration, login, and logout (just to name a few), combining these two frameworks will save you valuable time while providing you with the opportunity to hack on a manageable microframework like Flask.

This post originally appeared in the Toptal Engineering blog 

The Python Tutorial

Python is an easy to learn, powerful programming language. It has efficient high-level data structures and a simple but effective approach to object-oriented programming. Python’s elegant syntax and dynamic typing, together with its interpreted nature, make it an ideal language for scripting and rapid application development in many areas on most platforms.

The Python interpreter and the extensive standard library are freely available in source or binary form for all major platforms from the Python Web site,https://www.python.org/, and may be freely distributed. The same site also contains distributions of and pointers to many free third party Python modules, programs and tools, and additional documentation.

The Python interpreter is easily extended with new functions and data types implemented in C or C++ (or other languages callable from C). Python is also suitable as an extension language for customizable applications.

This tutorial introduces the reader informally to the basic concepts and features of the Python language and system. It helps to have a Python interpreter handy for hands-on experience, but all examples are self-contained, so the tutorial can be read off-line as well.

For a description of standard objects and modules, see The Python Standard LibraryThe Python Language Reference gives a more formal definition of the language. To write extensions in C or C++, read Extending and Embedding the Python Interpreter and Python/C API Reference Manual. There are also several books covering Python in depth.

This tutorial does not attempt to be comprehensive and cover every single feature, or even every commonly used feature. Instead, it introduces many of Python’s most noteworthy features, and will give you a good idea of the language’s flavor and style. After reading it, you will be able to read and write Python modules and programs, and you will be ready to learn more about the various Python library modules described in The Python Standard Library.

A Tutorial for Reverse Engineering Your Software's Private API: Hacking Your Couch

Traveling is my passion, and I’m a huge fan of Couchsurfing. Couchsurfing is a global community of travelers, where you can find a place to stay or share your own home with other travelers. On top of that, Couchsurfing helps you enjoy a genuine traveling experience while interacting with locals. I’ve been involved with the Couchsurfing community for over 3 years. I attended meetups at first, and then I was finally able to host people. What an amazing journey it was! I’ve met so many incredible people from all over the world and made lots of friends. This whole experience truly changed my life.

Reverse engineering software is fun with a tutorial and a good project idea.

I’ve hosted a lot of travelers myself, much more than I’ve actually surfed yet. While living in one of the major touristic destinations on the French Riviera, I received an enormous amount of couch requests (up to 10 a day during high season). As a freelance back-end developer, I immediately noticed that the problem with the couchsurfing.com website is that it doesn’t really handle such “high-load” cases properly. There is no information about the availability of your couch - when you receive a new couch request you can’t be sure if you are already hosting someone at that time. There should be a visual representation of your accepted and pending requests, so you can manage them better. Also, if you could make your couch availability public, you could avoid unnecessary couch requests. To better understand what I have in mind take a look at Airbnb calendar.

Lots of companies are notorious for not listening to their users. Knowing the history of Couchsurfing, I couldn’t count on them to implement this feature anytime soon. Ever since the website became a for-profit company, the community deteriorated. To better understand what I’m talking about, I suggest reading these two articles:

I knew that lot of community members would be happy to have this functionality. So, I decided to make an app to solve this problem. It turns out there is no public Couchsurfing API available. Here is the response I’ve received from their support team:

“Unfortunately we have to inform you that our API is not actually public and there are no plans at the moment to make it public.”

Breaking Into My Couch

It was time to use some of my favorite software reverse engineering techniques to break into Couchsurfing.com. I assumed that their mobile apps must use some sort of API to query the backend. So, I had to intercept the HTTP requests coming from a mobile app to the backend. For that purpose I set up a proxy in the local network, and connected my iPhone to it to intercept HTTP requests. This way, I was able to find access points of their private API and figure out their JSON payload format.

Finally I created a website which serves the purpose of helping people manage their couch requests, and show surfers a couch availability calendar. I published a link to it on the community forums (which are also quite segmented in my opinion, and it’s difficult to find information there). The reception was mostly positive, although some people didn’t like the idea that the website required couchsurfing.com credentials, which was a matter of trust really.

The website worked like this: you log in to the website with your couchsurfing.com credentials, and after a few clicks you get the html code which you can embed into your couchsurfing.com profile, and voila - you have an automatically updated calendar in your profile. Below is the screenshot of the calendar and here the articles on how I made it:

I’ve created a great feature for Couchsurfing, and I naturally assumed that they would appreciate my work - perhaps even offer me a position in their development team. I’ve sent an email to jobs(at)couchsurfing.comwith a link to the website, my resume, and a reference. A thank-you note left by one of my couchsurfing guests:

A few days later they followed up on my reverse engineering efforts. In the reply it was clear that the only thing they were concerned about was their own security, so they asked me to take down the blog posts I’ve written about the API, and eventually the website. I’ve taken down the posts immediately, as my intention was not to violate the terms of use and fish for user credentials, but rather to help the couchsurfing community. I had an impression that I was treated as a criminal, and the company focused solely the fact that my website requires user credentials.

 Read the full article at Toptal Engineering blog

Python Design Patterns: For Sleek And Fashionable Code

Python is a dynamic and flexible language. Python design patterns are a great way of harnessing its vast potential.Python’s philosophy is built on top of the idea of well thought out best practices. Python is a dynamic language (did I already said that?) and as such, already implements, or makes it easy to implement, a number of popular design patterns with a few lines of code. Some design patterns are built into Python, so we use them even without knowing. Other patterns are not needed due of the nature of the language.

For example, Factory is a structural Python design pattern aimed at creating new objects, hiding the instantiation logic from the user. But creation of objects in Python is dynamic by design, so additions like Factory are not necessary. Of course, you are free to implement it if you want to. There might be cases where it would be really useful, but they’re an exception, not the norm.

What is so good about Python’s philosophy? Let’s start with this (explore it in the Python terminal):

>>> import this
The Zen of Python, by Tim Peters
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

These might not be patterns in the traditional sense, but these are rules that define the “Pythonic” approach to programming in the most elegant and useful fashion.

We have also PEP-8 code guidelines that help structure our code. It’s a must for me, with some appropriate exceptions, of course. By the way, these exceptions are encouraged by PEP-8 itself:

“But most importantly: know when to be inconsistent – sometimes the style guide just doesn’t apply. When in doubt, use your best judgment. Look at other examples and decide what looks best. And don’t hesitate to ask!”

Combine PEP-8 with The Zen of Python (also a PEP - PEP-20), and you’ll have a perfect foundation to create readable and maintainable code. Add Design Patterns and you are ready to create every kind of software system with consistency and evolvability.

Python Design Patterns

What Is A Design Pattern?

Everything starts with the Gang of Four (GOF). Do a quick online search if you are not familiar with the GOF.

Design patterns are a common way of solving well known problems. Two main principles are in the bases of the design patterns defined by the GOF:

  • Program to an interface not an implementation.
  • Favor object composition over inheritance.

Let’s take a closer look at these two principles from the perspective of Python programmers.

Program to an interface not an implementation

Think about Duck Typing. In Python we don’t like to define interfaces and program classes according these interfaces, do we? But, listen to me! This doesn’t mean we don’t think about interfaces, in fact with Duck Typing we do that all the time.

Let’s say some words about the infamous Duck Typing approach to see how it fits in this paradigm: program to an interface.

If it looks like a duck and quacks like a duck, it's a duck!

If it looks like a duck and quacks like a duck, it's a duck!

We don’t bother with the nature of the object, we don’t have to care what the object is; we just want to know if it’s able to do what we need (we are only interested in the interface of the object).

Can the object quack? So, let it quack!

try:
bird.quack()
except AttributeError:
self.lol()

Did we define an interface for our duck? No! Did we program to the interface instead of the implementation? Yes! And, I find this so nice.

As Alex Martelli points out in his well known presentation about Design Patterns in Python, “Teaching the ducks to type takes a while, but saves you a lot of work afterwards!”

Favor object composition over inheritance

Now, that’s what I call a Pythonic principle! I have created fewer classes/subclasses compared to wrapping one class (or more often, several classes) in another class.

Instead of doing this:

class User(DbObject):
pass

We can do something like this:

class User:
_persist_methods = ['get', 'save', 'delete']
def __init__(self, persister):
self._persister = persister
def __getattr__(self, attribute):
if attribute in self._persist_methods:
return getattr(self._persister, attribute)

The advantages are obvious. We can restrict what methods of the wrapped class to expose. We can inject the persister instance in runtime! For example, today it’s a relational database, but tomorrow it could be whatever, with the interface we need (again those pesky ducks).

Composition is elegant and natural to Python.

Behavioral Patterns

Behavioural Patterns involve communication between objects, how objects interact and fulfil a given task. According to GOF principles, there are a total of 11 behavioral patterns in Python: Chain of responsibility, Command, Interpreter, Iterator, Mediator, Memento, Observer, State, Strategy, Template, Visitor.

Behavioural patterns deal with inter-object communication, controlling how various objects interact and perform different tasks.

Behavioural patterns deal with inter-object communication, controlling how various objects interact and perform different tasks.

Read the whole article here by Andrei Boyanov, Toptal Freelance Developer

WSGI: The Server-Application Interface for Python

In 1993, the web was still in its infancy, with about 14 million users and a hundred websites. Pages were static but there was already a need to produce dynamic content, such as up-to-date news and data. Responding to this, Rob McCool and other contributors implemented the Common Gateway Interface (CGI) in the National Center for Supercomputing Applications (NCSA) HTTPd web server (the forerunner of Apache). This was the first web server that could serve content generated by a separate application.

Since then, the number of users on the Internet has exploded, and dynamic websites have become ubiquitous. When first learning a new language or even first learning to code, developers, soon enough, want to know about how to hook their code into the web.

Source: Toptal 

8 Essential Python Interview Questions

Source: Toptal 

 

What will be the output of the code below? Explain your answer.

def extendList(val, list=[]):
list.append(val)
return list
list1 = extendList(10)
list2 = extendList(123,[])
list3 = extendList('a')
print "list1 = %s" % list1
print "list2 = %s" % list2
print "list3 = %s" % list3

How would you modify the definition of extendList to produce the presumably desired behavior?

What will be the output of the code below? Explain your answer.

def multipliers():
return [lambda x : i * x for i in range(4)]
print [m(2) for m in multipliers()]

How would you modify the definition of multipliers to produce the presumably desired behavior?

What will be the output of the code below? Explain your answer.

class Parent(object):
x = 1
class Child1(Parent):
pass
class Child2(Parent):
pass
print Parent.x, Child1.x, Child2.x
Child1.x = 2
print Parent.x, Child1.x, Child2.x
Parent.x = 3
print Parent.x, Child1.x, Child2.x

This post originally appeared on the Toptal Engineering blog 

 

What will be the output of the code below in Python 2? Explain your answer.

def div1(x,y):
print "%s/%s = %s" % (x, y, x/y)
def div2(x,y):
print "%s//%s = %s" % (x, y, x//y)
div1(5,2)
div1(5.,2)
div2(5,2)
div2(5.,2.)

Also, how would the answer differ in Python 3 (assuming, of course, that the above print statements were converted to Python 3 syntax)?

What will be the output of the code below?

list = ['a', 'b', 'c', 'd', 'e']
print list[10:]

Consider the following code snippet:

1. list = [ [ ] ] * 5
2. list  # output?
3. list[0].append(10)
4. list  # output?
5. list[1].append(20)
6. list  # output?
7. list.append(30)
8. list  # output?

What will be the ouput of lines 2, 4, 6, and 8? Explain your answer.

Given a list of N numbers, use a single list comprehension to produce a new list that only contains those values that are:
(a) even numbers, and
(b) from elements in the original list that had even indices

For example, if list[2] contains a value that is even, that value should be included in the new list, since it is also at an even index (i.e., 2) in the original list. However, if list[3] contains an even number, that number should not be included in the new list since it is at an odd index (i.e., 3) in the original list.

Given the following subclass of dictionary:

class DefaultDict(dict):
def __missing__(self, key):
return []

Will the code below work? Why or why not?

d = DefaultDict()
d['florp'] = 127