Python Design Patterns: For Sleek And Fashionable Code

Let’s say it again: Python is a high-level programming language with dynamic typing and dynamic binding. I would describe it as a powerful, high-level dynamic language. Many developers are in love with Python because of its clear syntax, well structured modules and packages, and for its enormous flexibility and range of modern features.

In Python, nothing obliges you to write classes and instantiate objects from them. If you don’t need complex structures in your project, you can just write functions. Even better, you can write a flat script for executing some simple and quick task without structuring the code at all.

At the same time Python is a 100 percent object-oriented language. How’s that? Well, simply put, everything in Python is an object. Functions are objects, first class objects (whatever that means). This fact about functions being objects is important, so please remember it.

So, you can write simple scripts in Python, or just open the Python terminal and execute statements right there (that’s so useful!). But at the same time, you can create complex frameworks, applications, libraries and so on. You can do so much in Python. There are of course a number of limitations, but that’s not the topic of this article.

However, because Python is so powerful and flexible, we need some rules (or patterns) when programming in it. So, let see what patterns are, and how they relate to Python. We will also proceed to implement a few essential Python design patterns.

Why Is Python Good For Patterns?

Any programming language is good for patterns. In fact, patterns should be considered in the context of any given programming language. Both the patterns, language syntax and nature impose limitations on our programming. The limitations that come from the language syntax and language nature (dynamic, functional, object oriented, and the like) can differ, as can the reasons behind their existence. The limitations coming from patterns are there for a reason, they are purposeful. That’s the basic goal of patterns; to tell us how to do something and how not to do it. We’ll speak about patterns, and especially Python design patterns, later.

Python is a dynamic and flexible language. Python design patterns are a great way of harnessing its vast potential.

Python is a dynamic and flexible language. Python design patterns are a great way of harnessing its vast potential.

Python’s philosophy is built on top of the idea of well thought out best practices. Python is a dynamic language (did I already said that?) and as such, already implements, or makes it easy to implement, a number of popular design patterns with a few lines of code. Some design patterns are built into Python, so we use them even without knowing. Other patterns are not needed due of the nature of the language.

For example, Factory is a structural Python design pattern aimed at creating new objects, hiding the instantiation logic from the user. But creation of objects in Python is dynamic by design, so additions like Factory are not necessary. Of course, you are free to implement it if you want to. There might be cases where it would be really useful, but they’re an exception, not the norm.

What is so good about Python’s philosophy? Let’s start with this (explore it in the Python terminal):

>>> import this
The Zen of Python, by Tim Peters
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

These might not be patterns in the traditional sense, but these are rules that define the “Pythonic” approach to programming in the most elegant and useful fashion.

We have also PEP-8 code guidelines that help structure our code. It’s a must for me, with some appropriate exceptions, of course. By the way, these exceptions are encouraged by PEP-8 itself:

“But most importantly: know when to be inconsistent – sometimes the style guide just doesn’t apply. When in doubt, use your best judgment. Look at other examples and decide what looks best. And don’t hesitate to ask!”

Combine PEP-8 with The Zen of Python (also a PEP - PEP-20), and you’ll have a perfect foundation to create readable and maintainable code. Add Design Patterns and you are ready to create every kind of software system with consistency and evolvability.

Python Design Patterns

What Is A Design Pattern?

Everything starts with the Gang of Four (GOF). Do a quick online search if you are not familiar with the GOF.

Design patterns are a common way of solving well known problems. Two main principles are in the bases of the design patterns defined by the GOF:

  • Program to an interface not an implementation.
  • Favor object composition over inheritance.

Let’s take a closer look at these two principles from the perspective of Python programmers.

Program to an interface not an implementation

Think about Duck Typing. In Python we don’t like to define interfaces and program classes according these interfaces, do we? But, listen to me! This doesn’t mean we don’t think about interfaces, in fact with Duck Typing we do that all the time.

Let’s say some words about the infamous Duck Typing approach to see how it fits in this paradigm: program to an interface.

If it looks like a duck and quacks like a duck, it's a duck!

If it looks like a duck and quacks like a duck, it's a duck!

We don’t bother with the nature of the object, we don’t have to care what the object is; we just want to know if it’s able to do what we need (we are only interested in the interface of the object).

Can the object quack? So, let it quack!

try:
bird.quack()
except AttributeError:
self.lol()

Did we define an interface for our duck? No! Did we program to the interface instead of the implementation? Yes! And, I find this so nice.

As Alex Martelli points out in his well known presentation about Design Patterns in Python, “Teaching the ducks to type takes a while, but saves you a lot of work afterwards!”

Favor object composition over inheritance

Now, that’s what I call a Pythonic principle! I have created fewer classes/subclasses compared to wrapping one class (or more often, several classes) in another class.

Instead of doing this:

class User(DbObject):
pass

We can do something like this:

class User:
_persist_methods = ['get', 'save', 'delete']
def __init__(self, persister):
self._persister = persister
def __getattr__(self, attribute):
if attribute in self._persist_methods:
return getattr(self._persister, attribute)

The advantages are obvious. We can restrict what methods of the wrapped class to expose. We can inject the persister instance in runtime! For example, today it’s a relational database, but tomorrow it could be whatever, with the interface we need (again those pesky ducks).

Composition is elegant and natural to Python.

Behavioral Patterns

Behavioural Patterns involve communication between objects, how objects interact and fulfil a given task. According to GOF principles, there are a total of 11 behavioral patterns in Python: Chain of responsibility, Command, Interpreter, Iterator, Mediator, Memento, Observer, State, Strategy, Template, Visitor.

Behavioural patterns deal with inter-object communication, controlling how various objects interact and perform different tasks.

Behavioural patterns deal with inter-object communication, controlling how various objects interact and perform different tasks.

I find these patterns very useful, but this does not mean the other pattern groups are not.

Iterator

Iterators are built into Python. This is one of the most powerful characteristics of the language. Years ago, I read somewhere that iterators make Python awesome, and I think this is still the case. Learn enough about Python iterators and generators and you’ll know everything you need about this particular Python pattern.

Chain of responsibility

This pattern gives us a way to treat a request using different methods, each one addressing a specific part of the request. You know, one of the best principles for good code is the Single Responsibility principle.

Every piece of code must do one, and only one, thing.

This principle is deeply integrated in this design pattern.

For example, if we want to filter some content we can implement different filters, each one doing one precise and clearly defined type of filtering. These filters could be used to filter offensive words, ads, unsuitable video content, and so on.

class ContentFilter(object):
def __init__(self, filters=None):
self._filters = list()
if filters is not None:
self._filters += filters
def filter(self, content):
for filter in self._filters:
content = filter(content)
return content
filter = ContentFilter([
offensive_filter,
ads_filter,
porno_video_filter])
filtered_content = filter.filter(content)

Command

This is one of the first Python design patterns I implemented as a programmer. That reminds me: Patterns are not invented, they are discovered. They exist, we just need to find and put them to use. I discovered this one for an amazing project we implemented many years ago: a special purpose WYSIWYM XML editor. After using this pattern intensively in the code, I read more about it on some sites.

The command pattern is handy in situations when, for some reason, we need to start by preparing what will be executed and then to execute it when needed. The advantage is that encapsulating actions in such a way enables Python developers to add additional functionalities related to the executed actions, such as undo/redo, or keeping a history of actions and the like.

Let’s see what a simple and frequently used example looks like:

class RenameFileCommand(object):
def __init__(self, from_name, to_name):
self._from = from_name
self._to = to_name
def execute(self):
os.rename(self._from, self._to)
def undo(self):
os.rename(self._to, self._from)
class History(object):
def __init__(self):
self._commands = list()
def execute(self, command):
self._commands.append(command)
command.execute()
def undo(self):
self._commands.pop().undo()
history = History()
history.execute(RenameFileCommand('docs/cv.doc', 'docs/cv-en.doc'))
history.execute(RenameFileCommand('docs/cv1.doc', 'docs/cv-bg.doc'))
history.undo()
history.undo()

Like what you're reading?
Get the latest updates first.
No spam. Just great engineering and design posts.

Creational Patterns

Let’s start by pointing out that creational patterns are not commonly used in Python. Why? Because of the dynamic nature of the language.

Someone wiser than I once said that Factory is built into Python. It means that the language itself provides us with all the flexibility we need to create objects in a sufficiently elegant fashion; we rarely need to implement anything on top, like Singleton or Factory.

In one Python Design Patterns tutorial, I found a description of the creational design patterns that stated these design “patterns provide a way to create objects while hiding the creation logic, rather than instantiating objects directly using a new operator.”

That pretty much sums up the problem: We don’t have a new operator in Python!

Nevertheless, let’s see how we can implement a few, should we feel we might gain an advantage by using such patterns.

Singleton

The Singleton pattern is used when we want to guarantee that only one instance of a given class exists during runtime. Do we really need this pattern in Python? Based on my experience, it’s easier to simply create one instance intentionally and then use it instead of implementing the Singleton pattern.

But should you want to implement it, here is some good news: In Python, we can alter the instantiation process (along with virtually anything else). Remember the __new__() method I mentioned earlier? Here we go:

class Logger(object):
def __new__(cls, *args, **kwargs):
if not hasattr(cls, '_logger'):
cls._logger = super(Logger, cls
).__new__(cls, *args, **kwargs)
return cls._logger

In this example, Logger is a Singleton.

These are the alternatives to using a Singleton in Python:

  • Use a module.
  • Create one instance somewhere at the top-level of your application, perhaps in the config file.
  • Pass the instance to every object that needs it. That’s a dependency injection and it’s a powerful and easily mastered mechanism.

Dependency Injection

I don’t intend to get into a discussion on whether dependency injection is a design pattern, but I will say that it’s a very good mechanism of implementing loose couplings, and it helps make our application maintainable and extendable. Combine it with Duck Typing and the Force will be with you. Always.

Duck? Human? Python does not care. It's flexible!

Duck? Human? Python does not care. It's flexible!

I listed it in the creational pattern section of this post because it deals with the question of when (or even better: where) the object is created. It’s created outside. Better to say that the objects are not created at all where we use them, so the dependency is not created where it is consumed. The consumer code receives the externally created object and uses it. For further reference, please read the most upvoted answer to this Stackoverflow question.

It’s a nice explanation of dependency injection and gives us a good idea of the potential of this particular technique. Basically the answer explains the problem with the following example: Don’t get things to drink from the fridge yourself, state a need instead. Tell your parents that you need something to drink with lunch.

Python offers us all we need to implement that easily. Think about its possible implementation in other languages such as Java and C#, and you’ll quickly realize the beauty of Python.

Let’s think about a simple example of dependency injection:

class Command:
def __init__(self, authenticate=None, authorize=None):
self.authenticate = authenticate or self._not_authenticated
self.authorize = authorize or self._not_autorized
def execute(self, user, action):
self.authenticate(user)
self.authorize(user, action)
return action()
if in_sudo_mode:
command = Command(always_authenticated, always_authorized)
else:
command = Command(config.authenticate, config.authorize)
command.execute(current_user, delete_user_action)

We inject the authenticator and authorizer methods in the Command class. All the Command class needs is to execute them successfully without bothering with the implementation details. This way, we may use the Command class with whatever authentication and authorization mechanisms we decide to use in runtime.

We have shown how to inject dependencies through the constructor, but we can easily inject them by setting directly the object properties, unlocking even more potential:

command = Command()
if in_sudo_mode:
command.authenticate = always_authenticated
command.authorize = always_authorized
else:
command.authenticate = config.authenticate
command.authorize = config.authorize
command.execute(current_user, delete_user_action)

There is much more to learn about dependency injection; curious people would search for IoC, for example.

But before you do that, read another Stackoverflow answer, the most upvoted one to this question.

Again, we just demonstrated how implementing this wonderful design pattern in Python is just a matter of using the built-in functionalities of the language.

Let’s not forget what all this means: The dependency injection technique allows for very flexible and easy unit-testing. Imagine an architecture where you can change data storing on-the-fly. Mocking a database becomes a trivial task, doesn’t it? For further information, you can check out Toptal’s Introduction to Mocking in Python.

You may also want to research PrototypeBuilder and Factory design patterns.

Structural Patterns

Facade

This may very well be the most famous Python design pattern.

Read the full article in 

Application Development with Rapid Application Development Framework AllcountJS

The idea of Rapid Application Development (RAD) was born as a response to traditional waterfall development models. Many variations of RAD exist; for example, Agile development and the Rational Unified Process. However, all such models have one thing in common: they aim to yield maximum business value with minimal development time through prototyping and iterative development. To accomplish this, the Rapid Application Development model relies on tools that ease the process. In this article, we shall explore one such tool, and how it can be used to focus on business value and optimization of the development process.

AllcountJS is an emerging open source framework built with rapid application development in mind. It is based on the idea of declarative application development using JSON-like configuration code that describes the structure and behavior of the application. The framework has been built on top of Node.js, Express, MongoDB and relies heavily on AngularJS and Twitter Bootstrap. Although it is reliant on declarative patterns, the framework still allows further customization through direct access to API where needed.

AllcountJS as your RAD Framework

Why AllcountJS as Your RAD Framework?

According to Wikipedia, there are at least one hundred tools that promise rapid application development, but this raises the question: how rapid is “rapid.” Do these tools allow a particular data-centric application to be developed in a few hours? Or, perhaps, it is “rapid” if the application can be developed in a few days or a few weeks. Some of these tools even claim that a few minutes is all it takes to produce a working application. However, it is unlikely that you could build a useful application in under five minutes and still claim to have satisfied every business need. AllcountJS doesn’t claim to be such a tool; what AllcountJS offers is a way to prototype an idea in a short period of time.

With AllcountJS framework, it is possible to build an application with a themeable auto-generated user interface, user management features, RESTful API and a handful of other features with minimal effort and time. It is possible to use AllcountJS for a wide variety of use cases, but it best suits applications where you have different collections of objects with different views for them. Typically, business applications are a good fit for this model.

AllcountJS has been used to build allcountjs.com, plus a project tracker for it. It is worth noting that allcountjs.com is a customized AllcountJS application, and that AllcountJS allows both static and dynamic views to be combined with little hassle. It even allows dynamically loaded parts to be inserted into static content. For example, AllcountJS manages a collection of demo application templates. There is a demo widget on the main page of allcountjs.com that loads a random application template from that collection. A handful of other sample applications are available in the gallery at allcountjs.com.

Getting Started

To demonstrate some of the capabilities of RAD framework AllcountJS, we will create a simple application for Toptal, which we will call Toptal Community. If you follow our blog you may already know that a similar application was built using Hoodie as part of one of our earlier blog posts. This application will allow community members to sign up, create events and apply to attend them.

In order to set up the environment, you should install Node.jsMongoDB and Git. Then, install AllcountJS CLI by invoking an “npm install” command and perform project init:

npm install -g allcountjs-cli
allcountjs init toptal-community-allcount
cd toptal-community-allcount
npm install

AllcountJS CLI will ask you to enter some info about your project in order to pre-fill package.json.

AllcountJS can be used as standalone server or as a dependency. In our first example we aren’t going to extend AllcountJS, so a standalone server should just work for us.

Inside this newly created app-config directory, we will replace contents of main.js JavaScript file with the following snippet of code:

A.app({
appName: "Toptal Community",
onlyAuthenticated: true,
allowSignUp: true,
appIcon: "rocket",
menuItems: [{
name: "Events",
entityTypeId: "Event",
icon: "calendar"
}, {
name: "My Events",
entityTypeId: "MyEvent",
icon: "calendar"
}],
entities: function(Fields) {
return {
Event: {
title: "Events",
fields: {
eventName: Fields.text("Event").required(),
date: Fields.date("Date").required(),
time: Fields.text("Starts at").masked("99:99").required(),
appliedUsers: Fields.relation("Applied users", "AppliedUser", "event")
},
referenceName: "eventName",
sorting: [['date', -1], ['time', -1]],
actions: [{
id: "apply",
name: "Apply",
actionTarget: 'single-item',
perform: function (User, Actions, Crud) {
return Crud.actionContextCrud().readEntity(Actions.selectedEntityId()).then(function (eventToApply) {
var userEventCrud = Crud.crudForEntityType('UserEvent');
return userEventCrud.find({filtering: {"user": User.id, "event": eventToApply.id}}).then(function (events) {
if (events.length) {
return Actions.modalResult("Can't apply to event", "You've already applied to this event");
} else {
return userEventCrud.createEntity({
user: {id: User.id},
event: {id: eventToApply.id},
date: eventToApply.date,
time: eventToApply.time
}).then(function () { return Actions.navigateToEntityTypeResult("MyEvent") });
}
});
})
}
}]
},
UserEvent: {
fields: {
user: Fields.fixedReference("User", "OnlyNameUser").required(),
event: Fields.fixedReference("Event", "Event").required(),
date: Fields.date("Date").required(),
time: Fields.text("Starts at").masked("99:99").required()
},
filtering: function (User) { return {"user.id": User.id} },
sorting: [['date', -1], ['time', -1]],
views: {
MyEvent: {
title: "My Events",
showInGrid: ['event', 'date', 'time'],
permissions: {
write: [],
delete: null
}
},
AppliedUser: {
permissions: {
write: []
},
showInGrid: ['user']
}
}
},
User: {
views: {
OnlyNameUser: {
permissions: {
read: null,
write: ['admin']
}
},
fields: {
username: Fields.text("User name")
}
}
}
}
}
});

Although AllcountJS works with Git repositories, for sake of simplicity we will not use it in this tutorial. To run the Toptal Community application, all we have to do is invoke AllcountJS CLI run command in the toptal-community-allcount directory.

allcountjs run

It is worth noting that MongoDB should be running when this command is executed. If all goes well, the application should be up and running at http://localhost:9080.

To login please use the username “admin” and password “admin”.

Less Than 100 Lines

You may have noticed that the application defined in main.js took only 91 lines of code. These lines include the declaration of all the behaviors that you may observe when you navigate to http://localhost:9080. So, what exactly is happening, under the hood? Let us take a closer look at each aspect of the application, and see how the code relates to them.

Sign in & Sign up

The first page you see after opening the application is a sign in. This doubles as a sign up page, assuming that the checkbox - labelled “Sign Up” - is checked before submitting the form.

Sign in & Sign up

This page is shown because the main.js file declares that only authenticated users may use this application. Moreover, it enables the ability for users to sign up from this page. The following two lines are all that was necessary for this:

A.app({
...,
onlyAuthenticated: true,
allowSignUp: true,
...
})

Welcome Page

After signing in, you’ll be redirected to a welcome page with an application menu. This portion of the application is generated automatically, based on the menu items defined under the “menuItems” key.

welcome page example

Along with a couple of other relevant configurations, the menu is defined in the main.js file as follows:

A.app({
...,
appName: "Toptal Community",
appIcon: "rocket",
menuItems: [{
name: "Events",
entityTypeId: "Event",
icon: "calendar"
}, {
name: "My Events",
entityTypeId: "MyEvent",
icon: "calendar"
}],
...
});

AllcountJS uses Font Awesome icons, so all icon names referenced in the configuration are mapped to Font Awesome icon names.

Browsing & Editing Events

After clicking on “Events” from the menu, you’ll be taken to the Events view shown in the screenshot below. It is a standard AllcountJS view that provides some generic CRUD functionalities on the corresponding entities. Here, you may search for events, create new events and edit or delete existing ones. There are two modes of this CRUD interface: list and form. This portion of the application is configured through the following few lines of JavaScript code.

A.app({
...,
entities: function(Fields) {
return {
Event: {
title: "Events",
fields: {
eventName: Fields.text("Event").required(),
date: Fields.date("Date").required(),
time: Fields.text("Starts at").masked("99:99").required(),
appliedUsers: Fields.relation("Applied users", "AppliedUser", "event")
},
referenceName: "eventName",
sorting: [['date', -1], ['time', -1]],
...
}
}
}
});

This example shows how entity descriptions are configured in AllcountJS. Notice how we are using a function to define the entities; every property of AllcountJS configuration can be a function. These functions can request dependencies to be resolved through its argument names. Before the function is called, appropriate dependencies are injected. Here, “Fields” is one of the AllcountJS configuration APIs used to describe entity fields. The property “Entities” contains name-value pairs where the name is an entity-type identifier and value is its description. An entity-type for events is described, in this example, where the title is “Events.” Other configurations, such as default-sort-ordering, reference name, and the like, may also be defined here. Default-sort-order is defined through an array of field names and directions, while the reference name is defined through a string (read more here).

allcountJS function

This particular entity-type has been defined as having four fields: “eventName,” “date,” “time” and “appliedUsers,” the first three of which are persisted in the database. These fields are mandatory, as indicated by the use of “required().” Values in these fields with such rules are validated before the form is submitted on the front-end as shown in the screenshot below. AllcountJS combines both client-side and server-side validations to provide the best user experience. The fourth field is a relationship that bears a list of users who have applied to attend the event. Naturally, this field is not persisted in the database, and is populated by selecting only those AppliedUser entities relevant to the event.

allcountjs development rules

Applying to Attend Events

When a user selects a particular event, the toolbar shows a button labelled “Apply.” Clicking on it adds the event to the user’s schedule. In AllcountJS, actions similar to this can be configured by simply declaring them in the configuration:

actions: [{
id: "apply",
name: "Apply",
actionTarget: 'single-item',
perform: function (User, Actions, Crud) {
return Crud.actionContextCrud().readEntity(Actions.selectedEntityId()).then(function (eventToApply) {
var userEventCrud = Crud.crudForEntityType('UserEvent');
return userEventCrud.find({filtering: {"user": User.id, "event": eventToApply.id}}).then(function (events) {
if (events.length) {
return Actions.modalResult("Can't apply to event", "You've already applied to this event");
} else {
return userEventCrud.createEntity({
user: {id: User.id},
event: {id: eventToApply.id},
date: eventToApply.date,
time: eventToApply.time
}).then(function () { return Actions.navigateToEntityTypeResult("MyEvent") });
}
});
})
}
}]

The property “actions” of any entity type takes an array of objects that describe the behavior of each custom action. Each object has an “id” property which defines a unique identifier for the action, the property “name” defines the display name and the property “actionTarget” is used to define the action context. Setting “actionTarget” to “single-item” indicates that the action should be performed with a particular event. A function defined under the property “perform” is the logic executed when this action is performed, typically when the user clicks on the corresponding button.

Dependencies may be requested by this function. For instance, in this example the function depends on “User,” “Actions” and “Crud.” When an action occurs, a reference to the user, invoking this action, can be obtained by requiring the “User” dependency. The “Crud” dependency, which allows the manipulation of database state for these entities, is also requested here. The two methods that return an instance of Crud object are: The method “actionContextCrud()” - returns CRUD for “Event” entity-type since the action “Apply” belongs to it, while the method “crudForEntityType()” - returns CRUD for any entity type identified by its type ID.

CRUD dependencies

The implementation of the action begins by checking if this event is already scheduled for the user, and if not, it creates one. If it is already scheduled, a dialog box is shown by returning the value from calling “Actions.modalResult()”. Besides showing a modal, an action may perform different types of operations in a similar way, such as “navigate to view,” “refresh view,” “show dialog,” and so on.

implementation of the action

User Schedule of Applied Events

After successfully applying to an event, the browser is redirected to the “My Events” view, which shows a list of events the user has applied to. The view is defined by the following configuration:

UserEvent: {
fields: {
user: Fields.fixedReference("User", "OnlyNameUser").required(),
event: Fields.fixedReference("Event", "Event").required(),
date: Fields.date("Date").required(),
time: Fields.text("Starts at").masked("99:99").required()
},
filtering: function (User) { return {"user.id": User.id} },
sorting: [['date', -1], ['time', -1]],
views: {
MyEvent: {
title: "My Events",
showInGrid: ['event', 'date', 'time'],
permissions: {
write: [],
delete: null
}
},
AppliedUser: {
permissions: {
write: []
},
showInGrid: ['user']
}
}
},

In this case, we are using a new configuration property, “filtering.” As with our earlier example, this function also relies on the “User” dependency. If the function returns an object, it is treated as a MongoDB query; the query filters the collection for events that belong only to the current user.

Another interesting property is “Views.” “View” is a regular entity-type, but it’s MongoDB collection is the same as for parent entity-type. This makes it possible to create visually different views for the same data in the database. In fact, we used this feature to create two different views for “UserEvent:” “MyEvent” and “AppliedUser.” Since the prototype of the sub-views is set to the parent entity type, properties that are not overridden are “inherited” from the parent type.

views

Listing Event Attendees

After applying to an event, other users may see a list of all the users planning to attend. This is generated as a result of the following configuration elements in main.js:

AppliedUser: {
permissions: {
write: []
},
showInGrid: ['user']
}
// ...
appliedUsers: Fields.relation("Applied users", "AppliedUser", "event")

“AppliedUser” is a read-only view for a “MyEvent” entity-type. This read-only permission is enforced by setting an empty array to the “Write” property of the permissions object. Also, as the “Read” permission isn&

Meet RxJava: The Missing Reactive Programming Library for Android

If you’re an Android developer, chances are you’ve heard of RxJava. It’s one of the most discussed libraries for enabling Functional Reactive Programming (FRP) in Android development. It’s touted as the go-to framework for simplifying concurrency/asynchronous tasks inherent in mobile programming.

But… what is RxJava and how does it “simplify” things?

Functional Reactive Programming for Android: An Introduction to RxJava

Untangle your Android from too many Java threads with RxJava.

While there are lots of resources already available online explaining what RxJava is, in this article my goal is to give you a basic introduction to RxJava and specifically how it fits into Android development. I’ll also give some concrete examples and suggestions on how you can integrate it in a new or existing project.

Why Consider RxJava?

At its core, RxJava simplifies development because it raises the level of abstraction around threading. That is, as a developer you don’t have to worry too much about the details of how to perform operations that should occur on different threads. This is particularly attractive since threading is challenging to get right and, if not correctly implemented, can cause some of the most difficult bugs to debug and fix.

Granted, this doesn’t mean RxJava is bulletproof when it comes to threading and it is still important to understand what’s happening behind the scenes; however, RxJava can definitely make your life easier.

Let’s look at an example.

Network Call - RxJava vs AsyncTask

Say we want to obtain data over the network and update the UI as a result. One way to do this is to (1) create an inner AsyncTask subclass in our Activity/Fragment, (2) perform the network operation in the background, and (3) take the result of that operation and update the UI in the main thread.

public class NetworkRequestTask extends AsyncTask<Void, Void, User> {
private final int userId;
public NetworkRequestTask(int userId) {
this.userId = userId;
}
@Override protected User doInBackground(Void... params) {
return networkService.getUser(userId);
}
@Override protected void onPostExecute(User user) {
nameTextView.setText(user.getName());
// ...set other views
}
}
private void onButtonClicked(Button button) {
new NetworkRequestTask(123).execute()
}

Harmless as this may seem, this approach has some issues and limitations. Namely, memory/context leaks are easily created since NetworkRequestTask is an inner class and thus holds an implicit reference to the outer class. Also, what if we want to chain another long operation after the network call? We’d have to nest two AsyncTasks which can significantly reduce readability.

In contrast, an RxJava approach to performing a network call might look something like this:

private Subscription subscription;
private void onButtonClicked(Button button) {
subscription = networkService.getObservableUser(123)
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(new Action1<User>() {
@Override public void call(User user) {
nameTextView.setText(user.getName());
// ... set other views
}
});
}
@Override protected void onDestroy() {
if (subscription != null && !subscription.isUnsubscribed()) {
subscription.unsubscribe();
}
super.onDestroy();
}

Using this approach, we solve the problem (of potential memory leaks caused by a running thread holding a reference to the outer context) by keeping a reference to the returned Subscription object. This Subscription object is then tied to the Activity/Fragment object’s #onDestroy() method to guarantee that the Action1#call operation does not execute when the Activity/Fragment needs to be destroyed.

Also, notice that that the return type of #getObservableUser(...) (i.e. an Observable<User>) is chained with further calls to it. Through this fluid API, we’re able to solve the second issue of using an AsyncTask which is that it allows further network call/long operation chaining. Pretty neat, huh?

Let’s dive deeper into some RxJava concepts.

Observable, Observer, and Operator - The 3 O’s of RxJava Core

In the RxJava world, everything can be modeled as streams. A stream emits item(s) over time, and each emission can be consumed/observed.

If you think about it, a stream is not a new concept: click events can be a stream, location updates can be a stream, push notifications can be a stream, and so on.

The stream abstraction is implemented through 3 core constructs which I like to call “the 3 O’s”; namely: theObservable, Observer, and the Operator. The Observable emits items (the stream); and the Observerconsumes those items. Emissions from Observable objects can further be modified, transformed, and manipulated by chaining Operator calls.

Observable

An Observable is the stream abstraction in RxJava. It is similar to an Iterator in that, given a sequence, it iterates through and produces those items in an orderly fashion. A consumer can then consume those items through the same interface, regardless of the underlying sequence.

Say we wanted to emit the numbers 1, 2, 3, in that order. To do so, we can use the Observable<T>#create(OnSubscribe<T>) method.

Observable<Integer> observable = Observable.create(new Observable.OnSubscribe<Integer>() {
@Override public void call(Subscriber<? super Integer> subscriber) {
subscriber.onNext(1);
subscriber.onNext(2);
subscriber.onNext(3);
subscriber.onCompleted();
}
});

Invoking subscriber.onNext(Integer) emits an item in the stream and, when the stream is finished emitting, subscriber.onCompleted() is then invoked.

This approach to creating an Observable is fairly verbose. For this reason, there are convenience methods for creating Observable instances which should be preferred in almost all cases.

The simplest way to create an Observable is using Observable#just(...). As the method name suggests, it just emits the item(s) that you pass into it as method arguments.

Observable.just(1, 2, 3); // 1, 2, 3 will be emitted, respectively

Observer

The next component to the Observable stream is the Observer (or Observers) subscribed to it. Observers are notified whenever something “interesting” happens in the stream. Observers are notified via the following events:

  • Observer#onNext(T) - invoked when an item is emitted from the stream
  • Observable#onError(Throwable) - invoked when an error has occurred within the stream
  • Observable#onCompleted() - invoked when the stream is finished emitting items.

To subscribe to a stream, simply call Observable<T>#subscribe(...) and pass in an Observer instance.

Observable<Integer> observable = Observable.just(1, 2, 3);
observable.subscribe(new Observer<Integer>() {
@Override public void onCompleted() {
Log.d("Test", "In onCompleted()");
}
@Override public void onError(Throwable e) {
Log.d("Test", "In onError()");
}
@Override public void onNext(Integer integer) {
Log.d("Test", "In onNext():" + integer);
}
});

The above code will emit the following in Logcat:

In onNext(): 1
In onNext(): 2
In onNext(): 3
In onNext(): 4
In onCompleted()

There may also be some instances where we are no longer interested in the emissions of an Observable. This is particularly relevant in Android when, for example, an Activity/Fragment needs to be reclaimed in memory.

To stop observing items, we simply need to call Subscription#unsubscribe() on the returned Subscription object.

Subscription subscription = someInfiniteObservable.subscribe(new Observer<Integer>() {
@Override public void onCompleted() {
// ...
}
@Override public void onError(Throwable e) {
// ...
}
@Override public void onNext(Integer integer) {
// ...
}
});
// Call unsubscribe when appropriate
subscription.unsubscribe();

As seen in the code snippet above, upon subscribing to an Observable, we hold the reference to the returned Subscription object and later invoke subscription#unsubscribe() when necessary. In Android, this is best invoked within Activity#onDestroy() or Fragment#onDestroy().

Operator

Items emitted by an Observable can be transformed, modified, and filtered through Operators before notifying the subscribed Observer object(s). Some of the most common operations found in functional programming (such as map, filter, reduce, etc.) can also be applied to an Observable stream. Let’s look at map as an example:

Observable.just(1, 2, 3, 4, 5).map(new Func1<Integer, Integer>() {
@Override public Integer call(Integer integer) {
return integer * 2;
}
}).subscribe(new Observer<Integer>() {
@Override public void onCompleted() {
// ...
}
@Override public void onError(Throwable e) {
// ...
}
@Override public void onNext(Integer integer) {
// ...
}
});

The code snippet above would take each emission from the Observable and multiply each by 2, producing the stream 2, 4, 6, 8, 10, respectively. Applying an Operator typically returns another Observable as a result, which is convenient as this allows us to chain multiple operations to obtain a desired result.

Given the stream above, say we wanted to only receive even numbers. This can be achieved by chaining afilter operation.

Observable.just(1, 2, 3, 4, 5).map(new Func1<Integer, Integer>() {
@Override public Integer call(Integer integer) {
return integer * 2;
}
}).filter(new Func1<Integer, Boolean>() {
@Override public Boolean call(Integer integer) {
return integer % 2 == 0;
}
}).subscribe(new Observer<Integer>() {
@Override public void onCompleted() {
// ...
}
@Override public void onError(Throwable e) {
// ...
}
@Override public void onNext(Integer integer) {
// ...
}
});


   

Python Best Practices and Tips by Toptal Developers

This resource contains a collection of Python best practices and Python tips provided by our Toptal network members. As such, this page will be updated on a regular basis to include additional information and cover emerging Python techniques. This is a community driven project, so you are encouraged to contribute as well, and we are counting on your feedback.

Python is a high level language used in many development areas, like web development (Django, Flask), data analysis (SciPy, scikit-learn), desktop UI (wxWidgets, PyQt) and system administration (Ansible, OpenStack). The main advantage of Python is development speed. Python comes with rich standard library, a lot of 3rd party libraries and clean syntax. All this allows a developer to focus on the problem they want to solve, and not on the language details or reinventing the wheel.

Check out the Toptal resource pages for additional information on Python. There is a Python hiring guide, Python job descriptioncommon Python mistakes, and Python interview questions.

Be Consistent About Indentation in the Same Python File.

Indentation level in Python is really important, and mixing tabs and spaces is not a smart, nor recommended practice. To go even further, Python 3 will simply refuse to interpret mixed file, while in Python 2 the interpretation of tabs is as if it is converted to spaces using 8-space tab stops. So while executing, you may have no clue at which indentation level a specific line is being considered.

For any code you think someday someone else will read or use, to avoid confusion you should stick with PEP-8, or your team-specific coding style. PEP-8 strongly discourage mixing tabs and spaces in the same file.

For further information, check out this Q&A on StackExchange:

  1. The first downside is that it quickly becomes a mess

… Formatting should be the task of the IDE. Developers have already enough work to care about the size of tabs, how much spaces will an IDE insert, etc. The code should be formatted correctly, and displayed correctly on other configurations, without forcing developers to think about it.

Also, remember this:

Furthermore, it can be a good idea to avoid tabs altogether, because the semantics of tabs are not very well-defined in the computer world, and they can be displayed completely differently on different types of systems and editors. Also, tabs often get destroyed or wrongly converted during copy-paste operations, or when a piece of source code is inserted into a web page or other kind of markup code.

 
This post originally appeared in Toptal
 

How to Create a Simple Python WebSocket Server Using Tornado

With the increase in popularity of real-time web applications, WebSockets have become a key technology in their implementation. The days where you had to constantly press the reload button to receive updates from the server are long gone. Web applications that want to provide real-time updates no longer have to poll the server for changes - instead, servers push changes down the stream as they happen. Robust web frameworks have begun supporting WebSockets out of the box. Ruby on Rails 5, for example, took it even further and added support for action cables.

In the world of Python, many popular web frameworks exist. Frameworks such as Django provide nearly everything necessary to build web applications, and anything that it lacks can be made up with one of the thousands of plugins available for Django. However, due to the way Python or most of its web frameworks work, handling long lived connections can quickly become a nightmare. The threaded model and global interpreter lock are often considered to be the achilles heel of Python.

But all of that has started to change. With certain new features of Python 3 and frameworks that already exist for Python, such as Tornado, handling long lived connections is a challenge no more. Tornado provides web server capabilities in Python that is specifically useful in handling long-lived connections.

 
Read the full article in Toptal 

Top Ten Front-End Design Rules For Developers

As front-end developers, our job is, essentially, to turn designs into reality via code. Understanding, and being competent in, design is an important component of that. Unfortunately, truly understanding front-end design is easier said than done. Coding and aesthetic design require some pretty different skill sets. Because of that, some front-end devs aren’t as proficient in the design aspect as they should be, and as a result, their work suffers.

My goal is to give you some easy-to-follow rules and concepts, from one front-end dev to another, that will help you go from start to finish of a project without messing up what your designers worked so hard on (or possibly even allowing you to design your own projects with decent results).

Of course, these rules won’t take you from bad to magnificent in the time it takes to read one article, but if you apply them to your work, they should make a big difference.

Do Stuff In A Graphics Program

It’s truly rare that you complete a project, and go from start to finish while maintaining every single aesthetic mutation in the design files. And, unfortunately, designers aren’t always around to run to for a quick fix.

Therefore, there always comes a point in any front-end job where you end up having to make some aesthetic-related tweaks. Whether it’s making the checkmark that shows when you check the checkbox, or making a page layout that the PSD missed, front-enders often end up handling these seemingly minor tasks. Naturally, in a perfect world this wouldn’t be the case, but I have yet to find a perfect world, hence we need to be flexible.

A good front-end developer has to use professional graphics tools. Accept no substitute.

A good front-end developer has to use professional graphics tools. Accept no substitute.

For these situations, you should always use a graphics program for mockups. I don’t care which tool you choose: Photoshop, Illustrator, Fireworks, GIMP, whatever. Just don’t just attempt to design from your code. Spend a minute launching a real graphics program and figuring out how it should look, then go to the code and make it happen. You may not be an expert designer, but you’ll still end up with better results.

Match the Design, Don’t Try To Beat It

Your job is not to impress with how unique your checkmark is; your job is to match it to the rest of the design.

Those without a lot of design experience can easily be tempted to leave their mark on the project with seemingly minor details. Please leave that to the designers.

Developers have to match the original front-end design as closely as possible.

Developers have to match the original front-end design as closely as possible.

Instead of asking “Does my checkmark look amazing?” you should be asking, “How well does my checkmark match the design?”

Your focus should always be on working with the design, not on trying to outdo it.

Typography Makes All the Difference

You’d be surprised to know how much of the end look of a design is influenced by typography. You’d be just as surprised to learn how much time designers spend on it. This is not a “pick-it-and-go” endeavor, some serious time and effort goes into it.

If you end up in a situation where you actually have to choose typography, you should spend a decent amount of time doing so. Go online and research good font pairings. Spend a few hours trying those pairings and making sure you end up with the best typography for the project.

Is this font right for your project? When in doubt, consult a designer.

Is this font right for your project? When in doubt, consult a designer.

If you’re working with a design, then make sure you follow the designer’s typography choices. This doesn’t just mean choosing the font, either. Pay attention to the line spacing, letter spacing, and so on. Don’t overlook how important it is to match the typography of the design.

Also, make sure you use the right fonts in the correct spot. If the designer uses Georgia for headers only and Open Sans for body, then you shouldn’t be using Georgia for body and Open Sans for headers. Typography can make or break aesthetics easily. Spend enough time making sure you are matching your designer’s typography. It will be time well spent.

Front-end Design Doesn’t Tolerate Tunnel Vision

You’ll probably be making small parts of the overall design.

Tunnel vision is a common pitfall for front-end developers. Don’t focus on a single detail, always look at the big picture.

Tunnel vision is a common pitfall for front-end developers. Don’t focus on a single detail, always look at the big picture.

An example I’ve been going with is making the checkmark for a design that includes custom checkboxes, without showing them checked. It’s important to remember that the parts you are making are small parts of an overall design. Make your checks as important as a checkmark on a page should look, no more, no less. Don’t get tunnel vision about your one little part and make it something it shouldn’t be.

In fact, a good technique for doing this is to take a screenshot of the program so far, or of the design files, and design within it, in the context in which it will be used. That way, you really see how it affects other design elements on the page, and whether it fits its role properly.

Relationships And Hierarchy

Pay special attention to how the design works with hierarchy. How close are the titles to the body of text? How far are they from the text above them? How does the designer seem to be indicating which elements/titles/text bodies are related and which aren’t? They’ll commonly do these things by boxing related content together, using varying white space to indicate relationships, using similar or contrasting colors to indicate related/unrelated content, and so on.

A good front-end developer will respect design relationships and hierarchy. A great developer will understand them.

A good front-end developer will respect design relationships and hierarchy. A great developer will understand them.

It’s your job to make sure that you recognize the ways in which the design accomplishes relationships and hierarchy and to make sure those concepts are reflected in the end product (including for content that was not specifically designed, and/or dynamic content). This is another area (like typography) where it pays to take extra time to make sure you’re doing a good job.

Be Picky About Whitespace And Alignment

This is a great tip for improving your designs and/or better implementing the designs of others: If the design seems to be using spacings of 20 units, 40 units, etc., then make sure every spacing is a multiple of 20 units.

This is a really drop-dead simple way for someone with no eye for aesthetics to make a significant improvement quickly. Make sure your elements are aligned down to the pixel, and that the spacing around every edge of every element is as uniform as possible. Where you can’t do that (such as places where you need extra space to indicate hierarchy), make them exact multiples of the spacing you’re using elsewhere, for example two times your default to create some separation, three times to create more, and so on.

Do your best to understand how the designer used whitespace and follow those concepts in your front-end build.

Do your best to understand how the designer used whitespace and follow those concepts in your front-end build.

A lot of devs achieve this for specific content in the design files, but when it comes to adding/editing content, or implementing dynamic content, the spacing can go all over the place because they didn’t truly understand what they were implementing.

Do your best to understand how the designer used whitespace and follow those concepts in your build. And yes, spend time on this. Once you think your work is done, go back and measure the spacing to ensure you have aligned and uniformly spaced everything as much as possible, then try out the code with lots of varying content to make sure it’s flexible.

If You Don’t Know What You’re Doing, Do Less

I’m not one of those people that thinks every project should use minimalist design, but if you’re not confident in your design chops and you need to add something, then less is more.

Less is more. If your designer did a good job to begin with, you should refrain from injecting your own design ideas.

Less is more. If your designer did a good job to begin with, you should refrain from injecting your own design ideas.

The designer took care of the main stuff; you only need to do minor fillers. If you’re not very good at design, then a good bet is to do as minimal amount as you can to make that element work. That way, you’re injecting less of your own design into the designer’s work, and affecting it as little as possible.

Let the designer’s work take center stage and let your work take the back seat.

Time Makes Fools Of Us All

I’ll tell you a secret about designers: 90 percent (or more) of what they actually put down on paper, or a Photoshop canvas, isn’t that great.

They discard far more than you ever see. It often takes many revisions and fiddling with a design to get it to the point where they’d even let the guy in the next cubicle see their work, never mind the actual client. You usually don’t go from a blank canvas to good design in one step; there’s a bunch iterations in between. People rarely make good work until they understand that and allow for it in their process.

If you think the design can be improved upon, consult your designer. It’s possible they already tried a similar approach and decided against it.

If you think the design can be improved upon, consult your designer. It’s possible they already tried a similar approach and decided against it.

So how do you implement this? One important method is taking time between versions. Work until it looks like something you like then put it away. Give it a few hours (leaving it overnight is even better), then open it up again and take a look. You’ll be amazed at how different it looks with fresh eyes. You’ll quickly pick out areas for improvement. They’ll be so clear you’ll wonder how you possibly missed them in the first place.

In fact, one of the better designers I’ve known takes this idea a lot further. He would start by making three different designs. Then, he’d wait at least 24 hours, look at them again and throw them all out and start from scratch on a fourth. Next, he’d allow a day between each iteration as it got better and better. Only when he opened it up one morning, and was totally happy, or at least, as close as a designer ever gets to totally happy, would he send it to the client. This was the process he used for every design he made, and it served him very well.

I don’t expect you to take it that far, but it does highlight how helpful time without “eyes on the design” can be. It’s an integral part of the design process and can make improvements in leaps and bounds.

Pixels Matter

You should do everything in your power to match the original design in your finished program, down to the last pixel.

Front-end developers should try to match the original design down to the last pixel.

Front-end developers should try to match the original design down to the last pixel.

In some areas you can’t be perfect. For example, your control over letter-spacing might not be quite as precise as that of the designer’s, and a CSS shadow might not exactly match a Photoshop one, but you should still attempt to get as close as possible. For many aspects of the design, you really can get pixel-perfect precision. Doing so can make a big difference in the end result. A pixel off here and there doesn’t seem like much, but it adds up and affects the overall aesthetic much more than you’d think. So keep an eye on it.

There are a number of [tools] that help you compare original designs to end results, or you can just take screenshots and paste them into the design file to compare each element as closely as possible. Just lay the screenshot over the design and make it semi-transparent so that you can see the differences. Then you know how much adjustment you have to make to get it spot on.

Get Feedback

It’s hard to gain an “eye for design.” It’s even harder to do it on your own. You should seek the input of othersto really see how you can make improvements.

I am not suggesting you grab your neighbor and ask for advice, I mean you should consult real designers and let them critique your work and offer suggestions.

Let designers critique your work. Put their criticism to good use and don’t antagonize them.

Let designers critique your work. Put their criticism to good use and don’t antagonize them.

It takes some bravery to do so, but in the end it is one of the most powerful things you can do to improve the project in the short-term, and to improve your skill level in the long run.

Even if all you have to fine tune is a simple checkmark, there are plenty of people willing to help you. Whether it’s a designer friend, or an online forum, seek out qualified people and get their feedback.

Build a long-lasting, productive relationship with your designers. It’s vital for useful feedback, quality, and execution.

Build a long-lasting, productive relationship with your designers. It’s vital for useful feedback, quality, and execution.

It may sound time consuming, and may cause friction between you and your designers, but in the big scheme of things, it’s worth it. Good front-end developers rely on valuable input from designers, even when it’s not something they like to hear.

Therefore, it’s vital to build and maintain a constructive relationship with your designers. You’re all in the same boat, so to get the best possible results you have to collaborate and communicate every step of the way. The investment in building bonds with your designers is well worth it, as it will help everyone do a better job and execute everything on time.

Conclusion

To summarize, here is a short list of design tips for front-end developers:

  • Design in a graphics program. Don’t design from code, not even the small stuff.
  • Match the design. Be conscious of the original design and don’t try to improve it, just match it.
  • Typography is huge. The time you spend making sure it’s right should reflect its importance.
  • Avoid tunnel vision. Make sure your additions stand out only as much as they should. They’re not more important just because you designed them.
  • Relationships and hierarchy: Understand how they work in the design so that you can implement them properly.
  • Whitespace and alignment are important. Make them accurate to the pixel and make them evenly throughout anything you add.
  • If you’re not confident in your skills, then make your additions as minimally styled as you can.
  • Take time between revisions. Come back later to see your design work with fresh eyes.
  • Pixel-perfect implementation is important wherever possible.
  • Be brave. Seek out experienced designers to critique your work.

Not every front-end developer is going to be a fantastic designer, but every front-end dev should at least becompetent in terms of design.

You need to understand enough about design concepts to identify what’s going on, and to properly apply the design to your end product. Sometimes, you can get away with blind copying if you’ve got a thorough designer (and if you’re detail oriented enough to truly copy it pixel for pixel).

However, in order to make large projects shine across many variations of content, you need some understanding of what’s going through the designer’s head. You don’t merely need to see what the design looks like, you need to know why it looks the way it does, and that way you can be mindful of technical and aesthetic limitations that will affect your job.

So, even as a front-end developer, part of your regular self-improvement should always include learning more about design. 

The original article was written by BRYAN GREZESZAK - FREELANCE SOFTWARE ENGINEER @ TOPTAL and can be read here.

Clean Code and The Art of Exception Handling

Exceptions are as old as programming itself. Back in the days when programming was done in hardware, or via low-level programming languages, exceptions were used to alter the flow of the program, and to avoid hardware failures. Today, Wikipedia defines exceptions as:

anomalous or exceptional conditions requiring special processing – often changing the normal flow of program execution…

And that handling them requires:

specialized programming language constructs or computer hardware mechanisms.

So, exceptions require special treatment, and an unhandled exception may cause unexpected behavior. The results are often spectacular. In 1996, the famous Ariane 5 rocket launch failure was attributed to an unhandled overflow exception. History’s Worst Software Bugs contains some other bugs that could be attributed to unhandled or miss-handled exceptions.

Over time, these errors, and countless others (that were, perhaps, not as dramatic, but still catastrophic for those involved) contributed to the impression that exceptions are bad.

The results of improperly handling exceptions have led us to believe that exceptions are always bad.

But exceptions are a fundamental element of modern programming; they exist to make our software better. Rather than fearing exceptions, we should embrace them and learn how to benefit from them. In this article, we will discuss how to manage exceptions elegantly, and use them to write clean code that is more maintainable.

Exception Handling: It’s a Good Thing

With the rise of object-oriented programming (OOP), exception support has become a crucial element of modern programming languages. A robust exception handling system is built into most languages, nowadays. For example, Ruby provides for the following typical pattern:

begin
do_something_that_might_not_work!
rescue SpecificError => e
do_some_specific_error_clean_up
retry if some_condition_met?
ensure
this_will_always_be_executed
end

There is nothing wrong with the previous code. But overusing these patterns will cause code smells, and won’t necessarily be beneficial. Likewise, misusing them can actually do a lot of harm to your code base, making it brittle, or obfuscating the cause of errors.

The stigma surrounding exceptions often makes programmers feel at a loss. It’s a fact of life that exceptions can’t be avoided, but we are often taught they must be dealt with swiftly and decisively. As we will see, this is not necessarily true. Rather, we should learn the art of handling exceptions gracefully, making them harmonious with the rest of our code.

Following are some recommended practices that will help you embrace exceptions and make use of them and their abilities to keep your code maintainableextensible, and readable:

  • maintainability: Allows us to easily find and fix new bugs, without the fear of breaking current functionality, introducing further bugs, or having to abandon the code altogether due to increased complexity over time.
  • extensibility: Allows us to easily add to our code base, implementing new or changed requirements without breaking existing functionality. Extensibility provides flexibility, and enables a high level of reusability for our code base.
  • readability: Allows us to easily read the code and discover it’s purpose without spending too much time digging. This is critical for efficiently discovering bugs and untested code.

These elements are the main factors of what we might call cleanliness or quality, which is not a direct measure itself, but instead is the combined effect of the previous points, as demonstrated in this comic:

"WTFs/m" by Thom Holwerda, OSNews

With that said, let’s dive into these practices and see how each of them affects those three measures.

Note: We will present examples from Ruby, but all of the constructs demonstrated here have equivalents in the most common OOP languages.

Always create your own ApplicationError hierarchy

Most languages come with a variety of exception classes, organized in an inheritance hierarchy, like any other OOP class. To preserve the readability, maintainability, and extensibility of our code, it’s a good idea to create our own subtree of application-specific exceptions that extend the base exception class. Investing some time in logically structuring this hierarchy can be extremely beneficial. For example:

class ApplicationError < StandardError; end
# Validation Errors
class ValidationError < ApplicationError; end
class RequiredFieldError < ValidationError; end
class UniqueFieldError < ValidationError; end
# HTTP 4XX Response Errors
class ResponseError < ApplicationError; end
class BadRequestError < ResponseError; end
class UnauthorizedError < ResponseError; end
# ...

Example of an application exception hierarchy.

Having an extensible, comprehensive exceptions package for our application makes handling these application-specific situations much easier. For example, we can decide which exceptions to handle in a more natural way. This not only boosts the readability of our code, but also increases the maintainability of our applications and libraries (gems).

From the readability perspective, it’s much easier to read:

rescue ValidationError => e

Than to read:

rescue RequiredFieldError, UniqueFieldError, ... => e

From the maintainability perspective, say, for example, we are implementing a JSON API, and we have defined our own ClientError with several subtypes, to be used when a client sends a bad request. If any one of these is raised, the application should render the JSON representation of the error in its response. It will be easier to fix, or add logic, to a single block that handles ClientErrors rather than looping over each possible client error and implementing the same handler code for each. In terms of extensibility, if we later have to implement another type of client error, we can trust it will already be handled properly here.

Moreover, this does not prevent us from implementing additional special handling for specific client errors earlier in the call stack, or altering the same exception object along the way:

# app/controller/pseudo_controller.rb
def authenticate_user!
fail AuthenticationError if token_invalid? || token_expired?
User.find_by(authentication_token: token)
rescue AuthenticationError => e
report_suspicious_activity if token_invalid?
raise e
end
def show
authenticate_user!
show_private_stuff!(params[:id])
rescue ClientError => e
render_error(e)
end

As you can see, raising this specific exception didn’t prevent us from being able to handle it on different levels, altering it, re-raising it, and allowing the parent class handler to resolve it.

Two things to note here:

  • Not all languages support raising exceptions from within an exception handler.
  • In most languages, raising a new exception from within a handler will cause the original exception to be lost forever, so it’s better to re-raise the same exception object (as in the above example) to avoid losing track of the original cause of the error. (Unless you are doing this intentionally).

Never rescue Exception

That is, never try to implement a catch-all handler for the base exception type. Rescuing or catching all exceptions wholesale is never a good idea in any language, whether it’s globally on a base application level, or in a small buried method used only once. We don’t want to rescue Exception because it will obfuscate whatever really happened, damaging both maintainability and extensibility. We can waste a huge amount of time debugging what the actual problem is, when it could be as simple as a syntax error:

# main.rb
def bad_example
i_might_raise_exception!
rescue Exception
nah_i_will_always_be_here_for_you
end
# elsewhere.rb
def i_might_raise_exception!
retrun do_a_lot_of_work!
end

You might have noticed the error in the previous example; return is mistyped. Although modern editors provide some protection against this specific type of syntax error, this example illustrates how rescue Exception does harm to our code. At no point is the actual type of the exception (in this case a NoMethodError) addressed, nor is it ever exposed to the developer, which may cause us to waste a lot of time running in circles.

Never rescue more exceptions than you need to

The previous point is a specific case of this rule: We should always be careful not to over-generalize our exception handlers. The reasons are the same; whenever we rescue more exceptions than we should, we end up hiding parts of the application logic from higher levels of the application, not to mention suppressing the developer’s ability to handle the exception his or herself. This severely affects the extensibility and maintainability of the code.

If we do attempt to handle different exception subtypes in the same handler, we introduce fat code blocks that have too many responsibilities. For example, if we are building a library that consumes a remote API, handling a MethodNotAllowedError (HTTP 405), is usually different from handling an UnauthorizedError(HTTP 401), even though they are both ResponseErrors.

As we will see, often there exists a different part of the application that would be better suited to handle specific exceptions in a more DRY way.

So, define the single responsibility of your class or method, and handle the bare minimum of exceptions that satisfy this responsibility requirement. For example, if a method is responsible for getting stock info from a remote a API, then it should handle exceptions that arise from getting that info only, and leave the handling of the other errors to a different method designed specifically for these responsibilities:

def get_info
begin
response = HTTP.get(STOCKS_URL + "#{@symbol}/info")
fail AuthenticationError if response.code == 401
fail StockNotFoundError, @symbol if response.code == 404
return JSON.parse response.body
rescue JSON::ParserError
retry
end
end

Here we defined the contract for this method to only get us the info about the stock. It handles endpoint-specific errors, such as an incomplete or malformed JSON response. It doesn’t handle the case when authentication fails or expires, or if the stock doesn’t exist. These are someone else’s responsibility, and are explicitly passed up the call stack where there should be a better place to handle these errors in a DRY way.

Resist the urge to handle exceptions immediately

This is the complement to the last point. An exception can be handled at any point in the call stack, and any point in the class hierarchy, so knowing exactly where to handle it can be mystifying. To solve this conundrum, many developers opt to handle any exception as soon as it arises, but investing time in thinking this through will usually result in finding a more appropriate place to handle specific exceptions.

One common pattern that we see in Rails applications (especially those that expose JSON-only APIs) is the following controller method:

# app/controllers/client_controller.rb
def create
@client = Client.new(params[:client])
if @client.save
render json: @client
else
render json: @client.errors
end
end

(Note that although this is not technically an exception handler, functionally, it serves the same purpose, since @client.save only returns false when it encounters an exception.)

Read the full post in Toptal Engineering blog 

Ruby vs. Python

Toptal freelance experts Damir Zekic and Amar Sahinovic argue the merits of Ruby versus Python, covering everything from speed to performance. Listen to the podcast and weigh in by voting on the superior language and commenting in the thread below. Listen to them debate here

How to Create a Simple Python WebSocket Server Using Tornado

With the increase in popularity of real-time web applications, WebSockets have become a key technology in their implementation. The days where you had to constantly press the reload button to receive updates from the server are long gone. Web applications that want to provide real-time updates no longer have to poll the server for changes - instead, servers push changes down the stream as they happen. Robust web frameworks have begun supporting WebSockets out of the box. Ruby on Rails 5, for example, took it even further and added support for action cables.

In the world of Python, many popular web frameworks exist. Frameworks such as Django provide nearly everything necessary to build web applications, and anything that it lacks can be made up with one of the thousands of plugins available for Django. However, due to the way Python or most of its web frameworks work, handling long lived connections can quickly become a nightmare. The threaded model and global interpreter lock are often considered to be the achilles heel of Python.

But all of that has started to change. With certain new features of Python 3 and frameworks that already exist for Python, such as Tornado, handling long lived connections is a challenge no more. Tornado provides web server capabilities in Python that is specifically useful in handling long-lived connections.

How to Create a Simple Python WebSocket Server using Tornado

In this article, we will take a look at how a simple WebSocket server can be built in Python using Tornado. The demo application will allow us to upload a tab-separated values (TSV) file, parse it and make its contents available at a unique URL.

Tornado and WebSockets

Tornado is an asynchronous network library and specializes in dealing with event driven networking. Since it can naturally hold tens of thousands of open connections concurrently, a server can take advantage of this and handle a lot of WebSocket connections within a single node. WebSocket is a protocol that provides full-duplex communication channels over a single TCP connection. As it is an open socket, this technique makes a web connection stateful and facilitates real-time data transfer to and from the server. The server, keeping the states of the clients, makes it easy to implement real-time chat applications or web games based on WebSockets.

WebSockets are designed to be implemented in web browsers and servers, and is currently supported in all of the major web browsers. A connection is opened once and messages can travel back and forth multiple times before the connection is closed.

Installing Tornado is rather simple. It is listed in PyPI and can be installed using pip or easy_install:

pip install tornado

Tornado comes with its own implementation of WebSockets. For the purposes of this article, this is pretty much all we will need.

WebSockets in Action

One of the advantages of using WebSocket is its stateful property. This changes the way we typically think of client-server communication. One particular use case of this is where the server is required to perform long slow processes and gradually stream results back to the client.

In our example application, the user will be able to upload a file through WebSocket. For the entire lifetime of the connection, the server will retain the parsed file in-memory. Upon requests, the server can then send back parts of the file to the front-end. Furthermore, the file will be made available at a URL which can then be viewed by multiple users. If another file is uploaded at the same URL, everyone looking at it will be able to see the new file immediately.

For front-end, we will use AngularJS. This framework and libraries will allow us to easily handle file uploads and pagination. For everything related to WebSockets, however, we will use standard JavaScript functions.

This simple application will be broken down into three separate files:

  • parser.py: where our Tornado server with the request handlers is implemented
  • templates/index.html: front-end HTML template
  • static/parser.js: For our front-end JavaScript

Opening a WebSocket

From the front-end, a WebSocket connection can be established by instantiating a WebSocket object:

new WebSocket(WEBSOCKET_URL);

This is something we will have to do on page load. Once a WebSocket object is instantiated, handlers must be attached to handle three important events:

  • open: fired when a connection is established
  • message: fired when a message is received from the server
  • close: fired when a connection is closed
$scope.init = function() {
$scope.ws = new WebSocket('ws://' + location.host + '/parser/ws');
$scope.ws.binaryType = 'arraybuffer';
$scope.ws.onopen = function() {
console.log('Connected.')
};
$scope.ws.onmessage = function(evt) {
$scope.$apply(function () {
message = JSON.parse(evt.data);
$scope.currentPage = parseInt(message['page_no']);
$scope.totalRows = parseInt(message['total_number']);
$scope.rows = message['data'];
});
};
$scope.ws.onclose = function() {
console.log('Connection is closed...');
};
}
$scope.init();

Since these event handlers will not automatically trigger AngularJS’s $scope lifecycle, the contents of the handler function needs to be wrapped in $apply. In case you are interested, AngularJS specific packages exist that make it easier to integrate WebSocket in AngularJS applications.

It’s worth mentioning that dropped WebSocket connections are not automatically reestablished, and will require the application to attempt reconnects when the close event handler is triggered. This is a bit beyond the scope of this article.

Selecting a File to Upload

Since we are building a single-page application using AngularJS, attempting to submit forms with files the age-old way will not work. To make things easier, we will use Danial Farid’s ng-file-upload library. Using which, all we need to do to allow a user to upload a file is add a button to our front-end template with specific AngularJS directives:

<button class="btn btn-default" type="file" ngf-select="uploadFile($file, $invalidFiles)"
accept=".tsv" ngf-max-size="10MB">Select File</button>

The library, among many things, allows us to set acceptable file extension and size. Clicking on this button, just like any <input type=”file”> element, will open the standard file picker.

Uploading the File

When you want to transfer binary data, you can choose among array buffer and blob. If it is just raw data like an image file, choose blob and handle it properly in server. Array buffer is for fixed-length binary buffer and a text file like TSV can be transferred in the format of byte string. This code snippet shows how to upload a file in array buffer format.

$scope.uploadFile = function(file, errFiles) {
ws = $scope.ws;
$scope.f = file;
$scope.errFile = errFiles && errFiles[0];
if (file) {
reader = new FileReader();
rawData = new ArrayBuffer();
reader.onload = function(evt) {
rawData = evt.target.result;
ws.send(rawData);
}
reader.readAsArrayBuffer(file);
}
}

The ng-file-upload directive provides an uploadFile function. Here you can transform the file into an array buffer using a FileReader, and send it through the WebSocket.

Note that sending large files over WebSocket by reading them into array buffers may not be the most optimum way to upload them as it can quickly occupy to much memory resulting in a poor experience.

Read the full article inToptal Engineering blog 

Data Mining for Predictive Social Network Analysis

Social networks, in one form or another, have existed since people first began to interact. Indeed, put two or more people together and you have the foundation of a social network. It is therefore no surprise that, in today’s Internet-everywhere world, online social networks have become entirely ubiquitous.

Within this world of online social networks, a particularly fascinating phenomenon of the past decade has been the explosive growth of Twitter, often described as “the SMS of the Internet”. Launched in 2006, Twitter rapidly gained global popularity and has become one of the ten most visited websites in the world. As of May 2015, Twitter boasts 302 million active users who are collectively producing 500 million Tweets per day. And these numbers are continually growing.

Read the full article in Toptal Engineering blog

To Python 3 and Back Again: Is It Worth the Switch?

Python 3 has been in existence for 7 years now, yet some still prefer to use Python 2 instead of the newer version. This is a problem especially for neophytes that are approaching Python for the first time. I realized this at my previous workplace with colleagues in the exact same situation. Not only were they unaware of the differences between the two versions, they were not even aware of the version that they had installed.

Inevitably, different colleagues had installed different versions of the interpreter. That was a recipe for disaster if they would’ve then tried to blindly share the scripts between them.

This wasn’t quite their fault, on the contrary. A greater effort for documenting and raising awareness is needed to dispel that veil of FUD (fear, uncertainty and doubt) that sometimes affects our choices. This post is thus thought for them, or for those who already use Python 2 but aren’t sure about moving to the next version, maybe because they tried version 3 only at the beginning when it was less refined and support for libraries was worse.

Read the full article in the Toptal Engineering blog

An Introduction to Mocking in Python

How to Run Unit Tests Without Testing Your Patience

More often than not, the software we write directly interacts with what we would label as “dirty” services. In layman’s terms: services that are crucial to our application, but whose interactions have intended but undesired side-effects—that is, undesired in the context of an autonomous test run.

For example: perhaps we’re writing a social app and want to test out our new ‘Post to Facebook feature’, but don’t want to actually post to Facebook every time we run our test suite.

The Python unittest library includes a subpackage named unittest.mock—or if you declare it as a dependency, simply mock—which provides extremely powerful and useful means by which to mock and stub out these undesired side-effects.

mocking and unit tests in python

Note: mock is newly included in the standard library as of Python 3.3; prior distributions will have to use the Mock library downloadable via PyPI.

Fear System Calls

To give you another example, and one that we’ll run with for the rest of the article, consider system calls. It’s not difficult to see that these are prime candidates for mocking: whether you’re writing a script to eject a CD drive, a web server which removes antiquated cache files from /tmp, or a socket server which binds to a TCP port, these calls all feature undesired side-effects in the context of your unit-tests.

As a developer, you care more that your library successfully called the system function for ejecting a CD as opposed to experiencing your CD tray open every time a test is run.

As a developer, you care more that your library successfully called the system function for ejecting a CD (with the correct arguments, etc.) as opposed to actually experiencing your CD tray open every time a test is run. (Or worse, multiple times, as multiple tests reference the eject code during a single unit-test run!)

Likewise, keeping your unit-tests efficient and performant means keeping as much “slow code” out of the automated test runs, namely filesystem and network access.

For our first example, we’ll refactor a standard Python test case from original form to one using mock. We’ll demonstrate how writing a test case with mocks will make our tests smarter, faster, and able to reveal more about how the software works.

Read the full article in Toptal Engineering blog 

How I Made Porn 20x More Efficient with Python Video Streaming

Intro

Porn is a big industry. There aren’t many sites on the Internet that can rival the traffic of its biggest players.

And juggling this immense traffic is tough. To make things even harder, much of the content served from porn sites is made up of low latency live video streams rather than simple static video content. But for all of the challenges involved, rarely have I read about the python developers who take them on. So I decided to write about my own experience on the job.

What’s the problem?

A few years ago, I was working for the 26th (at the time) most visited website in the world—not just the porn industry: the world.

At the time, the site served up porn video streaming requests with the Real Time Messaging protocol (RTMP). More specifically, it used a Flash Media Server (FMS) solution, built by Adobe, to provide users with live streams. The basic process was as follows:

  1. The user requests access to some live stream
  2. The server replies with an RTMP session playing the desired footage

For a couple reasons, FMS wasn’t a good choice for us, starting with its costs, which included the purchasing of both:

  1. Windows licenses for every machine on which we ran FMS.
  2. ~$4k FMS-specific licenses, of which we had to purchase several hundred (and more every day) due to our scale.

All of these fees began to rack up. And costs aside, FMS was a lacking product, especially in its functionality (more on this in a bit). So I decided to scrap FMS and write my own Python RTMP parser from scratch.

In the end, I managed to make our service roughly 20x more efficient.

Getting started

There were two core problems involved: firstly, RTMP and other Adobe protocols and formats were not open (i.e., publically available), which made them hard to work with. How can you reverse or parse files in a format about which you know nothing? Luckily, there were some reversing efforts available in the public sphere (not produced by Adobe, but rather by a group called OS Flash, now defunct) on which we based our work.

Note: Adobe later released “specifications” which contained no more information than what was already disclosed in the non-Adobe-produced reversing wiki and documents. Their (Adobe’s) specifications were of an absurdly low quality and made it near impossible to actually use their libraries. Moreover, the protocol itself seemed intentionally misleading at times. For example:

  1. They used 29-bit integers.
  2. They included protocol headers with big endian formatting everywhere—except for a specific (yet unmarked) field, which was little endian.
  3. They squeezed data into less space at the cost of computational power when transporting 9k video frames, which made little to no sense, because they were earning back bits or bytes at a time—insignificant gains for such a file size.

And secondly: RTMP is highly session oriented, which made it virtually impossible to multicast an incoming stream. Ideally, if multiple users wanted to watch the same live stream, we could just pass them back pointers to a single session in which that stream is being aired (this would be multicast video streaming). But with RTMP, we had to create an entirely new instance of the stream for every user that wanted access. This was a complete waste.

Three users demonstrating the difference between a multicast video streaming solution and a FMS streaming problem.

 

Read the full article in Toptal Engineering blog 

Django, Flask, and Redis Tutorial: Web Application Session Management Between Python Frameworks

Django Versus Flask: When Django is the Wrong Choice

I love and use Django in lots of my personal and client projects, mostly for more classical web applications and those involving relational databases. However, Django is no silver bullet.

By design, Django is very tightly coupled with its ORM, Template Engine System, and Settings object. Plus, it’s not a new project: it carries a lot of baggage to remain backwards compatible.

Some Python developers see this as a major problem. They say that Django isn’t flexible enough and avoid it if possible and, instead, use a Python microframework like Flask.

I don’t share that opinion. Django is great when used in the appropriate place and time, even if it doesn’t fit into every project spec. As the mantra goes: “Use the right tool for the job”.

(Even when it is not the right place and time, sometimes programming with Django can have unique benefits.)

In some cases, it can indeed be nice to use a more lightweight framework (like Flask). Often, these microframeworks start to shine when you realize how easy they are to hack on.

Microframeworks to the Rescue

In a few of my client projects, we’ve discussed giving up on Django and moving to a microframework, typically when the clients want to do some interesting stuff (in one case, for example, embedding ZeroMQ in the application object) and the project goals seem more difficult to achieve with Django.

More generally, I find Flask useful for:

  • Simple REST API backends
  • Applications that don’t require database access
  • NoSQL-based web apps
  • Web apps with very specific requirements, like custom URL configurations

At the same time, our app required user registration and other common tasks that Django solved years ago. Given its light weight, Flask doesn’t come with the same toolkit.

The question emerged: is Django an all-or-nothing deal?

The question emerged: is Django an all-or-nothing deal? Should we drop it completely from the project, or can we learn to combine it with the flexibility of other microframeworks or traditional frameworks? Can we pick and choose the pieces we want to use and eschew others?

Can we have the best of both worlds? I say yes, especially when it comes to session management.

(Not to mention, there are a lot of projects out there for Django freelancers.)

Now the Python Tutorial: Sharing Django Sessions

The goal of this post is to delegate the tasks of user authentication and registration to Django, yet use Redis to share user sessions with other frameworks. I can think of a few scenarios in which something like this would be useful:

  • You need to develop a REST API separately from your Django app but want to share session data.
  • You have a specific component that may need to be replaced later on or scaled out for some reason and still need session data.

For this tutorial, I’ll use Redis to share sessions between two frameworks (in this case, Django and Flask). In the current setup, I’ll use SQLite to store user information, but you can have your back-end tied to a NoSQL database (or a SQL-based alternative) if need be.

Understanding Sessions

To share sessions between Django and Flask, we need to know a bit about how Django stores its session information. The Django docs are pretty good, but I’ll provide some background for completeness.

Session Management Varieties

Generally, you can choose to manage your Python app’s session data in one of two ways:

  • Cookie-based sessions: In this scenario, the session data is not stored in a data store on the back-end. Instead, it’s serialized, signed (with a SECRET_KEY), and sent to the client. When the client sends that data back, its integrity is checked for tampering and it is deserialized again on the server.

  • Storage-based sessions: In this scenario, the session data itself is not sent to the client. Instead, only a small portion is sent (a key) to indicate the identity of the current user, stored on the session store.

In our example, we’re more interested in the latter scenario: we want our session data to be stored on the back-end and then checked in Flask. The same thing could be done in the former, but as the Django documentation mentions, there are some concerns about the security of the first method.

The General Workflow

The general workflow of session handling and management will be similar to this diagram:

A diagram showing the management of user sessions between Flask and Django using Redis.

Let’s walk through session sharing in a little more detail:

  1. When a new request comes in, the first step is to send it through the registered middleware in the Django stack. We’re interested here in the SessionMiddleware class which, as you might expect, is related to session management and handling:

    	class SessionMiddleware(object):
    	def process_request(self, request):
    	engine = import_module(settings.SESSION_ENGINE)
    	session_key = request.COOKIES.get(settings.SESSION_COOKIE_NAME, None)
    	request.session = engine.SessionStore(session_key)
    	
    	

    In this snippet, Django grabs the registered SessionEngine (we’ll get to that soon), extracts the SESSION_COOKIE_NAME from request (sessionid, by default) and creates a new instance of the selected SessionEngine to handle session storage.

  • Later on (after the user view is processed, but still in the middleware stack), the session engine calls its save method to save any changes to the data store. (During view handling, the user may have changed a few things within the session, e.g., by adding a new value to session object with request.session.) Then, the SESSION_COOKIE_NAME is sent to the client. Here’s the simplified version:

    	def process_response(self, request, response):
    	....
    	if response.status_code != 500:
    	request.session.save()
    	response.set_cookie(settings.SESSION_COOKIE_NAME,
    	request.session.session_key, max_age=max_age,
    	expires=expires, domain=settings.SESSION_COOKIE_DOMAIN,
    	path=settings.SESSION_COOKIE_PATH,
    	secure=settings.SESSION_COOKIE_SECURE or None,
    	httponly=settings.SESSION_COOKIE_HTTPONLY or None)
    	return response
    	
    	

We’re particularly interested in the SessionEngine class, which we’ll replace with something to store and load data to and from a Redis back-end.

Fortunately, there are a few projects that already handle this for us. Here’s an example from redis_sessions_fork. Pay close attention to the  save and load methods, which are written so as to (respectively) store and load the session into and from Redis:

class SessionStore(SessionBase):
"""
Redis session back-end for Django
"""
def __init__(self, session_key=None):
super(SessionStore, self).__init__(session_key)
def _get_or_create_session_key(self):
if self._session_key is None:
self._session_key = self._get_new_session_key()
return self._session_key
def load(self):
session_data = backend.get(self.session_key)
if not session_data is None:
return self.decode(session_data)
else:
self.create()
return {}
def exists(self, session_key):
return backend.exists(session_key)
def create(self):
while True:
self._session_key = self._get_new_session_key()
try:
self.save(must_create=True)
except CreateError:
continue
self.modified = True
self._session_cache = {}
return
def save(self, must_create=False):
session_key = self._get_or_create_session_key()
expire_in = self.get_expiry_age()
session_data = self.encode(self._get_session(no_load=must_create))
backend.save(session_key, expire_in, session_data, must_create)
def delete(self, session_key=None):
if session_key is None:
if self.session_key is None:
return
session_key = self.session_key
backend.delete(session_key)

It’s important to understand how this class is operating as we’ll need to implement something similar on Flask to load session data. Let’s take a closer look with a REPL example:

>>> from django.conf import settings
>>> from django.utils.importlib import import_module
>>> engine = import_module(settings.SESSION_ENGINE)
>>> engine.SessionStore()
<redis_sessions_fork.session.SessionStore object at 0x3761cd0>
>>> store["count"] = 1
>>> store.save()
>>> store.load()
{u'count': 1}

The session store’s interface is pretty easy to understand, but there’s a lot going on under the hood. We should dig a little deeper so that we can implement something similar on Flask.

Note: You might ask, “Why not just copy the SessionEngine into Flask?” Easier said than done. As we discussed in the beginning, Django is tightly coupled with its Settings object, so you can’t just import some Django module and use it without any additional work.

Django Session (De-)Serialization

As I said, Django does a lot of work to mask the complexity of its session storage. Let’s check the Redis key that’s stored in the above snippets:

>>> store.session_key
u"ery3j462ezmmgebbpwjajlxjxmvt5adu"

Now, lets query that key on the redis-cli:

redis 127.0.0.1:6379> get "django_sessions:ery3j462ezmmgebbpwjajlxjxmvt5adu"
"ZmUxOTY0ZTFkMmNmODA2OWQ5ZjE4MjNhZmQxNDM0MDBiNmQzNzM2Zjp7ImNvdW50IjoxfQ=="

What we see here is a very long, Base64-encoded string. To understand its purpose, we need to look at Django’s SessionBase class to see how it’s handled:

class SessionBase(object):
"""
Base class for all Session classes.
"""
def encode(self, session_dict):
"Returns the given session dictionary serialized and encoded as a string."
serialized = self.serializer().dumps(session_dict)
hash = self._hash(serialized)
return base64.b64encode(hash.encode() + b":" + serialized).decode('ascii')
def decode(self, session_data):
encoded_data = base64.b64decode(force_bytes(session_data))
try:
hash, serialized = encoded_data.split(b':', 1)
expected_hash = self._hash(serialized)
if not constant_time_compare(hash.decode(), expected_hash):
raise SuspiciousSession("Session data corrupted")
else:
return self.serializer().loads(serialized)
except Exception as e:
# ValueError, SuspiciousOperation, unpickling exceptions
if isinstance(e, SuspiciousOperation):
logger = logging.getLogger('django.security.%s' %
e.__class__.__name__)
logger.warning(force_text(e))
return {}

The encode method first serializes the data with the current registered serializer. In other words, it converts the session into a string, which it can later convert back into a session (look at the SESSION_SERIALIZER documentation for more). Then, it hashes the serialized data and uses this hash later on as a signature to check the integrity of the session data. Finally, it returns that data pair to the user as a Base64-encoded string.

By the way: before version 1.6, Django defaulted to using pickle for serialization of session data. Due to security concerns, the default serialization method is now django.contrib.sessions.serializers.JSONSerializer.

Encoding an Example Session

Let’s see the session management process in action. Here, our session dictionary will simply be a count and some integer, but you can imagine how this would generalize to more complicated user sessions.

>>> store.encode({'count': 1})
u'ZmUxOTY0ZTFkMmNmODA2OWQ5ZjE4MjNhZmQxNDM0MDBiNmQzNzM2Zjp7ImNvdW50IjoxfQ=='
>>> base64.b64decode(encoded)
'fe1964e1d2cf8069d9f1823afd143400b6d3736f:{"count":1}'

The result of the store method (u’ZmUxOTY…==’) is an encoded string containing the serialized user session and its hash. When we decode it, we indeed get back both the hash (‘fe1964e…’) and the session ({"count":1}).

Note that the decode method checks to ensure that the hash is correct for that session, guaranteeing integrity of the data when we go to use it in Flask. In our case, we’re not too worried about our session being tampered with on the client side because:

  • We aren’t using cookie-based sessions, i.e., we’re not sending all user data to the client.

  • On Flask, we’ll need a read-only SessionStore which will tell us if given key exists or not and return the stored data.

Extending to Flask

Next, let’s create a simplified version of the Redis session engine (database) to work with Flask. We’ll use the same SessionStore (defined above) as a base class, but we’ll need to remove some of its functionality, e.g., checking for bad signatures or modifying sessions. We’re more interested in a read-only SessionStore that will load the session data saved from Django. Let’s see how it comes together:

class SessionStore(object):
# The default serializer, for now
def __init__(self, conn, session_key, secret, serializer=None):
self._conn = conn
self.session_key = session_key
self._secret = secret
self.serializer = serializer or JSONSerializer
def load(self):
session_data = self._conn.get(self.session_key)
if not session_data is None:
return self._decode(session_data)
else:
return {}
def exists(self, session_key):
return self._conn.exists(session_key)
def _decode(self, session_data):
"""
Decodes the Django session
:param session_data:
:return: decoded data
"""
encoded_data = base64.b64decode(force_bytes(session_data))
try:
# Could produce ValueError if there is no ':'
hash, serialized = encoded_data.split(b':', 1)
# In the Django version of that they check for corrupted data
# I don't find it useful, so I'm removing it
return self.serializer().loads(serialized)
except Exception as e:
# ValueError, SuspiciousOperation, unpickling exceptions. If any of
# these happen, return an empty dictionary (i.e., empty session).
return {}

We only need the load method because it’s a read-only implementation of the storage. That means you can’t logout directly from Flask; instead, you might want to redirect this task to Django. Remember, the goal here is to manage sessions between these two Python frameworks to give you more flexibility.

Flask Sessions

The Flask microframework supports cookie-based sessions, which means all of the session data is sent to the client, Base64-encoded and cryptographically signed. But actually, we’re not very interested in Flask’s session support.

What we need is to get the session ID created by Django and check it against the Redis back-end so that we can be sure the request belongs to a pre-signed user. In summary, the ideal process would be (this syncs up with the diagram above):

  • We grab the Django session ID from the user’s cookie.
  • If the session ID is found in Redis, we return the session matching that ID.
  • If not, we redirect them to a login page.

It’ll be handy to have a decorator to check for that information and set the current user_id into the gvariable in Flask:

from functools import wraps
from flask import g, request, redirect, url_for
def login_required(f):
@wraps(f)
def decorated_function(*args, **kwargs):
djsession_id = request.cookies.get("sessionid")
if djsession_id is None:
return redirect("/")
key = get_session_prefixed(djsession_id)
session_store = SessionStore(redis_conn, key)
auth = session_store.load()
if not auth:
return redirect("/")
g.user_id = str(auth.get("_auth_user_id"))
return f(*args, **kwargs)
return decorated_function

In the example above, we’re still using the SessionStore we defined previously to fetch the Django data from Redis. If the session has an _auth_user_id, we return the content from the view function; otherwise, the user is redirected to a login page, just like we wanted.

Gluing Things Together

In order to share cookies, I find it convenient to start Django and Flask via a WSGI server and glue them together. In this example, I’ve used CherryPy:

from app import app
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()
d = wsgiserver.WSGIPathInfoDispatcher({
"/":application,
"/backend":app
})
server = wsgiserver.CherryPyWSGIServer(("127.0.0.1", 8080), d)

With that, Django will serve on “/” and Flask will serve on “/backend” endpoints.

In Conclusion

Rather than examining Django versus Flask or encouraging you only to learn the Flask microframework, I’ve welded together Django and Flask, getting them to share the same session data for authentication by delegating the task to Django. As Django ships with plenty of modules to solve user registration, login, and logout (just to name a few), combining these two frameworks will save you valuable time while providing you with the opportunity to hack on a manageable microframework like Flask.

This post originally appeared in the Toptal Engineering blog 

The Python Tutorial

Python is an easy to learn, powerful programming language. It has efficient high-level data structures and a simple but effective approach to object-oriented programming. Python’s elegant syntax and dynamic typing, together with its interpreted nature, make it an ideal language for scripting and rapid application development in many areas on most platforms.

The Python interpreter and the extensive standard library are freely available in source or binary form for all major platforms from the Python Web site,https://www.python.org/, and may be freely distributed. The same site also contains distributions of and pointers to many free third party Python modules, programs and tools, and additional documentation.

The Python interpreter is easily extended with new functions and data types implemented in C or C++ (or other languages callable from C). Python is also suitable as an extension language for customizable applications.

This tutorial introduces the reader informally to the basic concepts and features of the Python language and system. It helps to have a Python interpreter handy for hands-on experience, but all examples are self-contained, so the tutorial can be read off-line as well.

For a description of standard objects and modules, see The Python Standard LibraryThe Python Language Reference gives a more formal definition of the language. To write extensions in C or C++, read Extending and Embedding the Python Interpreter and Python/C API Reference Manual. There are also several books covering Python in depth.

This tutorial does not attempt to be comprehensive and cover every single feature, or even every commonly used feature. Instead, it introduces many of Python’s most noteworthy features, and will give you a good idea of the language’s flavor and style. After reading it, you will be able to read and write Python modules and programs, and you will be ready to learn more about the various Python library modules described in The Python Standard Library.

1 2  Next»