# remove HTML from the string content and place the result in doc
doc = html.document_fromstring(content)
# get the text content of doc (with no markup) and place the result in text_doc 
text_doc = doc.text_content()

thus text_doc now contains the text of the original string with no HTML markup in it. Now you can use something like Postmarkup so that users can use BBCode or a similar markup tool to style the text that they place on your site without having the security risk of allowing users to post HTML.

If you are using Django it is essential that you mark your strings as safe using the safe template tag so that Django does not automatically escape the HTML created legitimately from the BBCode.

A full example that uses Postmarkup and lxml to first remove any HTML in a post and then render the BBCode into HTML for display on a site:

from postmarkup import render_bbcode
from lxml import html
 
doc = html.document_fromstring(content)
bbcode = doc.text_content()
content = render_bbcode(bbcode)

The first task is to configure Arch Linux using the official installation manual which is excellent. I won’t be covering anything that is included in the installation guide as the provided documentation is already good enough.

Once you have finished with the initial configuration of Arch Linux we should move on to installing the requirements. We will be using SLiM as our login manager as that is nice and lightweight.

Run the following command to install all the requirements to get XFCE up and running:

pacman -S xfce4 ttf-liberation xorg-server dbus slim

you will also need to install the correct graphics drivers for your card. The following are the likely options:

xf86-video-ati
xf86-video-nvidia
xf86-video-intel

so install the correct one for your system like so:

pacman -S xf86-video-ati

now that is done you simply need to copy /etc/skel/.xinitrc to your home directory (and for all the other users you wish to be able to login using SLiM) like so:

cp /etc/skel/.xinitrc ~/.xinitrc

and uncomment the following line:

# exec startxfce4

so that it looks like this:

exec startxfce4

if the line does not exist in the file then simply add it to the bottom of the file and make sure that all other exec lines are commented out.

Now all you need to do is add dbus and slim to your list of daemons by editing /etc/rc.conf like so:

DAEMONS=(... dbus slim)

it is vital that dbus and slim are the last two entries in the daemons list. Simply restart your computer and you should be able to login to your new XFCE 4 desktop environment.

pacman -S ntp

In order to get this working visit http://www.pool.ntp.org/ and select the correct region. Once you know which ntp servers you want to use add the servers to /etc/ntp.conf as shown below:

server 0.europe.pool.ntp.org iburst
server 1.europe.pool.ntp.org iburst
server 2.europe.pool.ntp.org iburst
server 3.europe.pool.ntp.org iburst

substitute the servers for the correct servers in your region. The iburst option is recommended. Then we need to make sure that ntpd runs automatically so we add it to our DAEMONS section of the /etc/rc.conf file like so:

DAEMONS=(... !hwclock ntpd ...)

this disables the hardware clock and makes sure that ntpd is used instead to keep the time in sync.

Now lets just finally make sure ntpd runs as a non-root user and in a chroot environment for added security:

groupadd ntp
useradd -r -d /var/lib/ntp -g ntp -s /bin/false ntp

then

chown -R ntp:ntp /var/lib/ntp

once you have created the user and group for ntpd and changed the permissions on the /var/lib/ntp directory you just need to edit /etc/conf.d/ntp-client.conf by editing the NTPD_ARGS line so it looks like this:

NTPD_ARGS="-g -i /var/lib/ntp -u ntp:ntp"

and edit /etc/ntp.conf so that the following line:

driftfile       /var/lib/ntp/ntp.drift

reads:

driftfile       /ntp.drift

then finally restart ntpd and it should work perfectly:

rc.d restart ntpd

I first of all installed Arch Linux using the net install media so as to get the latest packages right from the get go. If you have a reasonable internet connection I highly recommend that you follow this path.

I won’t cover the initial installation as that is covered already by the excellent installation manual. The only thing I will say is that if you get errors about your locale you may need to add the following to your ~/.bashrc file:

export LANGUAGE=en_GB.UTF-8
export LANG=en_GB.UTF-8
export LC_ALL=en_GB.UTF-8

of course you’ll need to substitute the locale for the one that you want to use (the British one is used in the above example). This didn’t happen on my second install so it could be down to me getting something wrong on my first attempt.

Once you have completed the instructions in the installation guide we can move on to getting Gnome 3 installed and configured.

The first step is to create a non-root user for your own use and to also install the sudo command. The easiest way to create a new user is to use the adduser command.

adduser

Just follow through the commands choosing the default for all options other than the username, additional groups and your full name. For the additional groups section add the following exactly as shown:

audio,lp,optical,storage,video,wheel,games,power,scanner,floppy

this allows the user to use the specific software in each category. If you don’t have a floppy disk drive in your computer you can leave that group out.

I highly recommend that you follow my guide on getting ntpd installed before going further with this tutorial.

Now that we have ntpd properly configured we can move on to getting X11 and Gnome installed and working.

Installing X11 is a trivial process and simply consists of running the following command:

pacman -S xorg-server

you’ll also need to install the correct graphics drivers for your card. You have three options based on the type of graphics card that you have (there maybe more options but I doubt many people have a graphics card from a different manufacturer). These are listed below:

xf86-video-ati
xf86-video-nvidia
xf86-video-intel

pick the one that matches your card and install it using the following command:

pacman -S xf86-video-ati

obviously you’ll need to substitute the ATI drivers if you have a graphics card made by Intel or Nvidia.

Now that X11 is installed we can move straight on to getting Gnome 3 installed and configured. Install all the requirements as follows:

pacman -S gnome gdm dbus ttf-liberation

and when pacman asks you which packages to install just press return to select all of the options.

Now all you need to do is add dbus and gdm to your daemons configuration option in your rc.conf and away you go!

DAEMONS=(... dbus gdm)

Note: make sure dbus and gdm are the last two items in your DAEMONS list.

Congratulations, you should now have a fully working Gnome 3 install on your Arch Linux box. Any other customisation you wish to do is down to you.

Having tried Gunicorn and Nginx (which admittedly was easier to get working for a single site) I have to say that hosting Django apps using Nginx, FastCGI and Daemon Tools is by far the best method. It avoids having to write init scripts entirely (a task which is still somewhat nasty especially considering the alternatives on other non-Linux platforms such as launchd on Mac OS X which is infinitely superior).

Python

The only real requirements on the Python side are that you have virtualenv and flup installed. Having said that using Pip and virtualenvwrapper is also highly recommended in all Python based deployments no matter what the framework(s) you are using.

Also due to the fact that many Python projects make use of Mercurial you should also take the time to learn it if you do not already know how to use it.

Nginx

The first task is to install nginx. I always install nginx from source as it allows me to cut out some the modules that I have no need for and thus reduce memory footprint a bit more. You can decide if you want to install from source or not.

First we need to install the dependencies for building nginx (I am using Debian 6 here so change it to whatever your platform requires):

apt-get install gcc libpcre3-dev libssl-dev make

Then run configure, make and make install:

./configure --prefix=/opt/nginx --user=nginx \
--group=nginx --with-http_ssl_module --with-http_realip_module \
--with-http_secure_link_module --with-http_gzip_static_module \
--without-http_scgi_module --without-http_uwsgi_module \
--without-http_autoindex_module --without-http_ssi_module \
--without-http_split_clients_module --without-http_userid_module \
--without-http_memcached_module --without-mail_pop3_module \
--without-mail_imap_module --without-mail_smtp_module \
--without-http_auth_basic_module --without-http_charset_module
 
make
 
make install

(you need to be root or use sudo when running make install).

Now we add the user for nginx:

adduser --system --no-create-home --disabled-login \
--disabled-password --group nginx

Once that is done nginx is ready to go. We just need to setup an init script for it. Since I host with Linode (who are pretty darn good) I use the init script that they provide. If you use a distribution other than Debian / Ubuntu you might have to make some changes or get one specifically for your distribution.

wget http://library.linode.com/webservers/nginx/installation/reference/init-deb.sh
 
mv init-deb.sh /etc/init.d/nginx
 
chmod +x /etc/init.d/nginx
 
/usr/sbin/update-rc.d -f nginx defaults

Now nginx is installed. So lets start it and move onto the next stage. We will configure the nginx page separately after we have configured daemon tools.

/etc/init.d/nginx start

Daemon Tools

Now lets install Daemon Tools.

apt-get install daemontools-run

Once that has been completed create a directory to store our first sites configuration file. You will create a directory for every project you wish to run on the server. I generally tend to name each folder after the domain of the site to make it easy to tell which directory holds the configuration for each site (although honestly it is unlikely you will need to play around with these configuration files once they are setup and working properly).

mkdir /etc/service/domain.com

Now just create our configuration file for each Django project. Save it as a file called “run” (without the quotes) in the folder you created above for you domain name and then enter something like this:

#!/usr/bin/env bash
 
source /path/to/your/virtualenv/for/project
PROJ_DIR=/path/to/django/project
 
exec envuidgid username python $PROJ_DIR/manage.py \
    runfcgi method=threaded minspare=1 maxspare=2 host=127.0.0.1 \
    port=9001 pidfile=$PROJ_DIR/django_fcgi.pid daemonize=false

the configuration shown above is for a very low traffic site so if you want to use it in production increase the number the maxspare option to something like 5 – 10 and the minspare to at least 2. Be aware that this will require quite a bit of free RAM so you might have to play around with options depending on how much you have available. You will also need to change “username” in the above example to the name of the user you wish to run the Django apps process.

Now lets us return to our nginx configuration.

Nginx

In order to avoid problems with displaying URLs in our Django application we need to add a line to nginx’s default fastcgi_params file (note that this tutorial was written using version 0.8.54 of nginx, if you are using a different version it might be worth checking if it already exists before adding it).

So just add the following line to the fastcgi_params file:

fastcgi_param  PATH_INFO          $fastcgi_script_name;

then we need to add the following line to our Django projects settings.py file (personally I have a special production_settings.py file which I have imported in my settings.py file which I leave commented out during development and only uncomment when I deploy to production):

FORCE_SCRIPT_NAME = ''

then we just add the correct configuration for our domain to our nginx configuration file:

server {
    listen 80;
    server_name domain.com;
    rewrite ^/(.*) http://www.domain.com/$1 permanent;
}
 
server {
    listen 80;
    server_name www.domain.com;
 
    access_log /srv/www/domain.com/logs/access.log;
    error_log /srv/www/domain.com/logs/error.log;
 
    root /srv/www/domain.com/public_html;
    index index.html;
 
    location / {
        try_files $uri @django;
    }
 
    location /static {
        alias /path/to/project/static;
    }
 
    location /media {
        alias /path/to/project/media;
    }
 
    location @django {
        include /opt/nginx/conf/fastcgi_params;
        fastcgi_pass 127.0.0.1:9001;
        fastcgi_pass_header Authorization;
        fastcgi_intercept_errors off;
    }
}

and away we go.

This is the basic setup fully configured. All we need to do now is make sure that Daemon tools is running the correct version of our app and that nginx has all the changes we have made loaded. So simply restart out domains service:

svc -du /etc/service/domain.com/

and restart nginx:

/etc/init.d/nginx restart

for reference if you just want to stop your Django app:

svc -d /etc/service/domain.com/

or start it:

svc -u /etc/service/domain.com/

and now you should be able to view your site (assuming DNS has propogated) and see the spender of your new Django application running on possibly the easiest to maintain and update Django web stack available. Hopefully you use some form of distributed source code control such as Mercurial or Git. If so you should be able to update you project simply by pulling the latest version and then just restart the Daemon Tools service with:

svc -du /etc/service/domain.com/

I hope you have found this article useful. If so (or even if you have not) leave a comment. Also leave a comment if you have any problems and I’ll try and help you out.

Since Github is an American company they are legally required to not export certain cryptographic software. Thus any open source project that makes use of Github (or indeed any source code repository based in the US) is going to fall under US export law. I know for a fact that certain projects (OpenBSD and OpenSSH spring immediately to mind) take great pains to ensure that none of their cryptographic software is developed in places where it is export restricted to ensure that they can produce the best software possible. So from their point of view I can easily see why they would wish to avoid sites such as Github.

It is hard to see whether the US government could in fact force Github to limit the type of software that they did export though without irreparably damaging their business. The fact that Github has become so popular in the open source world is testament to the great service that they provide but I doubt many open source projects would sacrifice security just to use said service.

So is this really an issue? I have yet to hear of Github or any other company offering the same kind of service removing or limiting the export of cryptographic software, yet the sword of Damocles will forever be hanging over your head should you choose to use it.

Perhaps more open source software will make use of something like Gitorious which allows you to host your own repositories in a fairly pleasant manner.

The major problem with my original article was that I was passing everything through the C compiler rather than just using avr-as as one should. The other problem was that I was using the libc headers to get the names of the specific ports. The better solution is to take the include files that Atmel supply with AVR Studio 4 on the PC and just make a couple of small changes to make them work using avr-as. The include file I use for the ATmega168 can be found here.

By using this approach there is basically no difference between the assembly that you would use in AVR Studio 4 and that you can use on your Mac. This makes the entire development process extremely easy.

So lets list the source code of our updated LEDon.S file from the original article:

.include "/home/dev/m168.h"
 
	rjmp init
 
init:
 
	ser r16
	out DDRB, r16
	out DDRD, r16
 
	clr r16
 
	out PORTB, r16
	out PORTD, r16
 
	.global main
 
main:
 
	sbi PORTB, 0
	rjmp main

it should look pretty similar to the original.

The only real difference comes when you want to assemble it and produce a hex file which you then upload with AVRdude (as documented here). So you do that with these commands (assuming you have saved the above file as ledon.S and that the header file I provided is stored in /home/dev – if it is different change the include line to point to the correct path):

avr-as -mmcu=atmega168 -o ledon.out ledon.S
avr-objcopy -O ihex ledon.out ledon.hex

this should result in a nice ledon.hex file.

Now that you can use the “raw” assembler you can just read the data sheets and the AVR instruction reference and use the port names as is rather than having to use the rather stupid C macros that AVR libc provides. Obviously this technique is completely cross platform and will work on any UNIX based machine, not just Macs.

Although I am late to the party when it comes to discussing the new Mac OS X app store I think it should be noted that this post comes after some thought on the subject. The major disadvantage of the Mac app store is the fact that Mac OS X customers will start to see it as the only legitimate place to buy Mac software from. Apple have been very careful to make it clear that developers are free to distribute applications separately from the app store if they wish but I worry about it when I think of a consumer choosing between an app which is conveniently located on the app store and one which is not. Which app do you think they will be most likely to purchase?

The fact that Apple do not deny developers the chance to distribute applications on their own does not mean that the effects of the app store on the greater developer community will not be the same. If customers are trained by Apple to use the app store to look for application purchases (it seems likely that Apple will make the app store a prominent part of the next Mac OS X release) then why would a consumer even bother looking outside of the app store? Heck, they might not even think it is possible to get software anywhere else.

This kind of situation is one in which developers need to find a way in which to keep their applications in the publics minds if they are not going to be deploying their applications to the app store (for whatever reason, Apples restrictions seem unnecessarily harsh). This will take some very creative marketing and is something I do not envy those who are stuck with legacy applications that they have no hope of ever making compliant with the requirements that Apple have laid.

The other important issue at stake here is one of price. The iOS app store has basically centred around 99 cent applications, anything more than that is considered a premium price. By their very nature Mac OS X applications require much more developer work on average than iOS applications and that should be reflected in the price. Unless Apple bring iAds to Mac OS X (something that I feel is very likely) most developers may well feel they are going to be forced to lower prices to below what they would expect to charge before the app store existed.

Even though I have laid out some problems which I foresee occurring once Apple release the app store for Mac OS X I still think in general it will be a positive boost for Mac OS X developers. It will give them easy access to every single modern Mac owner and that is a lot of people. Those who design a popular application have the chance to make an awful lot of money on the Mac OS X app store, assuming of course you can comply with Apples requirements.

I am aware of the Pinax project which offers lots of already written Django modules but I checked fairly recently and they were still behind the Django releases by quite a margin. Whilst the project looks good and I will certainly be keeping an eye on it, I don’t think I would make use of it in a serious project. Especially considering some of the changes in Django 1.2 are so good.

One element of Django that I overlooked when initially going through the documentation was the generic views feature. At first glance this seems like either a redundant feature or one that is better used as a small prototyping feature but after closer inspection it is obvious that this is an extremely powerful tool. The fact that you can delegate the entire view code to the main Django distribution not only simplifies your application but also reduces the number of potential bugs that your application contains.

Simply put any page that either displays a list of objects of a certain model type or a single object of a certain model type is ripe for use with the generic page view feature. Thus all you need to do is implement a template and that will handle how the view is displayed, all the correct data will automatically be passed to the template for you.

The next important tool to talk about is the built in comment framework. This simplifies the process of allowing users to post comment against articles and the like. The first time I used Django I actually wrote a simple blog application that implemented this feature itself but while it was a little more flexible the comment framework is stable, well documented and maintained by more developers. Any failings of the comment framework are going to be offset by the increased time spent fixing bugs in your own implementation and possibly a poor design in the first place.

The only complaint I have is the somewhat simplistic moderation system that is currently in place. It does the trick, but is not exactly a killer feature.

The final thing I want to talk about is the cache framework and memcached. I was impressed when I saw that the cache framework included not only a site wide setting but one which allowed caching on a per view basis. This could potentially allow you some interesting possibilities to optimise certain parts of your site say during a Slashdot stampede while the rest of the site remains untouched. My only concern would be that the control does not extend to the object level as far as I can see. It certainly allows you access to a low level cache API which allows the caching of data when you need it but it would be nicer to have something to automatically cache an object given a certain set of parameters.

Anyway this has been my first post reflecting my initial exploration of the framework. I’m keen to keep going as Django really seems the perfect mix of simplicity and flexibility. Stay tuned for more information in the coming weeks.

The first thing that I noticed was the conceptual simplicity of the Django framework. I have become accustomed to frameworks which are ridiculously complicated compared to the goal that they are actually trying to achieve. So complex in fact that instead of looking good, the authors just end up looking stupid for getting so caught up in themselves. Java and C# frameworks seem to be the worst offenders when it comes to this phenomenon.

Setting up views and mapping URLs to those views is also extremely easy and the code that then loads a template from within said view is not going to be more than a couple of line (unless you are passing a ridiculous number of variables to the template system). This enables you to setup some complex views with a bare minimum of code. Another winning feature in my mind is the ease of producing JSON output. Whilst this is a feature of the Python standard library, querying the database and then immediately parsing it into JSON format for consumption by clients or indeed your own Javascript code on your website enables you to create complex web services with the bare minimum of fuss.

One of the features I need most when it comes to web frameworks is an easy way develop the application. Some web development tools require some rather complex setup to test on your own development server but I have found Django to be amazingly simple on this front. One technique I use is to set it up to use an SQLite 3 database on the local machine and test there whilst in the process of actual development and then transfer everything to the development server, change the database settings to a PostgreSQL database server, sync the database and then test the code to make sure that everything works as expected. This has lead to a much faster and easier to manage development cycle for me as I do not have to have my development server running (for PostgreSQL and Apache) in order to test the website. Instead I simply rely on the integrated development server provided by the manage.py script.

The integration of Markdown within the Django framework also allows you to let users create rich text in comments, forum posts and messages to one another which will certainly be appreciated by them. The only slight annoyance I have with it is that you can not limit the allowed markdown syntax which means that users can post comments in a header one style which potentially ruins your SEO for a given page.

Django’s project management features are pretty good too. I tend to use BBEdit on Mac OS X for my web app development and using that applications project management along with the separation of different parts of the web application by Django ensures that you never get overwhelmed by a bunch of random files that you need to remember their purpose. This makes it easy to develop modular parts of your site (a comment system for instance) independently of the parts which will actually make use of the comment system.

Web services, as I have said earlier, are the bread and butter of Django. It is rare nowadays for any serious website to only have web browser clients. Most offer some form of integration with desktop and mobile clients.

Overall I’m highly impressed with both Django and Python as a language. As this is the first real project that I have used it in it has been an eye opener. The Python standard library complements Django perfectly and being able to use the wealth of third party Python libraries available makes just about any conceivable task that you may have easy.

So what are you waiting for? Try it out. If you have any questions leave a comment and I’ll get back to you ASAP.

Note:

I started writing this article using Django 1.1.1 but since then Django 1.2 has been released. I have not had time to test the new release properly but some of the new additions certainly look good and a couple of small issues I had have been fixed. So upgrade if you can.