Note: I am talking about hashing opposed to encryption. Many people use the terms interchangeably, however they are very different operations and should not be confused.

• Hash functions convert variable sized data into a fixed sized result. They are deterministic and one-way – the same input always hashes to the same output, which cannot be reversed. Hashes have many more useful properties, but for storing passwords the important point is they cannot be reversed.
• Encryption is the process of transforming information to make it unreadable to anyone without the decryption key. The important difference is the ability to recover the original data given the correct decryption key.

In the situation where the database in compromised and users password hashes are exposed, it is a good bet that any decryption key we have stored could also be compromised. Since we have no need to know the users actual password (we can check for a valid login by hashing their input and comparing to the stored hash) we should never use encryption for storing passwords.

Hashing the password

At first glance password hashing seems like an incredibly simple problem – just run the password through a well known hash function and job done. Most advice these days seems to suggest the use of SHA-1 or SHA-2.

Hashing

SHA-1/2 are hash functions, designed by the NSA. Like all hash functions, they are one-way, so given a database of password hashes an attacker cannot reverse them. However passwords hashed using a simple hash function are still vulnerability to reversing using rainbow tables. Rainbow tables are mappings of resulting hash -> original password for dictionaries of common passwords, etc. To combat rainbow table attacks many people suggest the use of per-user salts.

Hashing + salt

A salt is a randomly generated string which is appended to a users password before hashing. This randomly generated string prevents the use of pre-generated rainbow tables as it increases the complexity and length of the final password, and hence the size of rainbow tables required are infeasibly large – the only solution is for an attacker to attempt a brute force attack using the correct salt.

Thanks to advances in parallel computing and graphics card technology, this is now a viable solution. Various tools such as IGHASHGPU and whitepixel now exist – with whitepixel boasting as many as 33.1 billion MD5 hashed password attempts per second using just 4 AMD Radeon 5970s.

Key stretching

Key stretching refers to techniques used to increase the time it takes to test a password – and hence decreasing the number of brute force password attempts that can be made per second. Using plain hashing algorithms we are talking one millionth of a second to generate a single hash on a regular CPU, or in the order of one billionth of a second using the above mentioned GPU software. If we can increase the time taken to generate a single attempt to 10s of milliseconds then we have effectively cut the brute force attempts by 4 or 5 orders of magnitude. In the situation where a legitimate user is attempting a login, an extra 10ms is in most cases unnoticeable.

BCrypt (Blowfish encryption)

I started this post by emphasising the difference between hashing and encryption, and stating that encryption should never be used for storing passwords – so you might be asking yourself why I am now talking about encryption?

BCrypt is an algorithm for irreversibly obscuring passwords using the blowfish encryption algorithm. It is important to understand that BCrypt does not simply encrypt passwords – the process is still a one-way process, just like hashing. BCrypt works by encryption a magic string, using a key derived from the provided password. The encrypted magic string is stored in the database, but the derived key is never stored.

The blowfish algorithm is important, as the amount of work done to encrypt a string, and hence time taken, is tuned using a cost parameter. By increasing this cost factor we can increase the time taken to test a password.

BCrypt is available as of PHP 5.3+, using the crypt function.

PBKDF2

PBKDF2 (Password-Based Key Derived Function) is a function to produce derived keys by repeatedly hashing the input many times. I’m not going to attempt to explain PBKDF2 in detail, however by tuning the number of times to repeatedly hash the input the cost can be increased, in the same way as for BCrypt. PBKF2 is used by some well known protocols/systems, such as WPA/WPA2, Grub 2, and LastPass.

Generating random data with PHP

Generating random data doesn’t sound like it should be challenging, and indeed it is not – however generating cryptographically strong random data is a bit harder.

The default random number source used by PHP is not considered to be cryptographically strong – however not to worry, there are various sources of cryptographically strong random data on most systems.

PHP 5.3+, OpenSSL extension

The OpenSSL extension, included by default in PHP 5.3+, provides a source of cryptographically strong random data.

/dev/urandom

For Linux/UNIX systems, cryptographically strong random data is produced from the device unlocked random, /dev/urandom. This can be read using the regular file access functions in PHP.

Microsoft CSP (Cryptographic Service Provider)

The Microsoft Cryptographic Service Provider API provides a source of cryptographically strong random data on Microsoft Windows systems. It can be accessed using the COM interface.

Solution used in FluxBB v2

For FluxBB v2 I have produced a library which makes use of BCrypt when available on the system (PHP 5.3+), and falls back to PBKDF2 otherwise. Cryptographically strong random data is obtained using the OpenSSL extension if available, will fall back to /dev/urandom, or in the worse case PHPs built in non-cryptographically-strong functions.

In 2008 I undertook a project entitled “Creation of a MORPG framework using Apache MINA and jMonkeyEngine” as part of my final year at university. The aim of this project was to effectively take what I had learned and turn it into a legitimate project, and investigate the viability of the game engine jMonkeyEngine for our game client. While this project worked to an extent, and in fact won me the Best Student Project 2008/9, I decided that jMonkeyEngine was providing much more than I needed or wanted. The original plan had been to continue work on the project after university, however after my dissertation was completed the project was dropped.

This year I decided to revisit the project, and started work on a new game client – this time avoiding any existing game engines, and making direct use of OpenGL (though JOGL). In the future I may try and write some blog posts with technical details or plans, however for now I just wanted to post a few screen-shots to prove it is at least vaguely working…

1. Format a USB stick using FAT (or possibly FAT32). It’s worth noting that the EZ Flash utility seems to dislike certain USB sticks, rejecting my 4GB iamaKey though accepting my 1GB so-old-that-the-brand-has-warn-off drive.
2. Unzip the new BIOS.
3. Rename the file to 1201N.ROM (case sensitive!).
4. Move it to the root of the USB stick.
5. Put the USB stick in the 1201N (if you prepared it elsewhere), and make sure all other USB devices are removed.
6. Reboot/Turn-on the 1201N.
7. Enter the BIOS (F2) and note any settings, then reset the BIOS to default settings. This is especially important if you have over-clocked or anything similar. Save and Exit (F10).
8. When the 1201N reboots, press and hold ALT+F2 to enter the EZ Flash utility.
9. The EZ Flash utility should automatically perform the update then reboot. Note: This took a long time for me (10mins+), it may be because I was using such an ancient and slow USB stick, or may just be an incredibly slow process.
10. Enter the BIOS again (F2) and put back any custom settings you had. Save and Exit (F10).

In PunBB 1.3 (which we forked into FluxBB 1.3 Legacy), Rickard introduced the use of hooks and eval, to allow extensions to insert code and extend the core software. Originally this system was designed to allow small changes, but by the time the software was finished it had mutated into a full blown extension system.

Along with the disadvantages inherent from a hook system (see below), FluxBB 1.3 Legacy used a rather horrible system of PHP code within XML code, which was parsed and inserted into the database on install. The aim was that then the install files could be deleted or overwritten and the extension wouldn’t be affected until it was uninstalled/updated. However due to the fact that most extensions referenced external files (be it included php files, or css and images) this theory did not work very well.

So, going back to the beginning, what are the different solutions available for writing extensible PHP software?

Hooks

Hooks are probably the most common way used to make PHP software extensible. The idea is fairly simple; throughout the code there are “hooks”, where extensions can insert code. Below is an example hook taken from FluxBB 1.3 Legacy:

if (isset($_POST['form_sent']))
{
	($hook = get_hook('po_form_submitted')) ? eval($hook) : null;
 
	// Make sure form_user is correct

In the above example the get_hook method will look for all the code which has been registered with the hook “po_form_submitted”, combine it and return it. The code will then be executed using the eval function.

The disadvantages of the hook system should be fairly obvious…

• Blocks of code can be inserted, but only at pre defined locations within the software. This means extra thought has to be given during development as to where hooks may be required. If someone writing an extension needs a hook in a place where there isn’t one, then they are out of luck.
• Insertion of code is all very well, but what if we want to change or remove some existing code? There is no way to do this using hooks. This leads to extensions with rather “interesting” algorithms to attempt to undo or block code in the core, which can be both messy and inefficient.

Despite these rather major sounding disadvantages, I would argue hooks are good! At least much better than requiring users to change code by hand…

In the example code given above the eval function was used to execute the inserted code, but where is the code actually stored, and is this the best way to execute it? There are a few options:

• Store the code in a database or XML etc. then load and eval it when required. This probably isn’t the best solution as it involves storing code in the database or other storage that isn’t designed for code. Caching would probably also be required, loading and parsing a large XML file on every page load would not be efficient.
• Keep the code for each hook in an individual PHP file and include it when required. This option seems reasonable, but could result in having a huge amount of tiny files being included, which could be hard to organize and may not be ideal if the server isn’t running any form of opcode cache.
• Keep the code all in one PHP file and register specific functions with each hook, then call these when required. This seems like the best solution, but introducing functions introduces problems of variable scope. The underlying code would need to be written carefully in a way that appropriate variables can be passed to each hook.

Patches

A totally different approach to extending PHP software is patching it. In a similar way to how a patch file is used, the software could read instructions from an extension and physically modify the appropriate code.

@@ -58,7 +58,9 @@
 // Did someone just hit "Submit" or "Preview"?
 if (isset($_POST['form_sent']))
 {
-	// Make sure form_user is correct
+	// This is some new code
+	$username = isset($_POST['username']) ? $_POST['username'] : null;
+

This means that once the extension has been installed, there is no extension code left to worry about; the physical files have been changed and the extension installation files can be removed.

At first this may seem like a much better solution than hooks; code can be inserted/changed/deleted anywhere in any files, there is no need to litter hooks all over the place. However it has three major disadvantages:

• Often in shared hosting environments PHP will be configured to run as a certain unix user (usually “www-data” or “nobody”). For the software to be able to patch itself, all files would require to be writeable by that user. This could prove to be a major security flaw as it would allow other users on the server to modify your files. A work around for this could be to modify the files over (S)FTP instead, but then the user would need to provide their hosting login details.
• Updating the core software becomes harder. Since the files have been physically modified it is no longer a simple task to replace them with updated files. In theory core updates could be applied the same way extensions are installed (i.e. as patches), but with a modified board conflicts are fairly likely, and code that been removed by extensions is likely to just be replaced when the core is updated.
• An extension with a bug, or even a malicious extension, could render the whole software unusable by deleting sections of code or inserting invalid code. Since the actual files would have physically been overwritten there would be no way to recover from this, other than the user manually uploading a backup (if they were organized enough to have one!).

To me this seems like a rather show stopping problem, what’s the point in a one-click install system if uninstallation can require manually removing the changes. The idea of a buggy extension being able to totally destroy a working install is not acceptable!

Conclusions

What will be use in FluxBB 2.0? Only time will tell, but I’m currently leaning towards making use of hooks again, but attaching functions loaded from native PHP scripts, rather than chunks of code to be evaled.

Have any suggestions or better ideas? Please leave a comment!

Note: RAID-5 requires a minimum of 3 drives, and all should be the same size. It provides the ability for one drive to fail without any data loss.

Usable space = (no. of drives - 1) * size of smallest drive

In my set up I started with 3x 1.5TB drives, giving 3.0TB usable space. I have now grown it to 4x 1.5TB drives, giving 4.5TB usable space.

Required software

To create, format and resize the array we need to use various filesystem manipulation tools, but they should all come with Ubuntu. The only software you should need to install is mdadm.

sudo aptitude install mdadm

Initial setup

1. Preparing the drives

Before we jump into creating the actual RAID array, we first need to prepare partitions for the array to use. This can be done using a GUI tool such as GParted (available in the Ubuntu repositories), but I prefer using the terminal.

To get a list of available drives/partitions:

sudo fdisk -l

This will output, for each drive you have, something along the lines of:

Disk /dev/sda: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000f1f05

There may well be slightly more information available too if you already have some partitions made. From here note the name of the drives you wish to use (for example, /dev/sda). For each drive, create a partition and mark it as a RAID partition.

sudo fdisk /dev/sda

This will open up fdisk, the partition manager. If you already have any partitions on the drive you should first delete them (obviously this will erase any data on them!). To create one partition that is the size of the whole drive:

Command (m for help): n
Command action
	e	extended
	p	primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-182401, default 1): [blank]
Last cylinder or +size or +sizeM or +sizeK (1-182401, default 182401): [blank]

When choosing the first and last cylinder simply hitting return will use the default values (which are probably the values you actually want).

Next we will mark the partition as being part of a RAID array, allowing mdadm to automatically detect it:

Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid auto)

So far our changes haven’t actually been written to disk, so finally, issue the command to write the changes to disk.

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Remember to repeat this step for each drive you wish to use in the array.

2. Creating the array

To create the array, we use the mdadm create flag (who’d have guessed that!). We also need to specify what RAID level we want, as well as how many devices and what they are.

sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sda1 /dev/sdb1 /dev/sdc1

The verbose flag tells it to output extra information. In the above command I am creating a RAID-5 array at /dev/md0, using 3 partitions. If you already have a RAID array set up then you may need to use /dev/md1 for example. The number of partitions you are using, and their names will probably be different, so do not just copy and paste all of the command above. Note that the partition name is something like /dev/sda1, whereas the drive name is something like /dev/sda; the 1 refers to the partition number.

While the array is being built you can view its status in the file /proc/mdstat. Here the watch command comes in handy:

sudo watch cat /proc/mdstat

This will output the contents of the file to the screen, refreshing every 2 seconds (by default). While the array is being built it will show how much of the “recovery” has been done, and an estimated time remaining.

Now that we have set up the array, we need to edit the mdadm configuration so it knows how to reassemble it when the system boots.

sudo nano /etc/mdadm/mdadm.conf

There should already be some sample configuration here:

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md0 level=raid5 num-devices=3 UUID=31283299:0c118170:8b4eb9a8:4e935bf2

The ARRAY line probably doesn’t already exist. It is used to describe an array (such as the one we just created). While similar, your array will not be the same as mine above so do not just copy and paste it! Luckily, mdadm provides a utility to fetch this line exactly (by looking for partitions marked as belonging to a RAID array, as we did when preparing the drives):

sudo mdadm --detail --scan

If you add the –verbose flag here it will also output the devices which are part of the array. Because we marked the partitions as being part of a RAID array mdadm can find them automatically so this isn’t required in our case, but it may be if you have more than one RAID array in your machine.

Now, I decided to wait for the array to finish syncing before moving on, but you should be able to continue and format the array while it syncs if you wish. In my case, building the array with 3x 1.5TB drives took around 6 hours.

3. Creating and mounting the filesystem

Now that the array is built we need to format it. What filesystem you choose is up to you, but I would probably recommend ext3. Note: For my array I chose to use ext4. So far I have not had any problems, but ext4 is still fairly new and there are still various bug reports going around mentioning data loss [bugzilla.kernel.org] (which I conveniently didn’t notice until after!).

sudo mkfs.ext3 /dev/md0

This will take a while, especially if your array is large.

If you chose to use ext2/3/4 you should also be aware of reserved space. By default ext2/3/4 will reserve 5% of the drives space, which only root is able to write to. This is done so a user cannot fill the drive and prevent critical daemons writing to it, but 5% of a large RAID array which isn’t going to be written to by critical daemons anyway, is a lot of wasted space. I chose to set the reserved space to 0%, using tune2fs:

sudo tune2fs -m 0 /dev/md0

Next we should add the array to the fstab, so that it will automatically be mounted when the system boots up. This can be done by editing the file /etc/fstab. For more detailed information, see the fstab Wikipedia article.

sudo nano /etc/fstab

Your fstab should already contain a few entries (if it doesn’t something is wrong!). At the bottom add a line similar to the following:

/dev/md0	/mnt/raid	ext4	defaults	0	0

I chose to mount my array on /mnt/raid, but you may well wish to mount it somewhere else. If the folder you chose doesn’t exist you will need to create it. As I said earlier I chose to use ext4, but here you will need to enter whatever filesystem you chose earlier.

Now, mount the array.

sudo mount -a

This will mount anything mentioned in the fstab that isn’t currently mounted but should be (hopefully your array!).

You should now have a working RAID-5 array! Next I would strongly suggest you investigate how to look after and monitor it, being able to cope with one drive failure is no use if you don’t notice when it fails and let a second fail!

Growing

One of the advantages of software RAID is the flexibility it gives you, that would normally only be available from high end (expensive) RAID cards. This includes the ability to grow an existing array (only for certain RAID levels), which means if you run out of space you can easily plug in a new drive and keep going.

1. Growing the array

Growing a RAID-5 array with mdadm is a fairly simple (though slow) task. First you will need to prepare the new drive in the same we we prepared the initial drives (step 1, above). To start the actual growing of the array we then add the new drive to the array as a spare:

sudo mdadm --add /dev/md0 /dev/sdd1

Then we grow the array onto this device. Because I had 3 drives before, the new drive obviously makes 4. Make sure to change this to whatever number you now have!

sudo mdadm --grow /dev/md0 --raid-devices=4

As when creating the array, we can follow the progress by checking the file /proc/mdstat.

sudo watch cat /proc/mdstat

Again, I decided to wait for the operation to finish before attempting to grow the filesystem, but you should be able to do it while the array syncs.

2. Growing the filesystem

Before we can grow the filesystem we need to unmount it:

sudo umount /dev/md0

Now that it is unmounted we can run a filesystem check to make sure everything is in order:

sudo fsck.ext3 -f /dev/md0

If any issues arise, let it attempt to fix them for you.

Once the filesystem has been checked we can perform the resize. If you decided to use something other than ext2/3/4 this may be done differently, but for the ext2/3/4 filesystems we use the resize2fs tool.

sudo resize2fs /dev/md0

This will automatically grow the filesystem to the new size of the device.

You can now remount the drive and start filling it up again!

Sources

• Software RAID 5 in Ubuntu with mdadm
• Growing a RAID5 array – MDADM
• Linux Raid Wiki
• Recovering reserved space in ext2 and ext3 filesystems

If you spot any errors, or have any problems then please leave a comment and let me know!

I decided to create a software RAID-5 array using the 4 drives, to give a total capacity of 4.5TB with the ability to handle 1 drive failure.

What is RAID?

RAID stands for Redundant Array of Inexpensive Disks, and is a way of organizing data over multiple drives to provide higher performance and/or reliability than available from a single drive.

What is RAID not?

Although RAID is redundant and can recover from drive failure, it alone is not a backup solution! If you delete a file on a RAID array it is deleted from all drives in that array, similarly if your data is corrupted then the corrupted data will be mirrored across the whole array.

Hardware RAID vs Fake RAID vs Software RAID

Before getting started you should think about how best to implement your RAID array, because once it’s done it can be hard (if not impossible without enough spare disks to back up all your data) to change.

Hardware RAID

Hardware RAID is usually considered the “real” implementation of RAID. It requires a dedicated RAID card, which handles maintaining the array; this has the advantage that the CPU is not required so performance is usually better. The disadvantages however are that you are then restricted to the RAID card you have; the implementations of RAID isn’t always compatible between different models, and almost definitely isn’t between different vendors, and the cost; hardware RAID is not cheap!

Fake RAID

Most motherboards these days come with RAID support, which is known as Fake RAID. What’s fake about it? There is no dedicated chip to maintain the array as with “real” RAID. This means that the processing is handed off to the CPU, so the performance benefit is no longer there. On the upside, it is cheap!

Software RAID

The third option when implementing RAID is Software RAID, which, as the name suggests, is done totally in software. This means, like with Fake RAID, the CPU is required to do the processing. The advantages however are it’s portability, and cost (free!). Because the implementation is entirely software based, there is no restriction on hardware any more; you can move the drives between computers without any problems. There is however now a restriction on software; if you created the RAID array under Linux (using mdadm), it will not be accessible under Windows (or any other OS that doesn’t have the same implementation of mdadm available). It is also worth mentioning that mdadm includes many features which can only be found on the high end (expensive) RAID cards, such as the ability to grow or shrink a live array.

Levels of RAID

Another thing you will need to decide on before starting is what level of RAID you require. For a detailed review of the different levels see the Wikipedia article about Standard RAID levels. I am using RAID-5 because it provides some redundancy but without sacrificing too much capacity (usable size = (num of drives – 1) * size of smallest drive), however there is some speculation that as disk sizes are increasing, RAID-5 is becoming less reliable. When I next get paid I may consider upgrading to RAID-6 (which, using software RAID, is easy!).

Follow up: Software RAID-5: using mdadm in Ubuntu 9.10.

Abstract – We present an approach to support massively multi-player games on peer-to-peer overlays. Our approach exploits the fact that players in MMGs display locality of interest, and therefore can form self-organizing groups based on their locations in the virtual world. To this end, we have designed scalable mechanisms to distribute the game state to the participating players and to maintain consistency in the face of node failures. The resulting system dynamically scales with the number of online players. It is more flexible and has a lower deployment cost than centralized games servers. We have implemented a simple game we call SimMud, and experimented with up to 4000 players to demonstrate the applicability of this approach.

We have also recently announced our plans to unveil a brand new website, on a new server which I have ordered from Gandi (where I also happen to rent the VPS for this website from). On the new server I have decided to give lighttpd a try as an alternative to Apache2, I guess time will tell if this was a good idea or not.

The new website, along with looking much cleaner and professional, has much more information and many more useful features than the old (and slightly out-of-date) one.

• Some important information that was missing from the old site, such as a a full feature list and information about; FluxBB’s history, future development and the development team.
• Localization support – Every page of the new site can now be easily translated into various different languages. Once the text has been finalized we hope some members of the international communities will provide us with some translations.
• Information about upgrading between releases, and converting from other software; including a list of other software which conversion from is supported.
• A page of recommended web hosts to get people started.
• A modification repository to host modifications, plugins, styles and language packs. This will allow mod developers to host their modifications in a central location where they can easily be found. It also gives us some control over which modifications are published, allowing us to reject any with obvious security vulnerabilities for example, which will hopefully improve the quality of the modifications.
• An updated IRC web chat, using the webchat.freenode.net script.
• A brand new user oriented wiki, providing guides on installing, setting up, using, moderating and administering FluxBB.

For anyone interested, a sneak preview of the new site:

For more information, Changes, updates and on-going progress…

If you find any problems with the skin or anything please let me know…