Using Flashcache to speed up your I/O on Ubuntu 12.04

In this tutorial you will learn how to implement Flashcache on your Linux box.

Flashcache is a write back block cache Linux kernel module, designed to speed up your I/O to your slow spinning disk. Flashcache has been developed and released by Facebook which uses the module to speed up their database servers. The module implements itself as a target for the Linux Device Mapper (The Device Mapper is also used by LVM, MD-Raid and more..). The Device Mapper combines 2 block devices (the cache device (SSD) and the backend device (Slow spinning disk)) into one device which uses the flashcache modules functions. One of the biggest upside of Flashcache is the fact that it is block-based, which means that it is filesystem independant!

This tutorial is based on the following setup;
OS: Ubuntu 12.04 x86_64
Kernel: 3.2.0-24-generic
Disksetup:

  • /dev/sda (slow spinning disk)
  • sda1 - /boot (ext2)
  • sda2 - / (ext4)
  • sda3 - swap
  • sda4 - /home (btrfs)
  • /dev/sdb (ssd)
  • (unformatted)

In the tutorial my goal is to use the entire sdb device to cache my /home on sda4.

Step 1: Cloning the Git repo

The first step is getting the flashcache source to compile the module. Facebook has published the source of this module on github which can be reached on https://github.com/facebook/flashcache .

Cloning this Git repo to my Linux is easy with the git tools:

(optional) # apt-get install git-core
# git clone https://github.com/facebook/flashcache.git
root@gerrit-HP-ProBook-6560b:/home/gerrit/test# git clone https://github.com/facebook/flashcache.git
Cloning into 'flashcache'...
remote: Counting objects: 879, done.
remote: Compressing objects: 100% (332/332), done.
remote: Total 879 (delta 608), reused 807 (delta 544)
Receiving objects: 100% (879/879), 244.56 KiB | 194 KiB/s, done.
Resolving deltas: 100% (608/608), done.

This will create a new directory called 'flashcache' in your current directory.

Step 2: Build the module and install it.

Now that we have the source code we need the next step is compiling it. Compiling the module is very easy.

In the 'flashcache' directory that has just been created you need to use the 'make' command to compile the source
The make command compiles both the module and all the tools needed to use flashcache.

HINT: The command 'make' without arguments will compile the module based on your current running kernel. If you want to build the module based on another installed kernel you can use the KERNEL_TREE parameter. For example, if I want to use my custom-kernel (called 'custom-kernel') as a base for the build I can run the following command:

(optional) # make KERNEL_TREE=/usr/src/custom-kernel
# make
root@gerrit-HP-ProBook-6560b:/home/gerrit/test/flashcache# make
make -C src KERNEL_TREE=/lib/modules/3.2.0-24-generic/build PWD=/home/gerrit/test/flashcache/src all
make[1]: Entering directory `/home/gerrit/test/flashcache/src'
make -C /lib/modules/3.2.0-24-generic/build M=/home/gerrit/test/flashcache/src modules V=0
make[2]: Entering directory `/usr/src/linux-headers-3.2.0-24-generic'
CC [M] /home/gerrit/test/flashcache/src/flashcache_conf.o
CC [M] /home/gerrit/test/flashcache/src/flashcache_main.o
CC [M] /home/gerrit/test/flashcache/src/flashcache_subr.o
CC [M] /home/gerrit/test/flashcache/src/flashcache_ioctl.o
CC [M] /home/gerrit/test/flashcache/src/flashcache_procfs.o
LD [M] /home/gerrit/test/flashcache/src/flashcache.o
Building modules, stage 2.
MODPOST 1 modules
CC /home/gerrit/test/flashcache/src/flashcache.mod.o
LD [M] /home/gerrit/test/flashcache/src/flashcache.ko
make[2]: Leaving directory `/usr/src/linux-headers-3.2.0-24-generic'
make -C utils all
make[2]: Entering directory `/home/gerrit/test/flashcache/src/utils'
cc -c -I.. -I. -DCOMMIT_REV=""1.0-150-g21c99dea2ee4"" -g flashcache_create.c -o flashcache_create.o
flashcache_create.c: In function ‘main’:
flashcache_create.c:314:11: warning: format ‘%d’ expects argument of type ‘int’, but argument 6 has type ‘sector_t’ [-Wformat]
flashcache_create.c:349:3: warning: format ‘%lu’ expects argument of type ‘long unsigned int’, but argument 10 has type ‘int’ [-Wformat]
cc flashcache_create.o -o flashcache_create
cc -c -I.. -I. -DCOMMIT_REV=""1.0-150-g21c99dea2ee4"" -g flashcache_destroy.c -o flashcache_destroy.o
cc flashcache_destroy.o -o flashcache_destroy
cc -c -I.. -I. -DCOMMIT_REV=""1.0-150-g21c99dea2ee4"" -g flashcache_load.c -o flashcache_load.o
cc flashcache_load.o -o flashcache_load
cc -c -I.. -I. -DCOMMIT_REV=""1.0-150-g21c99dea2ee4"" -g flashcache_setioctl.c -o flashcache_setioctl.o
cc flashcache_setioctl.o -o flashcache_setioctl
make[2]: Leaving directory `/home/gerrit/test/flashcache/src/utils'
make[1]: Leaving directory `/home/gerrit/test/flashcache/src'

If you experience any build errors you might be missing some kernel library's which you need to install. On Ubuntu you might need to install uuid-dev (apt-get install uuid-dev)

After the build is done you need to install the module, tools and man pages. This can be done by issuing the simple command:

# make install
root@gerrit-HP-ProBook-6560b:/home/gerrit/test/flashcache# make install
make -C src KERNEL_TREE=/lib/modules/3.2.0-24-generic/build PWD=/home/gerrit/test/flashcache/src install
make[1]: Entering directory `/home/gerrit/test/flashcache/src'
make -C /lib/modules/3.2.0-24-generic/build M=/home/gerrit/test/flashcache/src modules V=0
make[2]: Entering directory `/usr/src/linux-headers-3.2.0-24-generic'
Building modules, stage 2.
MODPOST 1 modules
make[2]: Leaving directory `/usr/src/linux-headers-3.2.0-24-generic'
install -o root -g root -m 0755 -d /lib/modules/3.2.0-24-generic/extra/flashcache/
install -o root -g root -m 0755 flashcache.ko /lib/modules/3.2.0-24-generic/extra/flashcache/
depmod -a
make -C utils all
make[2]: Entering directory `/home/gerrit/test/flashcache/src/utils'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/home/gerrit/test/flashcache/src/utils'
make -C utils install
make[2]: Entering directory `/home/gerrit/test/flashcache/src/utils'
install -d -m 755 /sbin/
install -m 755 flashcache_create flashcache_destroy flashcache_load flashcache_setioctl /sbin/
make[2]: Leaving directory `/home/gerrit/test/flashcache/src/utils'
make -C ocf install
make[2]: Entering directory `/home/gerrit/test/flashcache/src/ocf'
install -d -m 755 /usr/lib/ocf/resource.d/flashcache
install -m 755 flashcache /usr/lib/ocf/resource.d/flashcache
make[2]: Leaving directory `/home/gerrit/test/flashcache/src/ocf'
make[1]: Leaving directory `/home/gerrit/test/flashcache/src'

The installation has now been completed, on the next couple of steps we will load the module and begin our migration.

Step 3: Initializing the module

The flashcache module has been installed and needs to be initialized. This can be done by using the modprobe command.

# modprobe flashcache

By checking out your kernel messages you can see that it has been initialized.

# dmesg | tail
.............................
[ 5806.891504] flashcache: flashcache-1.0 initialized

Step 4: Creating the flashcache device

Now that the module has been loaded we can initialize our target. With the 'flashcache_create' tool we will combine the 2 block devices in the Device Mapper. The flashcache_create utility will write a superblock to the caching device (SSD) with the information it needs, and will issue the 'dmsetup' command to initialize the Device Mapper. Don't worry, nothing bad will happen to your backend device, nothing will be written or overwritten.

In my case I am migrating an existing /home mount to the flashcache device, unfortunately this cannot be done live. That's why in my case I need to unmount /home in order to create the device. Close all applications running and unmount:

# umount /home

After the unmount has finished I can create the flashcache device, this is done with the flashcache_create utility which works in the following way;

flashcache_create [-v] [-p back|thru|around] [-b block size] [-m md block size] [-s cache size] [-a associativity] device_name ssd_devname disk_devname

As you can see there are many options to set up your flashcache device. In my case the defaults are fine (mode=(write)back,blocksize=4k,associativity=512). If you are unsure about what options to use I suggest you check out the flashcache documentation by clicking on the link on the bottom of this page. Every option is well documented!

Creating the flashcache_device:

# flashcache_create home_cached /dev/sdb /dev/disk/by-uuid/1cf0f643-129c-492e-b86c-13b422ac6e4b

HINT: I am not using the /dev/sda4 device pointer, but the symlink based on the UUID. This is to make sure that even if the order of devices are changed that the flashcache device can be build by using the UUID.

This command creates a device on /dev/mapper/home_cached. This is the flashcache device we can mount to work on.

Step 5: Mounting and using your flashcache_device

Now that the device is active we can mount and use it. I am using the default mount options because they are sufficient enough:

# mount /dev/mapper/home_cached /home

Congratulations! Now you are using flashcache on your home directory. Have fun testing!

The flashcache module comes with statistics which can be viewed by issuing the dmsetup command:

# dmsetup status

(Optional) Step 6: Making flashcache bootable

In my case I want my flashcache device to be bootable. This is to make sure that the flashcache device can be mounted on /home on boot. Luckily the developers of Flashcache have been smart enough to implement this feature.
To make the devices bootable the flashcache has to be implemented in the initramfs, and a corresponding script to build the devices pre-mount.
In the source of flashcache (the git clone you just made) there is a Makefile.dkms. This makefile integrates some targets based on DKMS (Dynamic Kernel Module Support). Issuing the following command installs the scripts that are needed:

# make -f Makefile.dkms boot_conf
root@gerrit-HP-ProBook-6560b:/home/gerrit/test/flashcache# make -f Makefile.dkms boot_conf
make -C src/utils install
make[1]: Entering directory `/home/gerrit/test/flashcache/src/utils'
install -d -m 755 /sbin/
install -m 755 flashcache_create flashcache_destroy flashcache_load flashcache_setioctl /sbin/
make[1]: Leaving directory `/home/gerrit/test/flashcache/src/utils'
# do mkinitramfs install if present
if test -d /etc/initramfs-tools/hooks; then 
install -o root -g root src/flashcache.hook /etc/initramfs-tools/hooks/flashcache ; 
install -o root -g root src/utils/flashcache_scan /etc/initramfs-tools/scripts/init-premount/ ; 
update-initramfs -k 3.2.0-24-generic -u ; 
fi
update-initramfs: Generating /boot/initrd.img-3.2.0-24-generic

The make target installed the scripts in /etc/initramfs-tools and updated your initramfs, you can check them out if you are wondering what exactly they do!

Don't forget to update your /etc/fstab to point to the mapped flashcache device!

Links:
GIT Repo: https://github.com/facebook/flashcache/
Documentation: https://github.com/facebook/flashcache/blob/master/doc/flashcache-doc.txt
Sysadmin Docs: https://github.com/facebook/flashcache/blob/master/doc/flashcache-sa-guide.txt
Others: https://wiki.archlinux.org/index.php/Flashcache

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>