How to properly activate TRIM for your SSD on Linux: fstrim, lvm and dm-crypt

Unlike hard disk drives (HDDs), NAND flash memory that make SSD cannot overwrite existing data. This means that you first have to delete the old data before writing new one. Flash memory is divided into blocks, which is further divided in pages. The minimum write unit is a page, but the smallest erase unit is a block.

Data can be written directly into an empty page, but only whole blocks can be erased. Therefore, to reclaim the space taken up by invalid data, all the valid data from one block must be first copied and written into the empty pages of a new block. Only then can the invalid data in the original block be erased, making it ready for new valid data to be written.

Do you see the problem? This means that as time goes on, the SSD will internally fragment the blocks among the different pages, until that it reaches a point where there won’t be available any empty page. Then every time the drive needs to write a block into any of the semi-full pages, it first needs to copy the current blocks from the page to a buffer, then it has to delete the whole page to finally rewrite the old blocks along with the new one. This means that as time goes on the SSD performance degrades more and more, because for every write it has to go through a cycle of read-erase-modify-write. This is known as “write amplification”.

Without TRIM the disk is unable to know which blocks are in use by a file or which ones are marked as free space. This is because when a file is deleted, the only thing the OS does is to mark the blocks that were used by the file as free inside the file system index. But the OS won’t tell the disk about this. This means that over time the performance of the SSD disk will degrade more and more, and it don’t matters how much space you free, because the SSD won’t know about it.

What is TRIM? TRIM was invented for solving this problem. TRIM is the name of a command that the operating system can send to tell the SSD which blocks are free in the filesystem.
The SSD uses this information to internally defragment the blocks and keep free pages available to be written quickly and efficiently.

How to active TRIM on Linux? The first thing to know is that TRIM should be enabled on all I/O abstraction layers. This means that if you have an ext4 partition on top of LVM, which in turn is on top of an encrypted volume with LUKS/dm-crypt, then you must enable support for TRIM in these three layers: The filesystem, LVM and dm-crypt. There is no point in enabling it at the filesystem level if you don’t enable it also on the other layers. The TRIM command should be translated from one layer to another until reaching the SSD.

  1. Enabling TRIM support on dm-crypt

  2. We simply have to add the option discard inside our crypttab

    • $ cat /etc/crypttab

      # <target name>    <source device>    <key file>       <options>
      sda2_crypt         /dev/sda2          none             luks,discard
      

    Note: The usage of TRIM on dm-crypt could cause some security issues like the revelation of which sectors of your disk are unused.

  3. Enabling TRIM support on LVM

  4. We have to enable the option issue_discards in the LVM configuration.

    • $ cat /etc/lvm/lvm.conf

      # [...]
      devices {
         # [...]
         issue_discards = 1
         # [...]
      }
      # [...]
      
  5. Enabling TRIM support on the file system

  6. This is the most interesting part. Most people simply add the option “discard” in the mounting options at /etc/fstab. However, this means that every time you delete a file, the OS will be reporting in real-time to the SSD which blocks were occupied by that file and are not longer in use, and then the SSD will have to perform a defragmentation and deletion of those internal blocks, operation which will take an amount of time higher than desired.

    In order to optimize the performance of the SSD, I strongly advise you to avoid doing the TRIM operation in real time (whenever a file is deleted) because you would be putting an unnecessary extra amount of work over the SSD. In other words: You should not enable the discard option in fstab.

    Instead, what I recommend is to run a script periodically to tell the SSD which blocks are free with the command fstrim. Doing this operation daily or weekly is more than enough. This way we do not lose any performance due to TRIM when deleting files and we periodically keep informed the SSD about the free blocks.

    Other advantages of the fstrim way are:

    • If you didn’t enabled correctly the TRIM support in the above layers of your setup, you will receive an error when executing fstrim. On the other hand, if you were using the discard option at fstab you wouldn’t have received any error and you would end thinking that you managed to get TRIM working properly when you didn’t.
    • If you delete a file by mistake (you know it happens), you can recover it before anacron runs your script fstrim. On the other hand, if you were using the discard-at-fstab option you wouldn’t have any chance of recovering the file, because the OS would have told the SSD to TRIM that blocks as soon as you deleted the file, and consequently the SSD has irreversibly destroyed such blocks.

    Here you have simple script to run fstrim on the /, /boot and /home partitions, which can be programmed to be executed periodically by anacron

    • $ cat /etc/cron.weekly/dofstrim

      #! /bin/sh
      for mount in / /boot /home; do
      	fstrim $mount
      done
      

    It’s worth mentioning that both fstrim and discard-at-fstab options only work on filesystems that implement the ioctl FITRIM. Today, such filesystems are:

    ~/kernel/linux (v3.8) $ grep -lr FITRIM fs/ | cut -d/ -f2 | sort | uniq | xargs echo
    btrfs ext3 ext4 gfs2 jfs ocfs2 xfs
    

Note: On most setups you will have to rebuild your initramfs with update-initramfs -u (Debian and derivatives) or dracut -f (Redhat and derivatives) and reboot the machine after touching the configuration options of LVM or dm-crypt.

Update 25-Feb: Seems that Fedora users with a dm-crypt volume will be affected by this problem: https://bugzilla.redhat.com/890533

Leave a comment ?

22 Comments.

  1. the default install with encryption sets /boot as ext2. and i guess that doesn’t support Trim. what will the ramifications be to leave /boot as ext2 and not be able to Trim?

    • /boot is usually a very small partition (<500MB usually). So I think is not a big deal to have this without trim support. If you are worried by this, you can change the default to use ext3 for /boot

  2. thank you! excellent blog you have!

  3. Thanks for your great information. :grin:
    I used to add discard in fstab before. When I switched to LVM, I needed to implement another way to set the Trim in my SSD. I have followed the blog information to setup the Trim, is there any method to check my setup is correct? Just like
    https://sites.google.com/site/lightrush/random-1/checkiftrimonext4isenabledandworking
    or
    http://howto.unixdev.net/Test_LVM_Trim_Ext4.html

  4. Excellent article thanks!

  5. Is this still true? (from http://askubuntu.com/a/191945/178920)

    However, these ‘discards’ will only trigger on remove or reduce operations on LVs. Discards on the filesystem on top of a LV will not propagate! From the same LVM changelog: “Issue discards on lvremove and lvreduce etc. if enabled and supported.”

    • Yes. Not sure about Ubuntu 12.04, but at least on Debian 6 (Squeeze) is true.

      The support for TRIM to LVM was added in the version 2.02.85 (see changelog)

      So, if your Linux distribution ships an older version of LVM you may get no support for TRIM.

      • But doesn’t that mean the “discard” option in fstab is useless? Or does fstrim work around LVM2?

        • If your filesystem is on top of a LVM volume, and your version of LVM is older than 2.02.85 you won’t get TRIM support. Neither with the discard option nor the fstrim one.

          The good thing about fstrim is that it will report the errors. However discard option in fstab will fail silently..

          So the best advice to check if your FS has TRIM support enabled on all the required layers is to run fstrim. If fstrim works, then you have TRIM enabled correctly, otherwise you will get an error.

  6. Not being much of a bash person a couple of things confuse me on this.

    My machine is set up with system files on SSD and /tmp /var and /home on hard drives to avoid writes to the ssd. This leaves me wondering what to fstrim as all of these come off /.

    The other aspect comes from reading about using ssd disks in redhat docs. They suggest trimming when the ssd has run out of “fresh space”. Makes sense as this would ensure wear levelling. There doesn’t seem to be a command available to check how much “fresh space” is left though. All not helped by the fact that there seems to be little info about on what exactly goes on inside these disks.

    Thanks for the info by the way. Best I have found.

    John
    -

  7. How can I fstrim swap partition?

  8. I have this bash snippet for the cron.weekly job that gets the mountpoints based on the mounted devices info instead of hard coding them. Maybe it is useful to somebody else:


    #!/bin/sh

    # this cronjob will discard unused blocks on the ssd mounted filesystems

    # get the locally mounted block devices - those starting with "/dev:
    # run df -k, pipe the result through grep and save the sixth field in
    # in the mountpoint array
    mountpoint=( $(df -k | grep ^/dev | awk '{print $6}') )

    # loop through the array and run fstrim on every mountpoint
    for i in "${mountpoint[@]}"
    do
    /usr/sbin/fstrim -v $i;
    done

  9. Fantastic write-up! Really well written and organized. One of the most useful howto’s I’ve seen. ever.

  10. Great blog! The only problem I encountered in my Ubuntu 12.04 where / is on lvm is the “fstrim /” command can still be run without any error raised even if I added ‘issue_discard = 0′ in lvm.conf, then ‘update-initramfs -u’ and reboot performed. This is not expected. Any comment? Thanks!

  11. It seems that openSUSE 13.1 is currently not prepared for proper handling of the “discard” option in /etc/crypttab. The initrd does not contain the proper command.

    Workaround: In order to activate discard on boot time, I just hard-coded the “cryptsetup” commands in /lib/mkinitrd/boot/71-luks.sh to contain “luksOpen –allow-discards”. Works fine so far.

Leave a Comment


*


NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Trackbacks and Pingbacks: