Unlike hard disk drives (HDDs), NAND flash memory that make SSD cannot overwrite existing data. This means that you first have to delete the old data before writing new one. Flash memory is divided into blocks, which is further divided in pages. The minimum write unit is a page, but the smallest erase unit is a block.
Data can be written directly into an empty page, but only whole blocks can be erased. Therefore, to reclaim the space taken up by invalid data, all the valid data from one block must be first copied and written into the empty pages of a new block. Only then can the invalid data in the original block be erased, making it ready for new valid data to be written.
Do you see the problem? This means that as time goes on, the SSD will internally fragment the blocks among the different pages, until that it reaches a point where there won’t be available any empty page. Then every time the drive needs to write a block into any of the semi-full pages, it first needs to copy the current blocks from the page to a buffer, then it has to delete the whole page to finally rewrite the old blocks along with the new one. This means that as time goes on the SSD performance degrades more and more, because for every write it has to go through a cycle of read-erase-modify-write. This is known as “write amplification”.
Without TRIM the disk is unable to know which blocks are in use by a file or which ones are marked as free space. This is because when a file is deleted, the only thing the OS does is to mark the blocks that were used by the file as free inside the file system index. But the OS won’t tell the disk about this. This means that over time the performance of the SSD disk will degrade more and more, and it don’t matters how much space you free, because the SSD won’t know about it.
What is TRIM? TRIM was invented for solving this problem. TRIM is the name of a command that the operating system can send to tell the SSD which blocks are free in the filesystem.
The SSD uses this information to internally defragment the blocks and keep free pages available to be written quickly and efficiently.
How to active TRIM on Linux? The first thing to know is that TRIM should be enabled on all I/O abstraction layers. This means that if you have an ext4 partition on top of LVM, which in turn is on top of an encrypted volume with LUKS/dm-crypt, then you must enable support for TRIM in these three layers: The filesystem, LVM and dm-crypt. There is no point in enabling it at the filesystem level if you don’t enable it also on the other layers. The TRIM command should be translated from one layer to another until reaching the SSD.
-
Enabling TRIM support on dm-crypt
-
$ cat /etc/crypttab# <target name> <source device> <key file> <options> sda2_crypt /dev/sda2 none luks,
discard -
Enabling TRIM support on LVM
-
$ cat /etc/lvm/lvm.conf# [...] devices { # [...]issue_discards = 1# [...] } # [...] -
Enabling TRIM support on the file system
- If you didn’t enabled correctly the TRIM support in the above layers of your setup, you will receive an error when executing fstrim. On the other hand, if you were using the discard option at fstab you wouldn’t have received any error and you would end thinking that you managed to get TRIM working properly when you didn’t.
- If you delete a file by mistake (you know it happens), you can recover it before anacron runs your script fstrim. On the other hand, if you were using the discard-at-fstab option you wouldn’t have any chance of recovering the file, because the OS would have told the SSD to TRIM that blocks as soon as you deleted the file, and consequently the SSD has irreversibly destroyed such blocks.
-
$ cat /etc/cron.weekly/dofstrim#! /bin/sh for mount in / /boot /home; do fstrim $mount done
We simply have to add the option discard inside our crypttab
Note: The usage of TRIM on dm-crypt could cause some security issues like the revelation of which sectors of your disk are unused.
We have to enable the option issue_discards in the LVM configuration.
This is the most interesting part. Most people simply add the option “discard” in the mounting options at /etc/fstab. However, this means that every time you delete a file, the OS will be reporting in real-time to the SSD which blocks were occupied by that file and are not longer in use, and then the SSD will have to perform a defragmentation and deletion of those internal blocks, operation which will take an amount of time higher than desired.
In order to optimize the performance of the SSD, I strongly advise you to avoid doing the TRIM operation in real time (whenever a file is deleted) because you would be putting an unnecessary extra amount of work over the SSD. In other words: You should not enable the discard option in fstab.
Instead, what I recommend is to run a script periodically to tell the SSD which blocks are free with the command fstrim. Doing this operation daily or weekly is more than enough. This way we do not lose any performance due to TRIM when deleting files and we periodically keep informed the SSD about the free blocks.
Other advantages of the fstrim way are:
Here you have simple script to run fstrim on the /, /boot and /home partitions, which can be programmed to be executed periodically by anacron
It’s worth mentioning that both fstrim and discard-at-fstab options only work on filesystems that implement the ioctl FITRIM. Today, such filesystems are:
~/kernel/linux (v3.8) $ grep -lr FITRIM fs/ | cut -d/ -f2 | sort | uniq | xargs echo
btrfs ext3 ext4 gfs2 jfs ocfs2 xfs
Note: On most setups you will have to rebuild your initramfs with update-initramfs -u (Debian and derivatives) or dracut -f (Redhat and derivatives) and reboot the machine after touching the configuration options of LVM or dm-crypt.
Update 25-Feb-2013: Seems that Fedora users with a dm-crypt volume will be affected by this problem: https://bugzilla.redhat.com/890533
Update 14-Aug-2014: The following script can be used to automatically detect and fstrim all filesystems that have TRIM support enabled.
-
#!/bin/sh # # To find which FS support trim, we check that DISC-MAX (discard max bytes) # is great than zero. Check discard_max_bytes documentation at # https://www.kernel.org/doc/Documentation/block/queue-sysfs.txt # for fs in $(lsblk -o MOUNTPOINT,DISC-MAX,FSTYPE | grep -E '^/.* [1-9]+.* ' | awk '{print $1}'); do fstrim "$fs" done
the default install with encryption sets /boot as ext2. and i guess that doesn’t support Trim. what will the ramifications be to leave /boot as ext2 and not be able to Trim?
/boot is usually a very small partition (<500MB usually). So I think is not a big deal to have this without trim support. If you are worried by this, you can change the default to use ext3 for /boot
What about the swap partition? Should I run fstrim on it too?
How often do you delete items from /boot?
thank you! excellent blog you have!
Thanks for your great information. 😀
I used to add discard in fstab before. When I switched to LVM, I needed to implement another way to set the Trim in my SSD. I have followed the blog information to setup the Trim, is there any method to check my setup is correct? Just like
https://sites.google.com/site/lightrush/random-1/checkiftrimonext4isenabledandworking
or
http://howto.unixdev.net/Test_LVM_Trim_Ext4.html
If you don’t get any error when you run fstrim then all should be working correctly. fstrim will give you an error if TRIM support is not enabled correctly
Thanks clopez!
Excellent article thanks!
Is this still true? (from http://askubuntu.com/a/191945/178920)
Yes. Not sure about Ubuntu 12.04, but at least on Debian 6 (Squeeze) is true.
The support for TRIM to LVM was added in the version 2.02.85 (see changelog)
So, if your Linux distribution ships an older version of LVM you may get no support for TRIM.
But doesn’t that mean the “discard” option in fstab is useless? Or does fstrim work around LVM2?
If your filesystem is on top of a LVM volume, and your version of LVM is older than 2.02.85 you won’t get TRIM support. Neither with the discard option nor the fstrim one.
The good thing about fstrim is that it will report the errors. However discard option in fstab will fail silently..
So the best advice to check if your FS has TRIM support enabled on all the required layers is to run fstrim. If fstrim works, then you have TRIM enabled correctly, otherwise you will get an error.
Not being much of a bash person a couple of things confuse me on this.
My machine is set up with system files on SSD and /tmp /var and /home on hard drives to avoid writes to the ssd. This leaves me wondering what to fstrim as all of these come off /.
The other aspect comes from reading about using ssd disks in redhat docs. They suggest trimming when the ssd has run out of “fresh space”. Makes sense as this would ensure wear levelling. There doesn’t seem to be a command available to check how much “fresh space” is left though. All not helped by the fact that there seems to be little info about on what exactly goes on inside these disks.
Thanks for the info by the way. Best I have found.
John
–
How can I fstrim swap partition?
Discard is a valid mount option (/etc/fstab) for swap. The other option is to use the flag -d when adding the swap. Check the manpage of swapon
Hi Clopez,
Thanks for sharing this info – Gracias 😛
I reached this page, by going through your original question on Serverfault.com (
with ref. to your question – you had asked about doing this set-up w/ RAID 0:
—-FS (OS)—-|
——LVM——|
—-RAID 1—–|
—-dm-crypt—|
—SSD1+SSD2—v
However, your post here doesn’t cover the RAID 0 set-up part. Highly appreciate if you can share on how software RAID 1 was achieved (if at all) in this set-up.
Many thanks.
cheers,
s#
This question on serverfault.com is not from myself.
About TRIM over RAID, I never tested it, but my understanding is that it should work out of the box if you have a recent enough kernel that supports it.
MD RAID needs specific support to forward DISCARD requests and that hasn’t even been proposed until 3.6 https://lkml.org/lkml/2012/3/11/261
Hi Clopez,
thanks for tip.
did realize that I have got the RAID #s all over the place in my original comment :D…. but I guess you did get the drift that I was looking for RAID 1 :).
cheers.
S#
I have this bash snippet for the cron.weekly job that gets the mountpoints based on the mounted devices info instead of hard coding them. Maybe it is useful to somebody else:
#!/bin/sh
# this cronjob will discard unused blocks on the ssd mounted filesystems
# get the locally mounted block devices - those starting with "/dev:
# run df -k, pipe the result through grep and save the sixth field in
# in the mountpoint array
mountpoint=( $(df -k | grep ^/dev | awk '{print $6}') )
# loop through the array and run fstrim on every mountpoint
for i in "${mountpoint[@]}"
do
/usr/sbin/fstrim -v $i;
done
Fantastic write-up! Really well written and organized. One of the most useful howto’s I’ve seen. ever.
Great blog! The only problem I encountered in my Ubuntu 12.04 where / is on lvm is the “fstrim /” command can still be run without any error raised even if I added ‘issue_discard = 0’ in lvm.conf, then ‘update-initramfs -u’ and reboot performed. This is not expected. Any comment? Thanks!
It seems that openSUSE 13.1 is currently not prepared for proper handling of the “discard” option in /etc/crypttab. The initrd does not contain the proper command.
Workaround: In order to activate discard on boot time, I just hard-coded the “cryptsetup” commands in /lib/mkinitrd/boot/71-luks.sh to contain “luksOpen –allow-discards”. Works fine so far.
Hi,
it seems that trim is now activated by default on ubuntu 14.04 lts and its derivatives. However, if you are using LVM (not encrypted) and have snapshots of logical volumes, then the system is unable to trim. I can confirm this behaviour. Any ideas?
On Debian (and the like) you can also do
#! /bin/sh
for mount in / /boot /home; do
fstrim $mount
done
Isn’t that a nice paste slip
No preview here.
$ sudo dpkg-reconfigure initramfs-toolsTHX for this tip.
for fs in $(lsblk -o MOUNTPOINT,DISC-MAX,FSTYPE | grep -E '^/.* [1-9]+.* ' | awk '{print $1}'); do
fstrim "$fs"
done
I think, better, will be:
fstrim -a -v
use “man fstrim” for more info
I usually use this in puppet configuration, such as:
cron::weekly { 'fstrim_weekly':
minute => '30',
hour => '12',
weekday => '0',
user => 'root',
command => "fstrim -a -v >> /var/log/fstrim.log; done";
}
OMG, Sorry, my bug))
That must be correct
cron::weekly { 'fstrim_weekly':
minute => '30',
hour => '12',
weekday => '0',
user => 'root',
command => "fstrim -a -v >> /var/log/fstrim.log";
}
No need for looping scripts anymore, there is
fstrim -anow.Thanks for this nice article!
Muchas gracias por este tutorial, mi configuración de portátil es exactamente esta (dm-crypt, LVM, ext4) y es justo lo que necesitaba 😉
Hello, I’ve tried doing the same thing (taking into consideration that my architecture is the same, I have a VG encrypted with luks, and then multiple lvs) but unfortunatelly I still get the error of
fstrim: /usr: FITRIM ioctl failed: Operation not supportedI am using RHEL 6.6
Do you have any idea how can I get it work? Thanks
I’d like to thank you for the effort put into this excellent and clearly written tutorial. Good work and much appreciated
Great writeup, especially in view of the following suspected bug: https://blog.algolia.com/when-solid-state-drives-are-not-that-solid/.
So you know wanna be sure that trim is NOT enabled… 👿
Thank you!
Thank you for this excellently written tutorial and guide. Not only did it explain everything it needed to, but when it made conclusions, it cited why! I recently migrated our Ubuntu (w/ LUKS) machine to an SSD and then came here to figure out all the complex trim stuff. Excellent article and thanks again!
The script from August 2014 update emits every fs name twice on Debian 8.1 resulting in two calls to fstrim.
I use the following script (reduced commands, we have awk already and I’m a traditional UNIX “saving resources” guy):
eval $(lsblk -o MOUNTPOINT,DISC-MAX,FSTYPE | awk '
$1 ~ /^\// && $2 ~ /^[1-9]+/ { fs[$1] = $1; }
END { for (var in fs) { print "fstrim " var ";"; }
}')
To test just replace ‘eval’ by ‘echo’ or place the ‘echo’ inside the printf statement just before ‘fstrim’.
Or, even better:
fstrim –all
😀
Just wanted to thank you for this very useful article, particularly the explanation of why it is not so wise to enable
discardin/etc/fstab.Muy bien explicado, claro y sin enrollarse con conceptos técnicos que no son de interés alguno para quienes el ordenador es una herramienta y no un fin en sí mismo.
Gracias por compartir la información en castellano, algo muy necesario y que por desgracia no es tan habitual como debería (información como esta, de calidad y clara, quiero decir, que hay muchos blogs linuxeros hispanos pero con un nivel bastante mejorable, normalmente), carencia que es motivo de que mucha gente interesada en pasarse al “mundo libre” huya despavorida al ver que tiene que etar haciendo esfuerzos para entender artículos en inglés técnico cada vez que tiene algún problemilla y al final acabaen volviendo a Windows.
Saludos.
hi, is it possible to benefit Trim function with a liveUSB linux attached to an windowsXP-SSD laptop, in order to trim that SSD ?
(please no remarks about why I’m using XP etc..Thanks!)