FreeNAS uses ZFS to store all your data. ZFS has a bunch of features that ensures all your data will be safe but not only that it has some very effective read and write caching techniques.
The write cache is called the ZFS Intent Log (ZIL) and read cache is the Level 2 Adjustable Replacement Cache (L2ARC).
The ZFS Intent Log is a logging mechanism where all the of data to be written is stored, then later flushed as a transactional write. It is typically stored on a fast device such as a SSD, for writes smaller than 64kB the ZIL stores the data on the fast device, but for larger sizes the data is not stored in the ZIL, only the pointers to the synced data is stored.
Due to this not all applications can take advantage of of the ZIL. Database applications have the largest benefit of a ZIL (MySQL,PostgreSQL,Oracle) as well as NFS and iSCSI. Typical moving around of data around the file system will not see much of a benefit.
Rules of Thumb:
– ZILs are typically very small. Just 1-2GB of ZIL storage for a server with Gb LAN is overkill. Do not oversize your ZIL excessively.
– ZILs do not commit writes to disk asynchronously and therefore can result in lost writes if the drive fails. Because of this it is recommended you put your ZILs in a mirrored configuration.
In a ZFS system a caching technique called ARC caches as much of your dataset in RAM as possible. This allows frequently data to accessed very quickly, much faster than having to go the backing HDD array. It follows that more RAM means more ARC space which in turn means more data can be cached. The maximum space ARC will occupy can be changed but typically it uses up to 7/8 of your RAM, however will release memory back to system as other applications require it. So if you had 128GB of RAM in your system and your total dataset size was 1TB, 11.2% of all your data will reside in the ARC resulting in significantly faster access times.
The ARC has a very intelligent algorithm to manage the cache to ensure that data present is data you actually want cached. You don’t want data you never actually use in your cache as it would be a waste of resources.
Although this algorithm is very effective, as your data-set increases the % of your data that can be stored in the cache gets less and less, and the cache becomes less effective.
This is where the L2ARC comes in, it is an extension of the ARC cache hence the name Level 2 ARC. Using fast disks such as SSDs the cache size can increase dramatically when the option of adding more RAM becomes either physically or economically impossible.
Now unlike the ARC cache, the L2ARC does not populate itself with data as soon as you set it up, its loads itself with data as it gets ejected out of the ARC. Therefore it can take some time to fully populate your L2ARC and see the added performance benefit.