"No Space Left on Device" but df Shows Free Space? You Ran Out of Inodes

The disk is 80% free, df -h looks perfectly healthy, yet every write fails with No space left on device. You cannot create a small file, services refuse to start, even touch bails. This is a "space mystery" every operator runs into eventually. The problem usually is not capacity — it is inodes.

A Linux filesystem has two independent limits: one governs data blocks, the other inodes. df -h only reports the first, which is why you see free space; df -i reports the second, which may already be pegged at 100%. This article explains the difference, then shows how to find which directory ate all the inodes, how to clean it safely, and how to stop it from coming back.

df -h and df -i: two separate ledgers

A file on disk consumes two things: data blocks hold its contents, and an inode holds its "registry entry." The inode stores the size, permissions, owner, timestamps, and pointers to the data blocks — but not the contents themselves. Every file, directory, and symlink you create spends exactly one inode, even a zero-byte file.

The trap is that blocks and inodes are two independent ledgers, and exhausting either one produces the same No space left on device error. You tell them apart by behavior:

• Out of blocks → large files will not write, but you can still create empty files; df -h shows Use% near 100%.
• Out of inodes → nothing new can be created (not even empty files or directories), but existing files can still be appended to; df -h may show only 30% used while df -i shows IUse% at 100%.

So the first step is always both commands. Here is a textbook inode-exhausted box — note the contrast between the two outputs:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        59G   18G   39G  32% /

$ df -i
Filesystem      Inodes IUsed   IFree IUse% Mounted on
/dev/sda1      3932160 3920000   12160  100% /

The disk is one-third full, but the inodes are gone. The inode count is fixed at format time (mkfs); ext4 allocates one inode per 16 KB by default, which is why a 59G disk gets about 3.9 million inodes. You cannot add more afterward — increasing the count means backing up and reformatting with mkfs.ext4 -N <count> to change the ratio. Plan that restore before you start: data you encrypted locally comes back on a fresh disk only with its key — see Local Encryption and Data Recovery. So the right approach is not "add inodes" but "find the files eating them and delete them."

What ate the inodes: find the small-file hogs

Inode exhaustion is almost always caused by a huge number of tiny files. A few hundred thousand files of a few KB each occupy hardly any data blocks, but they spend one inode apiece — the box above holds only 18G of actual data but 3.9 million files. The usual sources:

• PHP session files (sess_* under /var/lib/php/sessions/) where garbage collection never runs;
• A mail queue piling up bounced messages (/var/spool/exim, /var/spool/postfix);
• Temporary files under /tmp and /var/tmp that systemd-tmpfiles never cleans, or a script that writes and never deletes;
• Application caches, node_modules, and npm/yarn caches full of thousands of tiny files;
• A broken cron job creating one file per second and never cleaning up.

The inode-aware du is the fastest way to locate them (GNU coreutils 8.22+ supports --inodes), and its output names the guilty directory outright:

$ du --inodes -x -s /var/* 2>/dev/null | sort -rn | head
1523310  /var/lib
 840021  /var/lib/php/sessions
 610204  /var/cache
  83110  /var/log

/var/lib/php/sessions accounts for 840,000 inodes on its own — the culprit is basically identified. On an older coreutils without --inodes, count files per directory instead, ignoring size entirely:

find /var -xdev -type f | cut -d/ -f3 | sort | uniq -c | sort -rn | head

Once a directory shows an abnormal count, drill into it to confirm:

find /var/lib/php/sessions -type f | wc -l

An ls will show a directory stuffed with sess_-prefixed files, many days or weeks old — the session cleanup mechanism simply never ran.

Clean up safely: do not rm a giant glob

Be careful here; two traps are the most common.

First, avoid rm /var/lib/php/sessions/* — hundreds of thousands of files will blow past the shell's argument list limit (ARG_MAX), and expanding the glob is slow on its own. Use find -delete or batch through xargs:

find /var/lib/php/sessions -type f -name 'sess_*' -mtime +1 -delete

-mtime +1 removes only sessions older than one day and leaves active ones alone, which is relatively safe. Run the same command without -delete first to see how many files and whether the scope is right, then actually run it.

Second, df -i may still read 100% after you delete — that usually means "deleted but not released." On Linux, as long as a process still holds a file open, its inode and blocks stay allocated even after unlink. Check with lsof:

$ lsof +L1 | grep -v COMMAND | head
php-fpm   1234 www-data  3w  REG  8,1  0  0  12345 /var/lib/php/sessions/sess_abc (deleted)

+L1 lists files that have been deleted but are still open — a NLINK of 0 is the tell. Find that process, restart it, or truncate the path with > /path/to/file, and the inode returns immediately. Scanning with lsof before deleting saves a lot of dead ends.

Fix the root cause: keep inodes from silently filling again

Cleaning up once treats the symptom. To fix it for good, do three things.

1. Make the session and temp-file cleanup actually run. PHP's gc pairs session.gc_maxlifetime (when a session is stale, default 1440 seconds) with session.gc_probability / gc_divisor (how often it fires, default 1/100). Many inode-exhausted servers trace back to gc_probability set to 0, so gc never runs. Debian/Ubuntu add a second layer: /etc/cron.d/php runs sessionclean on a schedule — confirm that cron still exists and was not disabled by a PHP upgrade. systemd-tmpfiles periodically cleans /tmp and /var/tmp; confirm the relevant units are running.

2. Alert on both meters. Adding only disk usage to monitoring is not enough; add df -i inode usage with its own thresholds — warn at 80%, critical at 90%. Many overnight outages happen because inodes crept toward 100% with nobody watching.

3. Find the process creating small files nonstop. A steep inode-growth curve means something is continuously manufacturing tiny files — usually a cache strategy gone wrong or temp files that are created and never removed. Once you locate the directory, fix it in the application code or config rather than deleting by hand every few days.

To see how much inode headroom you have and how large the pool is:

df -i
tune2fs -l /dev/sda1 | grep -iE 'inode (count|size)'

tune2fs shows the filesystem's total inode count (Inode count) and per-inode size (Inode size — 256 bytes by default on ext4). That last number is why you cannot just crank the inode count up freely: every inode reserves 256 bytes of metadata space, so doubling the count also doubles the metadata footprint.

Next time df -h looks fine but writes fail, run df -i first — it is the first step in telling the two resource types apart, and the one most easily skipped. Running out of blocks and running out of inodes have entirely different fixes: the former means deleting large files, growing the disk, or rotating logs; the latter means clearing huge numbers of small files and fixing what produced them. Pick the wrong direction and no amount of deleting helps.

---

When troubleshooting remotely, connect with an SSH client like Termark and run df -i, du --inodes, and find right in the terminal — copying the result to a teammate is just as easy. This class of "resource silently filled up" problem is ultimately a storage-policy question: what to keep, where, how long, and who cleans it. The client side has the same question locally — see Data Storage Path.

References

• df man page
• du man page
• lsof man page
• tune2fs man page