Skip to main content

Any Idea why I'm unable to update #NixOS due to "No space left on device" even if there are 23G available?

Edit: The issue was caused by number of inodes.

this path will be fetched (458.58 MiB download, 464.67 MiB unpacked):
  /nix/store/4mc1wf8b6jad08wlfqaw6zvv7007d51d-linux-firmware-20240513-zstd
copying path '/nix/store/4mc1wf8b6jad08wlfqaw6zvv7007d51d-linux-firmware-20240513-zstd' from 'https://cache.nixos.org'...
building '/nix/store/l4drcr95rzznb9n18rl95cjhzrvwi96q-system-path.drv'...
error: mkdir /nix/store/swcchba29bqmm9mjfgs7flsmhw1fw4wg-system-path//share/locale/az: No space left on device at /nix/store/x65ah6kl9sqjwp4pqm0fn9kb20bc0974-builder.pl line 267.
error: builder for '/nix/store/l4drcr95rzznb9n18rl95cjhzrvwi96q-system-path.drv' failed with exit code 28
error: 1 dependencies of derivation '/nix/store/6zlwarcc34qv5h2rfq2pj83h7zzcimm8-nixos-system-hikari-24.05pre628669.6c0b7a92c301.drv' failed to build

nixpkgs > df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        395M     0  395M   0% /dev
tmpfs           3.9G   16K  3.9G   1% /dev/shm
tmpfs           2.0G  4.3M  2.0G   1% /run
tmpfs           3.9G  1.1M  3.9G   1% /run/wrappers
/dev/dm-0       102G   74G   23G  77% /
/dev/sda1       511M  200M  312M  40% /boot/efi
tmpfs           789M   44K  789M   1% /run/user/1000
#nixos
in reply to Daniel Siepmann

Pol Dellaiera
Pol Dellaiera
• •
Also had the issue with an even greater space left, I really don't know where it comes from. Therefore, `nix-collect-garbage -d` is your friend.
in reply to Pol Dellaiera

Daniel Siepmann
Daniel Siepmann
•

Well, it indeed was inodes as mentioned in https://fosstodon.org/@phaer/112474210853953708 .
I always keep last three generations of my user profile and system profile.

Not sure why it is an issue on that machine after using the setup more than a year now.

Your command seems to remove all the old generations … I always ran nix store gc after deleting all generations older than 3 for system and user and 12 days for home-manager. I typically update once a week.

I've now run your command and reduced the number of generations to two. Let's see how long this will work from now on.

Thank you :)

in reply to Pol Dellaiera

Daniel Siepmann
Daniel Siepmann
•

I had another look today. I still use channels and it looks like every channel is added as root and not garbage collected via my usual commands.

Any idea where I can find further info on how to delete no longer used channels as roots? That should solve all issues I guess.

The issue only seems to exist for the #NixOS, not home-manager.
https://git.daniel-siepmann.de/danielsiepmann/nixpkgs/src/branch/main/home/packages/custom/update-nixos-system/default.nix is my current script to update and clean things up.

#nixos
in reply to Daniel Siepmann

Pol Dellaiera
Pol Dellaiera
• •
I'm sorry it's been a while that I switched to flake, I never used channels since years :( Have you tried to ask on https://discourse.nixos.org/ ?

NixOS Discourse

NixOS community forum. Hosted by flyingcircus.io, running on NixOS 23.05.
NixOS Discourse
in reply to Daniel Siepmann

TheDragon
TheDragon
• •
@Pol nh (short for nix helper) is available in nixpkgs, and offers some more comprehensive GC options - might be worth a try!
@Pol Dellaiera
in reply to Daniel Siepmann

phaer
phaer
• •
maybe you ran out of inodes? You should be able to check with `df -i`
in reply to phaer

Robert Hensing
Robert Hensing
• •
@phaer I bet that's the issue. You'd need about double the default if you're on ext4. Also if your individual inode size is too small you might need more depth than permitted in the btree for /nix/store's entries, or did ext4 get fixed to allow deeper btrees?
@phaer
in reply to phaer

Daniel Siepmann
Daniel Siepmann
•

Thanks, indeed it looks like that's the issue:

df -i
Filesystem      Inodes   IUsed   IFree IUse% Mounted on
/dev/dm-0      6799360 6778817   20543  100% /
in reply to Daniel Siepmann

Quentin Stafford-Fraser
Quentin Stafford-Fraser
• •
Something deep in the back of my memory tells me that you *might* get the same message if your disk had run out of inodes . The nix store must generate a *lot* of directory entries, so I guess that’s possible. You could try `df -i`…
in reply to Daniel Siepmann

Peder Bergebakken Sundt
Peder Bergebakken Sundt
• •
If not the inodes, you might want to clear out /tmp or even look at your boot partition. https://discourse.nixos.org/t/what-to-do-with-a-full-boot-partition/2049/11

What to do with a full /boot partition?

I’m harvesting some questions coming from newcomers (and not-so-new as well) and this appears more than I’d think it would, maybe we should state clearly in the manual that not only the store increase in size during normal operation but /boot does as…
NixOS Discourse
⇧