Growing a ZFS Root Pool
About a month ago I migrated my Ultra 20’s root filesystem from UFS to ZFS using my HOWTO: Migrate a UFS Root Filesystem to ZFS procedure. Today I performed my first live upgrade since, which worked a treat, however now I’m running a bit low on space.
Not for much longer…
Following the live upgrade my root pool is looking a little on the leaner side of how I’d like it:
# zfs list NAME USED AVAIL REFER MOUNTPOINT rootpool 8.51G 1.33G 35K /rootpool rootpool/ROOT 7.00G 1.33G 18K /rootpool/ROOT rootpool/ROOT/snv_99 6.93G 1.33G 4.23G / rootpool/ROOT/snvz_98 71.7M 1.33G 4.08G /.alt.tmp.b-TM.mnt/ rootpool/dump 1.00G 1.33G 1.00G - rootpool/swap 512M 1.54G 297M - #
My rootpool is currently made up of a single 10GB slice which was one of the two slices on the same disk that I used to flip-flop between as I live upgraded the system whilst it was still on UFS.
As I no longer need to do this, thanks to ZFS and the now ZFS aware live upgrade (it uses snapshots and clones), I thought I’d add my now redundant extra slice to the rootpool.
My first thought was: “I’ll just run zpool add and add the slice“, which I attempted, however ZFS didn’t like this idea:
# zpool add -f rootpool c1d0s4 cannot add to 'rootpool': root pool can not have multiple vdevs or separate logs #
So I thought: “Well ZFS isn’t like UFS in that it doesn’t actually lay a filesystem on the entire disk, so in theory if I just change the underlying partitions, they’re contiguous, ZFS should pick it up“.
Well, I thought they were contiguous. As luck would have it, they were separated by a now redundant swap slice, so I blitzed that and merged the 3 into a single slice, without changing the starting cylinder of my original slice, and rebooted.
Update: As Mr Zog commented, I neglected to say exactly what I did here, so here’s what I did when I “blitzed that and merged the 3 into a single slice“:
I used format(1M) to repartition the disk by going to format -> [select the disk] -> partition. I then deleted the old swap slice (slice 1) and the slice I was trying to add (slice 4) and grew the original root slice (slice 0) by specifying the new final cylinder (the final cylinder of the old slice 4).
Hey presto!!!
ZFS has detected this and I now have a much larger root pool, and a lot less wasted space on the disk too:
# zfs list NAME USED AVAIL REFER MOUNTPOINT rootpool 8.52G 13.1G 35K /rootpool rootpool/ROOT 7.00G 13.1G 18K /rootpool/ROOT rootpool/ROOT/snv_99 6.93G 13.1G 4.23G / rootpool/ROOT/snvz_98 71.7M 13.1G 4.08G /.alt.tmp.b-TM.mnt/ rootpool/dump 1.00G 13.1G 1.00G - rootpool/swap 512M 13.3G 297M - #
15 Responses to “Growing a ZFS Root Pool”
Speak Your Mind 
Search
Recommended Book
Related Entries
- ZFS 2GB Swap
- Solaris 10 10/08 (aka Update 6) is Now Available
- OpenSolaris snv_90 Update Pain Free
- ZFS Cheatsheet
- HOWTO: Migrate a UFS Root Filesystem to ZFS
- phpSmug 1.0.5 and 1.1.3 Now Available
- ZSH Prompt Colours in OpenSolaris snv_109 and Later
- Compiling Devilspie 0.22 on OpenSolaris/Nevada
- OpenSolaris Community Will Beat Linux in 2007
- Links for 9 April 2007 - 13 April 2007
I did this too..and it worked! except that all i did was expand a freshly-installed ZFS root slice by stealing some cylinders from the giant chunk of unallocated space and tacking it on. amazing. thanks for the tip!
Hi.
For people like me who are mostly ignorant of Solaris, you did not explain -how- to do anything.
You showed us what -failed-: # zpool add -f rootpool c1d0s4
Maybe you should show us what worked.
– Zog
Good point Zog. I neglected to say how I “blitzed that and merged the 3 into a single slice, without changing the starting cylinder of my original slice”. This was done by adjusting the partition table with format(1M):
format -> [select the disk] -> partition.I’ve added an update note to the main details.
Thanks for pointing out I wasn’t particularly clear.
Mine was the same case but the swap was at starting so i tried to expand the root slice towards starting .. but i got error while labeling it Cannot label disk when partitions are in use as described. as my rootslice is mounted as zfs root .. is there any work around for this ?
I don’t believe you’ll be able to grow the slice successfully that way, but I’ve never tried it so could be wrong.
You may be able to relabel your disk by preventing Solaris from performing the “in use check” by setting the NOINUSE_CHECK env variable to 1, eg
# NOINUSE_CHECK=1; formatVery confusing, specially the post of ZOG is NOT ANSWERED !
# zpool add -f rootpool c1d0s4 didn’t work, ok.
I then deleted the old swap slice (slice 1) and the slice I was trying to add (slice 4) and grew the original root slice (slice 0) by specifying the new final cylinder (the final cylinder of the old slice 4).
How dpes it mean in
# zpool add …… ?!?
Just try to explain what to remove, and what las slice of the second …
Sorry, but this not clear.
M: I’m sorry if it’s not clear to you. I’ve not detailed exactly what to do in format(1M) because it’s a commonly used tool that is also well documented in chapters 11 and 12 of the “System Administration Guide: Devices and File Systems” guide on docs.sun.com.
I was experimenting with doing exactly this on a VM, and for some reason, if I changed the zpool slice so that it’s last cylinder was the last cylinder on the disk when I rebooted, the server would kernel panic. I was still able to boot to failsafe mode, and when I did a zpool list, it said there were no zpools found. When I did a ‘zpool import’ it displayed my zpool, but said that it was faulted and that it’s device was missing.
I set the last cylinder back to what it was before I attempted the expansiopn, and was able to boot, and ‘zpool scrub’ found no errors. After some experimenting, I found out that if I set the last cylinder of the zpool slice to the 2nd last cylinder of the disk, then the expansion worked ok.
Funny thing is, if I create the zpool on that same virtual disk with the last cylinder of the zpool slice set the the last cylinder of the disk, it works ok. It just doesn’t seem to like it when I “expand” the zpool to include that last slice. Maybe that is some querk of the vmware virtual disk, that the last “cylinder” isn’t a full cylinder if the size of the disk is not a multiple of the cylinder size?
John, I suspect this may be something to do with where ZFS stores data. ZFS puts copies of it’s label at the end of the disk too. When you create the filesystem on a whole disk, ZFS secretly uses this space for itself, so you won’t be able to expand ZFS over this.
This is a cool tip! Thanks for posting it, Col.
One additional step that folks might need to do is to enable autoexpand, e.g.,
pfexec zpool set autoexpand=on rpool
Until I did that, “zpool list” kept showing the old size, even after reboot.
Ah, good tip there Mike. I didn’t need to do this when I originally wrote the post, however things are likely to have changed. I’ve just checked and you do indeed need to flip this option to on. Nice one. Thanks.
@Mike Kupfer
this does not work on my OpenSolaris 2009.06 NAS:
root@sv2:~# pfexec zpool set autoexpand=on rpool
cannot set property for ‘rpool’: invalid property ‘autoexpand’
root@sv2:~# uname -a
SunOS sv2 5.11 snv_111b i86pc i386 i86pc Solaris
Do I need an upgrade for this?
/Sunfreak
@sunfreak, you need to be running snv_117 or later to get the autoexpand property. This is when PSARC 2008/353 was integrated.
So to answer your question, YES, you will need to upgrade to get this property, however you may find a reboot (possibly reconfig) will still allow ZFS to reflect the increase in size. I didn’t need to set this property when I initially wrote this post as it didn’t exist.
I just recently loaded a root pool using live upgrade, inspired by this article. All went well, but like any other technologist out there who’s been around for a while, I’m concerned about performance.
Specifically what I’d like to ask is about the performance hit (or improvement) imposed when the swap device is placed within the zpool as a zvol.
Has anyone done a comparison between the performance of swap within the zpool vs. a dedicated partition (for example placed on the outer cylinders where read-time is traditionally faster)?
As a system admin on older versions of solaris, it was always a rule of thumb to create more than one swap device (as a partition) on seperate disks, seperate controllers and on the outer edges (cylinders). This to gain parallelism and improve swap performance.
Albeit I never actually ran a test to prove if this technique really does help swap performance because most of the focus is to prevent swap from occuring at all. But, still, we should know the risks or rewards for using swap within the zfs volume in order to establish our initial system design.
I have not found anything posted about this issue or any tests to support either approach.
@Ray That’s a good question and I think it’s a subject for a future post. I’ll need to do some in depth testing and investigation and see what I can find. Watch this space.