FreeBSD : RAID5 with gvinum (non-boot)
GEEK00L was wondering whether I have messed around with software based RAID5 under FreeBSD. He needs to set up the same on FreeBSD at work. Since I have not touched anything on RAID5, I gave it a shot and it turned out to be relatively easy. More or less the setup concept is similar to gmirror configuration.
SETUP SCENARIO
3 identical IDE HDD were used in this configuration. (4 identical IDE HDD actually. I will use ad0 in gvinum RAID5 with root in the near future.)
ad0: 76319MB < MAXTOR STM3802110A 3.AAJ> at ata0-master UDMA100
ad1: 76319MB < MAXTOR STM3802110A 3.AAJ> at ata0-slave UDMA100
ad2: 76319MB < MAXTOR STM3802110A 3.AAJ> at ata1-master UDMA100
ad3: 76319MB < MAXTOR STM3802110A 3.AAJ> at ata1-slave UDMA100
DISK PREPARATION
You can skip this step if you are using new hdd. or Probably just do the following and clean them up.
# dd if=/dev/zero of=/dev/ad1 bs=512 count=79
# dd if=/dev/zero of=/dev/ad2 bs=512 count=79
# dd if=/dev/zero of=/dev/ad3 bs=512 count=79
Disk formatting
The hard way:-
# fdisk -v -B -I /dev/ad1
# fdisk -v -B -I /dev/ad2
# fdisk -v -B -I /dev/ad3
# bsdlabel -w -B ad1s1
# bsdlabel -w -B ad2s1
# bsdlabel -w -B ad3s1
# bsdlabel -e ad1s1
# /dev/ad1s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
c: 156301425 0 unused 0 0 # “raw” part, don’t edit
d: 156301425 0 4.2BSD 2048 16384 28552
The above size is for my own configuration. Please amend according to your system.
Creating disk protofile
bsdlabel ad1s1 > label.ad1s1
Restoring disk protofile
bsdlabel -R ad2s1 label.ad1s1
bsdlabel -R ad3s1 label.ad1s1
The easy way, you can use sysinstall to partition the hdd.
RAID VOLUME CONFIGURATION
Create a gvinum raid volume configuration file.
# vi raid.conf
drive d1 device /dev/ad1s1d
drive d2 device /dev/ad2s1d
drive d3 device /dev/ad3s1d
volume rvol
plex org raid5 512k
sd drive d1
sd drive d2
sd drive d3
Initializing gvinum
# kldload geom_vinum
# gvinum create raid.conf
3 drives:
D d3 State: up /dev/ad3s1 A: 0/76318 MB (0%)
D d2 State: up /dev/ad2s1 A: 0/76318 MB (0%)
D d1 State: up /dev/ad1s1 A: 0/76318 MB (0%)1 volume:
V rvol State: up Plexes: 1 Size: 149 GB
1 plex:
P rvol.p0 R5 State: up Subdisks: 3 Size: 149 GB
3 subdisks:
S rvol.p0.s2 State: up D: d3 Size: 74 GB
S rvol.p0.s1 State: up D: d2 Size: 74 GB
S rvol.p0.s0 State: up D: d1 Size: 74 GB
Saving the gvinum configuration
# gvinum start rvol
# gvinum saveconfig
In dmesg, you will see these messages. It seems that I have some issues with stripe size. I will iron out this next time as it is still in working stage.
gvinum: size of sd rvol.p0.s0 is not a multiple of plex stripesize, taking off 446464 bytes
gvinum: size of sd rvol.p0.s1 is not a multiple of plex stripesize, taking off 446464 bytes
gvinum: size of sd rvol.p0.s2 is not a multiple of plex stripesize, taking off 446464 bytes
GEOM_VINUM: subdisk rvol.p0.s2 state change: stale -> up
GEOM_VINUM: subdisk rvol.p0.s1 state change: stale -> up
GEOM_VINUM: plex rvol.p0 state change: down -> degraded
GEOM_VINUM: subdisk rvol.p0.s0 state change: stale -> up
GEOM_VINUM: plex rvol.p0 state change: degraded -> up
The gvinum is a rather user-friendly tool. You can display the raid volume by issuing gvinum list. Apart from this, you drop down to its command prompt by just invoking gvinum.
# gvinum
gvinum -> help
COMMANDS
checkparity [-f] plex
Check the parity blocks of a RAID-5 plex.
create description-file
Create as per description-file or open editor.
l | list [-r] [-v] [-V] [volume | plex | subdisk]
List information about specified objects.
ld [-r] [-v] [-V] [volume]
List information about drives.
ls [-r] [-v] [-V] [subdisk]
List information about subdisks.
lp [-r] [-v] [-V] [plex]
List information about plexes.
lv [-r] [-v] [-V] [volume]
List information about volumes.
move | mv -f drive object …
Move the object(s) to the specified drive.
quit Exit the vinum program when running in interactive mode. Nor-
mally this would be done by entering the EOF character.
rename [-r] [drive | subdisk | plex | volume] newname
Change the name of the specified object.
rebuildparity plex [-f]
Rebuild the parity blocks of a RAID-5 plex.
rm [-r] volume | plex | subdisk | drive
Remove an object.
saveconfig
Save vinum configuration to disk after configuration failures.
setstate [-f] state [volume | plex | subdisk | drive]
Set state without influencing other objects, for diagnostic pur-
poses only.
start [-S size] volume | plex | subdisk
Allow the system to access the objects.
gvinum ->
FORMAT AND MOUNT GVINUM RAID5 VOLUME
# newfs /dev/gvinum/rvol
# mount /dev/gvinum/rvol /mnt
Note : Replace rvol with whatever volume name you’ve set in raid.conf. In my case, it is volume “rvol”.
Loading gvinum upon boot
Edit /boot/loader.conf and set :-
geom_vinum_load="YES"
Remember to edit /etc/fstab to enable mounting on boot.
