Ubuntu 8.04.1 Server LTSにDRBDを入れてほげる

すべて VirtualBox 上の Ubuntu Server で実行しています.

セットアップ

仮想マシンの用意

同一構成の仮想マシンを2つ用意します.

Memory 384MB
HDD 8GB
ACPI 有効
IO ACPI 有効
PAE/NX 有効
SATA 有効
Ubuntu 8.04.1 Server のインストール
基本 4.0GB / ext3 システム
論理 4.2GB /opt ext3 DRBD用
swap 400MB swap swap
  • ソフトウェアは何も入れない

ホスト*1上で実行.ユーザ名は rust とします.

% sudo apt-get install bridge-utils
% sudo vim /etc/network/interfaces
----add----
auto br0
iface br0 inet dhcp
  bridge_ports eth0
-----------
% sudo /etc/init.d/networking restart
% sudo VBoxAddIf vbox0 rust br0        # Primary 用
% sudo VBoxAddIf vbox1 rust br0        # Secondary 用
% VBoxManage modifyvm "Ubuntu 8.04.1 Server Primary" -hostifdev1 vbox0
% VBoxManage modifyvm "Ubuntu 8.04.1 Server Secondary" -hostifdev1 vbox1

これで2つの仮想マシンがお互いに通信できるようになります.

DRBD のインストール

以後は Primary/Secondary 共に実行していきます.

外部からアクセスするために,openssh-server をインストール
$ sudo apt-get install openssh-server

後はホスト上から操作します.

DRBD のインストール
$ sudo apt-get install build-essential binutils cpp gcc autoconf automake1.9 libtool \
autotools-dev g++ make flex
Kernel sourceのダウンロードとビルドなど

DRBD は 8.2.7 を.

$ sudo -i
# apt-get install libncurses5-dev linux-source-2.6.24
# cd /usr/src
# tar jxvf linux-source-2.6.24.tar.bz2
# cd linux-source-2.6.24
# make mrproper
# cp /boot/config-2.6.24-19-server /usr/src/linux-source-2.6.24/.config
# make menuconfig
# make prepare
# make
# cd ../
# wget http://oss.linbit.com/drbd/8.2/drbd-8.2.7.tar.gz
# tar zxvf drbd-8.2.7.tar.gz
# cd drbd-8.2.7.tar.gz
# make KDIR=/usr/src/linux-source-2.6.24
# make install
# mv /lib/modules/2.6.24.6/kernel/drivers/block/drbd.ko \
/lib/modules/2.6.24-19-server/kernel/drivers/block/
# modprobe drbd
# echo 'drbd' >> /etc/modules
# update-rc.d drbd defaults
 Adding system startup for /etc/init.d/drbd ...
   /etc/rc0.d/K20drbd -> ../init.d/drbd
   /etc/rc1.d/K20drbd -> ../init.d/drbd
   /etc/rc6.d/K20drbd -> ../init.d/drbd
   /etc/rc2.d/S20drbd -> ../init.d/drbd
   /etc/rc3.d/S20drbd -> ../init.d/drbd
   /etc/rc4.d/S20drbd -> ../init.d/drbd
   /etc/rc5.d/S20drbd -> ../init.d/drbd
# lsmod | grep drbd
drbd                  213000  0 
cn                      9632  1 drbd
設定と実行と

次は設定.サンプルとここを参考にしました.(設定ファイルはこちらhttp://gist.github.com/60611)

# cp /etc/drbd.conf /etc/drbd.conf.org
# /etc/init.d/drbd restart
Restarting all DRBD resources/dev/drbd0: Failure: (114) Lower device is already claimed. This usually means it is mounted.
Command '/sbin/drbdsetup /dev/drbd0 disk /dev/sda6 /dev/sda6 internal --set-defaults --create-device --on-io-error=detach' terminated with exit code 10
.

などとエラーになる.なので unmount して fstab からも外す.

# umount /opt
# vim /etc/fstab
# /etc/init.d/drbd restart
Restarting all DRBD resources/dev/drbd0: Failure: (119) No valid meta-data signature found.

	==> Use 'drbdadm create-md res' to initialize meta-data area. <==

Command '/sbin/drbdsetup /dev/drbd0 disk /dev/sda6 /dev/sda6 internal --set-defaults --create-device --on-io-error=detach' terminated with exit code 10
.

またしてもエラー.どうも初期化しないといけないようなので,初期化する.res はリソース名なので,この場合は r0 を指定.

# drbdadm create-md r0
md_offset 4170145792
al_offset 4170113024
bm_offset 4169981952

Found ext3 filesystem which uses 4072412 kB
current configuration leaves usable 4072248 kB

Device size would be truncated, which
would corrupt data and result in
'access beyond end of device' errors.
You need to either
   * use external meta data (recommended)
   * shrink that filesystem first
   * zero out the device (destroy the filesystem)
Operation refused.

Command 'drbdmeta /dev/drbd0 v08 /dev/sda6 internal create-md' terminated with exit code 40
drbdadm create-md r0: exited with code 40

またしてもまたしてもエラー.今度は容量制限をオーバーしたようなので,仕方なくリサイズして再起動.

# fdisk /dev/sda

The number of cylinders for this disk is set to 1044.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00048f37

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1         486     3903763+  83  Linux
/dev/sda2             487        1044     4482135    5  Extended
/dev/sda5             994        1044      409626   82  Linux swap / Solaris
/dev/sda6             487         993     4072446   83  Linux

Partition table entries are not in disk order

Command (m for help): d
Partition number (1-6): 6

Command (m for help): n
Command action
   l   logical (5 or over)
   p   primary partition (1-4)
l
First cylinder (487-1044, default 487): 
Using default value 487
Last cylinder or +size or +sizeM or +sizeK (487-993, default 993): +2000M

Command (m for help): p

Disk /dev/sda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00048f37

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1         486     3903763+  83  Linux
/dev/sda2             487        1044     4482135    5  Extended
/dev/sda5             994        1044      409626   82  Linux swap / Solaris
/dev/sda6             487         730     1959898+  83  Linux

Partition table entries are not in disk order

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.
# reboot

再起動したら初期化.

# drbdadm create-md r0
md_offset 2006929408
al_offset 2006896640
bm_offset 2006835200

Found some data 
 ==> This might destroy existing data! <==

Do you want to proceed?
[need to type 'yes' to confirm] yes

Writing meta data...
initialising activity log
NOT initialized bitmap
New drbd meta data block sucessfully created.

今度はうまくいったようだ.そして起動.

# /etc/init.d/drbd restart
Restarting all DRBD resources.
# cat /proc/drbd 
version: 8.0.11 (api:86/proto:86)
GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12 11:56:43
 0: cs:Connected st:Secondary/Secondary ds:Inconsistent/Inconsistent C r---
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0
	resync: used:0/31 hits:0 misses:0 starving:0 dirty:0 changed:0
	act_log: used:0/257 hits:0 misses:0 starving:0 dirty:0 changed:0

Inconsistent なのでどちらも Secondary として動いている.同期をとるために,Primary の方で下記を実行.

# drbdadm -- --overwrite-data-of-peer primary all
/etc/init.d/drbd status                        
drbd driver loaded OK; device status:
version: 8.0.11 (api:86/proto:86)
GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12 11:56:43
m:res  cs          st                 ds                     p  mounted  fstype
0:r0   SyncSource  Primary/Secondary  UpToDate/Inconsistent  C
...    sync'ed:    2.3%               (1917560/1959800)K

同期が始まりました.これは「初期フル同期」というらしいです.Secondary ではこんな感じ.

# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.0.11 (api:86/proto:86)
GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12 11:56:43
m:res  cs          st                 ds                     p  mounted  fstype
0:r0   SyncTarget  Secondary/Primary  Inconsistent/UpToDate  C
...    sync'ed:    4.4%               (1878456/1959800)K

しばらくして同期が完了すると,

on Primary
# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.0.11 (api:86/proto:86)
GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12 11:56:43
m:res  cs         st                 ds                 p  mounted  fstype
0:r0   Connected  Primary/Secondary  UpToDate/UpToDate  C

on Secondary
# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.0.11 (api:86/proto:86)
GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12 11:56:43
m:res  cs         st                 ds                 p  mounted  fstype
0:r0   Connected  Secondary/Primary  UpToDate/UpToDate  C

となります.

format と mount

次は format します.なんとなく ext3 で.そして mount も.

# mkfs.ext3 /dev/drbd0 
mke2fs 1.40.8 (13-Mar-2008)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
122880 inodes, 489950 blocks
24497 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=503316480
15 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912

Writing inode tables: done                            
Creating journal (8192 blocks): 
done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 26 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
# mount /dev/drbd0 /opt
# ls /opt/
lost+found
# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             3.7G  2.9G  650M  82% /
varrun                189M   44K  189M   1% /var/run
varlock               189M  4.0K  189M   1% /var/lock
udev                  189M   52K  189M   1% /dev
devshm                189M     0  189M   0% /dev/shm
/dev/drbd0            1.9G   35M  1.8G   2% /opt
実験など

ここにある実験を同じようにやってみます.
1. Primary でファイルをコピー後に,強制的に Secondary に降格させる.

# cp /etc/drbd.conf /opt
# ls /opt/
drbd.conf  lost+found
# umount /opt
# drbdadm secondary r0   
# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.0.11 (api:86/proto:86)
GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12 11:56:43
m:res  cs         st                   ds                 p  mounted  fstype
0:r0   Connected  Secondary/Secondary  UpToDate/UpToDate  C

2. 次にSecondary を Primary に昇格させて,mount する.

# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.0.11 (api:86/proto:86)
GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12 11:56:43
m:res  cs         st                   ds                 p  mounted  fstype
0:r0   Connected  Secondary/Secondary  UpToDate/UpToDate  C
# drbdadm primary r0
# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.0.11 (api:86/proto:86)
GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12 11:56:43
m:res  cs         st                 ds                 p  mounted  fstype
0:r0   Connected  Primary/Secondary  UpToDate/UpToDate  C
# mount /dev/drbd0 /opt
# ls /opt/
drbd.conf  lost+found

よし,レプリケーション成功.さらに元に戻す.
3. unmount して Primary(旧Secondary)を Secondary に降格する.

# umount /opt
# drbdadm secondary r0

4. Secondary(旧Primary)を Primary に昇格する.

# drbdadm primary r0
# mount /dev/drbd0 /opt
# ls /opt/
drbd.conf  lost+found

ひとまず実験は終わり.次は Heatbeat と組み合わせてほげるかな.

*1:VirtualBox を動かしている Ubuntu