ログ溜め込んで解析する基盤としてMongoDBイイんじゃない？ってのは前からよく聞く話でしたが、
こないだのMongoTokyo2012(秀逸なまとめ)みてたら、みんなやってんだなぁと。
以前WindowsでSpringDataからいじった事はあったのですが、(MongoDBをSpring Dataからホゲホゲしてみる | shinodogg.com)
運用考えた構成で試してみようかな的な感じで。
■ MongoDBのインストールと動作確認
Webサイト(http://www.mongodb.org/downloads)からLinux 64-bitのURLをコピって
wgetでバイナリをダウンロード
# cd /usr/local/src # export http_proxy=ごにょごにょ # wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.0.2.tgz 略 Saving to: `mongodb-linux-x86_64-2.0.2.tgz' 100%[======================================>] 38,350,811 6.81M/s in 5.2s 2012-01-24 10:18:08 (7.10 MB/s) - `mongodb-linux-x86_64-2.0.2.tgz' saved [38350811/38350811]
解凍すると↓こんな感じ。ダンプとかリストアとかtopとか、ほうほうって感じですね。
# tar xvf mongodb-linux-x86_64-2.0.2.tgz mongodb-linux-x86_64-2.0.2/ mongodb-linux-x86_64-2.0.2/THIRD-PARTY-NOTICES mongodb-linux-x86_64-2.0.2/GNU-AGPL-3.0 mongodb-linux-x86_64-2.0.2/README mongodb-linux-x86_64-2.0.2/bin/ mongodb-linux-x86_64-2.0.2/bin/mongo mongodb-linux-x86_64-2.0.2/bin/mongotop mongodb-linux-x86_64-2.0.2/bin/mongos mongodb-linux-x86_64-2.0.2/bin/mongodump mongodb-linux-x86_64-2.0.2/bin/mongostat mongodb-linux-x86_64-2.0.2/bin/mongofiles mongodb-linux-x86_64-2.0.2/bin/mongoimport mongodb-linux-x86_64-2.0.2/bin/mongosniff mongodb-linux-x86_64-2.0.2/bin/mongorestore mongodb-linux-x86_64-2.0.2/bin/mongod mongodb-linux-x86_64-2.0.2/bin/bsondump mongodb-linux-x86_64-2.0.2/bin/mongoexport
てか、configureとかmakeとか要らんのねーって。この辺も楽チンなとこでしょうか。
/usr/local/mongodbって感じにしておきたいので、ディレクトリ移動して
シンボリックリンク貼っておきやす。
# mv mongodb-linux-x86_64-2.0.2 /usr/local # cd /usr/local # ln -s mongodb-linux-x86_64-2.0.2 mongodb # ls -l | grep mongodb lrwxrwxrwx 1 root root 26 2012-01-24 10:24 mongodb -> mongodb-linux-x86_64-2.0.2 drwxr-xr-x 3 root root 4096 2011-12-15 06:26 mongodb-linux-x86_64-2.0.2
調子に乗ってそのまま起動したらなんか怒られました。
所定のディレクトリがねぇだろうがゴルァって話みたいです。
# /usr/local/mongodb/bin/mongod
/usr/local/mongodb/bin/mongod --help for help and startup options
Tue Jan 24 10:28:42 [initandlisten] MongoDB starting : pid=3275 port=27017 dbpath=/data/db/ 64-bit host=VirtualBoxUbuntu
Tue Jan 24 10:28:42 [initandlisten] db version v2.0.2, pdfile version 4.5
Tue Jan 24 10:28:42 [initandlisten] git version: 514b122d308928517f5841888ceaa4246a7f18e3
Tue Jan 24 10:28:42 [initandlisten] build info: Linux bs-linux64.10gen.cc 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Tue Jan 24 10:28:42 [initandlisten] options: {}
Tue Jan 24 10:28:42 [initandlisten] exception in initAndListen: 10296 dbpath (/data/db/) does not exist, terminating
Tue Jan 24 10:28:42 dbexit:
Tue Jan 24 10:28:42 [initandlisten] shutdown: going to close listening sockets...
Tue Jan 24 10:28:42 [initandlisten] shutdown: going to flush diaglog...
Tue Jan 24 10:28:42 [initandlisten] shutdown: going to close sockets...
Tue Jan 24 10:28:42 [initandlisten] shutdown: waiting for fs preallocator...
Tue Jan 24 10:28:42 [initandlisten] shutdown: lock for final commit...
Tue Jan 24 10:28:42 [initandlisten] shutdown: final commit...
Tue Jan 24 10:28:42 [initandlisten] shutdown: closing all files...
Tue Jan 24 10:28:42 [initandlisten] closeAllFiles() finished
Tue Jan 24 10:28:42 dbexit: really exiting now
# echo $?
100
データを格納しておくようのディレクトリが必要なのねって事で作って叩きます。
# mkdir -p /data/db
# /usr/local/mongodb/bin/mongod
/usr/local/mongodb/bin/mongod --help for help and startup options
Tue Jan 24 10:30:37 [initandlisten] MongoDB starting : pid=3281 port=27017 dbpath=/data/db/ 64-bit host=ほげほげ
Tue Jan 24 10:30:37 [initandlisten] db version v2.0.2, pdfile version 4.5
Tue Jan 24 10:30:37 [initandlisten] git version: 514b122d308928517f5841888ceaa4246a7f18e3
Tue Jan 24 10:30:37 [initandlisten] build info: Linux bs-linux64.10gen.cc 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Tue Jan 24 10:30:37 [initandlisten] options: {}
Tue Jan 24 10:30:38 [initandlisten] journal dir=/data/db/journal
Tue Jan 24 10:30:38 [initandlisten] recover : no journal files present, no recovery needed
Tue Jan 24 10:30:38 [initandlisten] preallocateIsFaster=true 14.9
Tue Jan 24 10:30:39 [initandlisten] preallocateIsFaster=true 15.24
Tue Jan 24 10:30:41 [initandlisten] preallocateIsFaster=true 15.32
Tue Jan 24 10:30:41 [initandlisten] preallocateIsFaster check took 3.469 secs
Tue Jan 24 10:30:41 [initandlisten] preallocating a journal file /data/db/journal/prealloc.0
503316480/1073741824 46%
954204160/1073741824 88%
Tue Jan 24 10:30:50 [initandlisten] preallocating a journal file /data/db/journal/prealloc.1
377487360/1073741824 35%
597688320/1073741824 55%
Tue Jan 24 10:30:59 [initandlisten] File I/O errno:2 No such file or directory
preallocating a journal file /data/db/journal/prealloc.2
Tue Jan 24 10:30:59 [initandlisten] File I/O errno:2 No such file or directory
waiting for connections on port 27017
Tue Jan 24 10:30:59 [websvr] admin web console waiting for connections on port 28017
それっぽく上がったみたいなのでバックグラウンドにしておきます。
＃ /etc/init.dがどうのとか、そういうのは置いておいて。。
^Z zsh: suspended /usr/local/mongodb/bin/mongod # bg [1] + continued /usr/local/mongodb/bin/mongod
データを入れたり出したりしてみます。
MongoDBのシェルはbin/mongoなので↓のようにシェルを起動します。
# ./mongo MongoDB shell version: 2.0.2 connecting to: test Tue Jan 24 11:44:06 [initandlisten] connection accepted from 127.0.0.1:52563 #1 >
↓を参考にしてドキュメントをインサートしようとしたら、、
http://www.mongodb.org/pages/viewpage.action?pageId=7209154
diskの残りがねーよ、と。。
Tue Jan 24 12:32:15 [FileAllocator] will try again in 10 seconds Tue Jan 24 12:32:25 [FileAllocator] allocating new datafile /data/db/test.ns, filling with zeroes... Tue Jan 24 12:32:25 [FileAllocator] creating directory /data/db/_tmp Tue Jan 24 12:32:25 [FileAllocator] error failed to allocate new file: /data/db/test.ns size: 16777216 errno:28 No space left on device
またまた、とか思ってみてみたら↓そんなに食うんですかいっていう…
/data/db/journal# ls -l total 1754916 -rw------- 1 root root 1073741824 2012-01-24 10:30 j._0 -rw------- 1 root root 721301504 2012-01-24 10:30 prealloc.1 -rw------- 1 root root 1888256 2012-01-24 10:30 prealloc.2 /data/db/journal# du -skh 1.7G .
とりあえずjounal配下を全部消して、
/data/db/journal# rm -r * /data/db/journal# ls -l total 0
プロセスkillして、、
# kill 3281 VirtualBoxUbuntu# Tue Jan 24 12:35:05 got kill or ctrl c or hup signal 15 (Terminated), will terminate after current cmd ends Tue Jan 24 12:35:05 [interruptThread] now exiting Tue Jan 24 12:35:05 dbexit: Tue Jan 24 12:35:05 [interruptThread] shutdown: going to close listening sockets... Tue Jan 24 12:35:05 [interruptThread] closing listening socket: 6 Tue Jan 24 12:35:05 [interruptThread] closing listening socket: 7 略 Tue Jan 24 12:35:05 [interruptThread] journalCleanup... Tue Jan 24 12:35:05 [interruptThread] removeJournalFiles Tue Jan 24 12:35:05 [interruptThread] shutdown: removing fs lock... Tue Jan 24 12:35:05 dbexit: really exiting now [1] + exit 12 /usr/local/mongodb/bin/mongod
ジャーナル無しで上げ直します
# /usr/local/mongodb/bin/mongod --nojournal
Tue Jan 24 12:37:59 [initandlisten] MongoDB starting : pid=3580 port=27017 dbpath=/data/db/ 64-bit host=VirtualBoxUbuntu
Tue Jan 24 12:37:59 [initandlisten] db version v2.0.2, pdfile version 4.5
Tue Jan 24 12:37:59 [initandlisten] git version: 514b122d308928517f5841888ceaa4246a7f18e3
Tue Jan 24 12:37:59 [initandlisten] build info: Linux bs-linux64.10gen.cc 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Tue Jan 24 12:37:59 [initandlisten] options: { nojournal: true }
気を取り直して、ドキュメントを突っ込んでみます。
/usr/local/mongodb/bin# ./mongo
MongoDB shell version: 2.0.2
connecting to: test
> doc = { author: 'joe',
... created : new Date('03/28/2009'),
... title : 'Yet another blog post',
... text : 'Here is the text...',
... tags : [ 'example', 'joe' ],
... comments : [ { author: 'jim', comment: 'I disagree' },
... { author: 'nancy', comment: 'Good post' }
... ]
... }
{
"author" : "joe",
"created" : ISODate("2009-03-27T15:00:00Z"),
"title" : "Yet another blog post",
"text" : "Here is the text...",
"tags" : [
"example",
"joe"
],
"comments" : [
{
"author" : "jim",
"comment" : "I disagree"
},
{
"author" : "nancy",
"comment" : "Good post"
}
]
}
> db.posts.insert(doc);
コメントのauthorを指定して検索も出来ました。
> db.posts.find( { "comments.author" : "jim" } )
{ "_id" : ObjectId("4f1e27f47d7053b05fb67e39"), "author" : "joe", "created" : ISODate("2009-03-27T15:00:00Z"), "title" : "Yet another blog post", "text" : "Here is the text...", "tags" : [ "example", "joe" ], "comments" : [ { "author" : "jim", "comment" : "I disagree" }, { "author" : "nancy", "comment" : "Good post" } ] }
■ Replica Setsで冗長性を確保する
Shardingとかもあれですが、とりあえず冗長性確保したいのと、
Master/Slaveじゃなくて自動フェールオーバーな仕掛けを持った
Replica Setsってのがあるので、ソレを試してみたいな、と。
↓の記事がとてもよくまとまっていて分かりやすかったので参考にさせていただきました。
MongoDBのReplica Setsについての概要 | ryopekoの日記
それぞれのデータ用のディレクトリ作ります。
mkdir -p /data/replica1 mkdir -p /data/replica2 mkdir -p /data/replica3
それぞれをnojournalで立ち上げます。restオプションはブラウザで確認できるようにする
オプションらしいです。
今回はそれぞれがどんなログ出すのか後からみてみたいので、
コンソールのタブを3つ立ち上げてそれぞれでやってみます。
タブ1： /usr/local/mongodb/bin/mongod --replSet replica --port 27017 --dbpath /data/replica1 --rest --nojournal タブ2： /usr/local/mongodb/bin/mongod --replSet replica --port 27018 --dbpath /data/replica2 --rest --nojournal タブ3： /usr/local/mongodb/bin/mongod --replSet replica --port 27019 --dbpath /data/replica3 --rest --nojournal
どこのノードでも↓のように設定がちゃんとされてないんじゃね？系のログが出ています。
Tue Jan 24 13:48:16 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) Tue Jan 24 13:48:16 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done Tue Jan 24 13:48:26 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
↓にアクセスするとHadoop並みのイカした画面が出てきます。
localhost:28017 localhost:28018 localhost:28019
続いてレプリカの設定を入れます。
/usr/local/mongodb/bin# ./mongo localhost:27017
MongoDB shell version: 2.0.2
connecting to: localhost:27017/test
> conf = {_id: 'replica', members: [
... {_id: 0, host: 'localhost:27017'},
... {_id: 1, host: 'localhost:27018'},
... {_id: 2, host: 'localhost:27019'}]
... }
{
"_id" : "replica",
"members" : [
{
"_id" : 0,
"host" : "localhost:27017"
},
{
"_id" : 1,
"host" : "localhost:27018"
},
{
"_id" : 2,
"host" : "localhost:27019"
}
]
}
> rs.initiate(conf)
{
"info" : "Config now saved locally. Should come online in about a minute.",
"ok" : 1
}
ステータスを見てみるとまだStartup的な。
> rs.status()
{
"set" : "replica",
"date" : ISODate("2012-01-24T09:20:22Z"),
"myState" : 2,
"members" : [
{
"_id" : 0,
"name" : "localhost:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"optime" : {
"t" : 1327396814000,
"i" : 1
},
"optimeDate" : ISODate("2012-01-24T09:20:14Z"),
"self" : true
},
{
"_id" : 1,
"name" : "localhost:27018",
"health" : 1,
"state" : 5,
"stateStr" : "STARTUP2",
"uptime" : 8,
"optime" : {
"t" : 0,
"i" : 0
},
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2012-01-24T09:20:22Z"),
"pingMs" : 18677460,
"errmsg" : "initial sync need a member to be primary or secondary to do our initial sync"
},
{
"_id" : 2,
"name" : "localhost:27019",
"health" : 1,
"state" : 5,
"stateStr" : "STARTUP2",
"uptime" : 8,
"optime" : {
"t" : 0,
"i" : 0
},
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2012-01-24T09:20:22Z"),
"pingMs" : 18661632,
"errmsg" : "initial sync need a member to be primary or secondary to do our initial sync"
}
],
"ok" : 1
}
SECONDARY>
もう一回叩いてやると、狙った感じになっています。
SECONDARY> rs.status()
{
"set" : "replica",
"date" : ISODate("2012-01-24T09:22:16Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "localhost:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"optime" : {
"t" : 1327396814000,
"i" : 1
},
"optimeDate" : ISODate("2012-01-24T09:20:14Z"),
"self" : true
},
{
"_id" : 1,
"name" : "localhost:27018",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 122,
"optime" : {
"t" : 1327396814000,
"i" : 1
},
"optimeDate" : ISODate("2012-01-24T09:20:14Z"),
"lastHeartbeat" : ISODate("2012-01-24T09:22:14Z"),
"pingMs" : 68
},
{
"_id" : 2,
"name" : "localhost:27019",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 122,
"optime" : {
"t" : 1327396814000,
"i" : 1
},
"optimeDate" : ISODate("2012-01-24T09:20:14Z"),
"lastHeartbeat" : ISODate("2012-01-24T09:22:14Z"),
"pingMs" : 68
}
],
"ok" : 1
}
PRIMARY>
プライマリにデータを入れて、
PRIMARY> doc = { author: 'joe',
... created : new Date('03/28/2009'),
... title : 'Yet another blog post',
... text : 'Here is the text...',
... tags : [ 'example', 'joe' ],
... comments : [ { author: 'jim', comment: 'I disagree' },
... { author: 'nancy', comment: 'Good post' }
... ]
... }
{
"author" : "joe",
"created" : ISODate("2009-03-27T15:00:00Z"),
"title" : "Yet another blog post",
"text" : "Here is the text...",
"tags" : [
"example",
"joe"
],
"comments" : [
{
"author" : "jim",
"comment" : "I disagree"
},
{
"author" : "nancy",
"comment" : "Good post"
}
]
}
PRIMARY> db.posts.insert(doc);
セカンダリからデータを取得してみます。slaveOkってのをやってやらないと
error: { “$err” : “not master and slaveok=false”, “code” : 13435 }
って出て、データが取得できませんでした。
SECONDARY> rs.slaveOk();
not master and slaveok=false
SECONDARY> db.posts.find( { "comments.author" : "jim" } )
{ "_id" : ObjectId("4f1e79261c0db27b00f1954c"), "author" : "joe", "created" : ISODate("2009-03-27T15:00:00Z"), "title" : "Yet another blog post", "text" : "Here is the text...", "tags" : [ "example", "joe" ], "comments" : [ { "author" : "jim", "comment" : "I disagree" }, { "author" : "nancy", "comment" : "Good post" } ] }
ちなみにrsはhelpのメソッドもあって↓こんな感じに使うんだぜ的な
SECONDARY> rs.help();
rs.status() { replSetGetStatus : 1 } checks repl set status
rs.initiate() { replSetInitiate : null } initiates set with default settings
rs.initiate(cfg) { replSetInitiate : cfg } initiates set with configuration cfg
rs.conf() get the current configuration object from local.system.replset
rs.reconfig(cfg) updates the configuration of a running replica set with cfg (disconnects)
rs.add(hostportstr) add a new member to the set with default attributes (disconnects)
rs.add(membercfgobj) add a new member to the set with extra attributes (disconnects)
rs.addArb(hostportstr) add a new member which is arbiterOnly:true (disconnects)
rs.stepDown([secs]) step down as primary (momentarily) (disconnects)
rs.freeze(secs) make a node ineligible to become primary for the time specified
rs.remove(hostportstr) remove a host from the replica set (disconnects)
rs.slaveOk() shorthand for db.getMongo().setSlaveOk()
db.isMaster() check who is primary
reconfiguration helpers disconnect from the database so the shell will display
an error, even if the command succeeds.
see also http://<mongod_host>:28017/_replSet for additional diagnostic info
↓Web画面から見てもフムフムって感じになっています。
これでプライマリ(27017)落としてセカンダリ(27018か27019)がプライマリになればOKです。
さっそくやってみると、、
・27017のログ
Tue Jan 24 19:02:36 [interruptThread] closeAllFiles() finished Tue Jan 24 19:02:36 [interruptThread] shutdown: removing fs lock... Tue Jan 24 19:02:36 dbexit: really exiting now
・27018のログ
Tue Jan 24 19:02:36 [rsSync] replSet syncThread: 10278 dbclient error communicating with server: localhost:27017
Tue Jan 24 19:02:37 [rsHealthPoll] DBClientCursor::init call() failed
Tue Jan 24 19:02:37 [rsHealthPoll] replSet info localhost:27017 is down (or slow to respond): DBClientBase::findN: transport error: localhost:27017 query: { replSetHeartbeat: "replica", v: 1, pv: 1, checkEmpty: false, from: "localhost:27018" }
Tue Jan 24 19:02:37 [rsHealthPoll] replSet member localhost:27017 is now in state DOWN
Tue Jan 24 19:02:37 [rsMgr] replSet info electSelf 1
Tue Jan 24 19:02:37 [rsMgr] replSet PRIMARY
・27019のログ
Tue Jan 24 19:02:36 [rsHealthPoll] replSet info localhost:27017 is down (or slow to respond): DBClientBase::findN: transport error: localhost:27017 query: { replSetHeartbeat: "replica", v: 1, pv: 1, checkEmpty: false, from: "localhost:27019" }
Tue Jan 24 19:02:36 [rsHealthPoll] replSet member localhost:27017 is now in state DOWN
Tue Jan 24 19:02:36 [rsMgr] not electing self, localhost:27018 would veto
Tue Jan 24 19:02:37 [conn172] replSet info voting yea for localhost:27018 (1)
Tue Jan 24 19:02:38 [rsHealthPoll] replSet member localhost:27018 is now in state PRIMARY
画面からみても↓のように狙った通り。お利口さんです。
27018(プライマリ)にデータを入れて、27019(セカンダリ)からデータを取得してみます。
# ./mongo localhost:27018
MongoDB shell version: 2.0.2
connecting to: localhost:27018/test
PRIMARY> db.things.save({"name":"Shinohara","age":"32"});
↓無事取れました。
# ./mongo localhost:27019
MongoDB shell version: 2.0.2
connecting to: localhost:27019/test
SECONDARY> db.things.find();
error: { "$err" : "not master and slaveok=false", "code" : 13435 }
SECONDARY> rs.slaveOk();
not master and slaveok=false
SECONDARY> db.things.find();
{ "_id" : ObjectId("4f1e837e59ff27bd08cf3187"), "name" : "Shinohara", "age" : "32" }
最後は落とした27017を戻してやって画面から確認したら
セカンダリとしてちゃんと戻ってくれました↓
—
確かにお手軽な感じなんだなぁってのは実際に手を動かしてみるとよくわかりますね。
NoSQLデータベースファーストガイド
posted with amazlet at 12.01.24
佐々木 達也
秀和システム
売り上げランキング: 182936
秀和システム
売り上げランキング: 182936
コメント
[…] CheckMongoDBのReplica SetsをUbuntuで試してみる | shinodogg.comで立てた MongoDBにApacheのログを突っ込みたいわけなんですが、 […]
[…] MongoDBをインストール MongoDBのReplica SetsをUbuntuで試してみる – shinodogg.com – （この記事を書いている時点での最新版2.2.0を使いました） […]