Long live your app: MongoDB Replication
The scenario
Not so long ago a customer requested that we provide the highest server reliability that we could. The application was pretty big, with about seven servers per environment. We needed to increase reliability in web servers and databases. In this article, I'll explain how we achieved that from the database side.
Configuring MongoDB
First, it is important to notice one thing, as the name states, MondoDB's Replication will be copying your database ‘n’ times (once for each replica member). With that said, to obtain the maximum benefit from this practice you will want to have your replica set distributed in N different physical servers (i.e. Ubuntu servers). We'll explain how this works below.Having N servers set and ready only to practice could be expensive and time-consuming. Therefore, we’ll be performing all of the examples on this article in one single server –your computer– to allocate all of the replica’s members. And we will be creating different folders and pretend that each one of them is a path to a different physical server. So let’s create folders that will be hosting each replica’s member. In our case, our DB's path is under ‘/usr/local/var/mongodb/’, so we’ll create three folders (a replica needs at least three members) under that route:mkdir -p /usr/local/var/mongodb/rs0-0 /usr/local/var/mongodb/rs0-1 /usr/local/var/mongodb/rs0-2
If you don’t have your database’s path, you can get it by running this in MongoDB's terminal:db.serverCmdLineOpts()
Within the output command above, locate "dbpath" inside "parsed".Example:> db.serverCmdLineOpts()
{
"argv" : [
"/usr/local/opt/mongodb/mongod",
"run",
"--config",
"/usr/local/etc/mongod.conf"
],
"parsed" : {
"bind_ip" : "127.0.0.1",
"command" : [
"run"
],
"config" : "/usr/local/etc/mongod.conf",
"dbpath" : "/usr/local/var/mongodb",
"logappend" : "true",
"logpath" : "/usr/local/var/log/mongodb/mongo.log"
},
"ok" :
}With our fictitious servers ready, the next step is to create three "mongod" clients (one for each replica set member).Note: you need to leave each client running, so you might want to run them in background by appending an ‘&’ at the end of each command. If you need to bring those commands to the foreground, then execute ‘fg’ in the corresponding window where your background jobs are running. Here is an example of how to run mongod clients:mongod --port 27018 --dbpath /usr/local/var/mongodb/rs0-0 --replSet rs0
mongod --port 27019 --dbpath /usr/local/var/mongodb/rs0-1 --replSet rs0
mongod --port 27020 --dbpath /usr/local/var/mongodb/rs0-2 --replSet rs0
Explaining commands:
mongod: executes mongodb.--port: indicates the port that the mongod client will be listening to.--dbpath: specifies the path to your application’s database. In our case, this is pointing to the directories previously created.--replSet: sets the name of the replica set.Now you can connect to each mongod client to start configuring the replica set!
Configuring your Replica Set
To configure your replica set, first you need to be connected to one client. You do it by executing the ‘mongo’ command in your terminal and append the --port option to indicate which client you want to be connecting to. In our case, we will be connecting to our client with port 27018:mongo --port 27018
Once connected, we can start configuring our replica set.
Defining Replica Set Configuration
Configuring the replica set is pretty simple and straightforward, all we need is to set some basic parameters. Go to the mongo shell you just opened and type:rsconf = {
_id: 'rs0',
members: [
{
_id: 0,
host: 'localhost:27018'
},
{
_id: 1,
host: 'localhost:27019'
},
{
_id: 2,
host: 'localhost:27020'
}
]
}
Explaining options:
rsconf: the name of the object or variable that will contain the replica set configuration._id: sets the name of the replica set. This needs to match the --replSet option that you set when executing the mongod client. In our case: rs0.members: sets all the elements that will be conforming our replica set. Elements should be contained within and array following this format:member's _id: the element's identifierhost: the ip address of the server that is hosting the element. It has to be followed by the port where the mongod client is listening to.Now, apply the configuration to the replica set. In the Mongo shell, type the following:rs.initiate(rsconf)
If everything went OK, you should see this:{
"info" : "Config now saved locally. Should come online in about a minute.",
"ok" : 1
}
You can see additional information by executing rs.isMaster().At this point, your replica set is up and running!So you might be thinking now, "so far so good, but I haven’t seen any real advantages yet...”Let us explain a bit more. What makes MongoDB Replication a good strategy to increase your app’s reliability is the fact that your database will be always available for your web application. You might be wondering ‘How is that possible?’ As you might be guessing, having your database replicated N times gives you the opportunity to prevent data loss and sustain a persistent connection between your database and your web app.MongoDB Replication consists on having a database replicated N times, where one element is the Master/Primary (this is the only one interacting directly with your app) and the rest of them are Slaves/Secondaries (these are interacting with the master).This is how it works: when the master element dies, MongoDB automatically chooses a slave element and turns it into a master that will be accepting all IO operations. Your database will be always ‘alive’.
Browsing Secondary elements
You can see your replica set slave elements by logging into a mongo shell for each one of them and typing rs.isMaster(). Don’t forget to set the --port option like this:mongo --port 27019
mongo --port 27020Once you are in the mongo shell, the first indicator that you will see is:rs0:SECONDARY>
Then, after you executed rs.isMaster(), you should see something like:{
"setName" : "rs0",
"ismaster" : false,
"secondary" : true,
"hosts" : [
"localhost:27020",
"localhost:27019",
"localhost:27018"
],
"primary" : "localhost:27018",
"me" : "localhost:27020",
"maxBsonObjectSize" : 16777216,
"maxMessageSizeBytes" : 48000000,
"localTime" : ISODate("2015-03-25T15:40:49.983Z"),
"ok" : 1
}
Where "ismaster" and "secondary" will tell you that that specific client is an slave element. Also, you can see that "primary" option is indicating the IP address of the element that is the master.And that’s it! You are done configuring MongoDB! Continue next to see how to configure your Rails app to make it work with the replica set.
Configuring the Rails application
If you do not have a Rails application created go ahead and create a new one. Since we are not going to use ActiveRecord, you can append --skip-active-record to the "rails new" command to skip ActiveRecord’s installation and configuration.Once you have the new Rails app, you will need to include Mongoid in your Gemfile and execute "bundle install".Execute "rails g mongoid:config". This will create config/mongoid.yml where you will have to add the configuration for your app’s db. The configuration of this file is a little bit different than usual, so here you have an example of how it would look for our exercise:development:
sessions:
default:
database: mongodbreplicaset_development
hosts:
- localhost:27018
- localhost:27019
- localhost:27020As you can notice, the three hosts are the MongoDB clients we created at the beginning; make sure they are still up and running.After configuring mongoid.yml, go ahead and create a controller, a model and some views. You can use "rails g scaffold" to speed up the process ;). In this case, we are going to create the basic CRUD for users.Now go ahead and start the Rails server and add some information in your app.
Checking that MongoDB Replication is working.
Now we are ready to test our replica set, we need to log into the three mongod clients:mongo --port 27018
mongo --port 27019
mongo --port 27020Inside the client with port 27018 (the master) check that the database you set inside mongoid.yml was created:show dbs;
You should see the database you used in your mongoid.yml, in our case we have this:mongodbreplicaset_development 0.0625GB
Then select your database, in our case:use mongodbreplicaset_development;
And check that the information you captured in your web app is there:db.users.find({})
You should see the data you entered through your web application, in our case:{ "_id" : ObjectId("5512e50d4d61638eb1000000"), "name" : “John", "age" : 30 }
Now go to the slave elements and repeat the process; you should see the same info that is persisting in your master element. Note: when trying to execute "db.users.find({})" you will receive a restriction error message, that’s normal, go ahead and execute:rs.slaveOk()
NOTE: By default MongoDB protects all the info from being read, so with rs.slaveOk() you are telling MongoDB it is OK for an external user to have access to stored data in the replica set.
Testing database reliability: kill the master
Let's put al of this to a test. Go to your mongod clients and identify the master. If you don’t remember which one is your master, login into a mongo shell of any of the mongod clients and execute "rs.isMaster();". In the output, look for the field “primary” to get the IP address of the member that is currently the master/primary.Our master element is the client running in the port 27018, so we will kill it by executing the following:ps -Af | grep mongod -> in the output look for the process’ id and copy it.
kill -9 process_idNow, go back to a Mongo shell and execute rs.isMaster(). In the output, you should see under "primary" that a new member replaced the one that is dead.
Is my application still working?
Go to your browser and enter new information. Everything should be smooth and quiet as usual, your web app is working correctly.
Reviving the dead member
Once you know your web app is still up and running, then go back and execute the mongod client you just killed in the previous step, in our case:mongod --port 27018 --dbpath /usr/local/var/mongodb/rs0-0 --replSet rs0 and log into a Mongo shell for that client:mongo --port 27018
As you may notice, this member is not longer the master; another member has taken its place and has let it with a secondary role. If you want to know which is the new primary you can use rs.isMaster() again and identify who is the new master.But what happened with the info save while this member was gone?Let’s look for it. Connect to the member you just revived and execute next:use mongodbreplicaset_development;
rs.slaveOk();
db.users.find({})You will see all the information that was added through your web app when this replica member was down:{ "_id" : ObjectId("5512e50d4d61638eb1000000"), "name" : "John", "age" : 30 }
{ "_id" : ObjectId("5512eec44d616391fd000000"), "name" : "Marie", "age" : 29 }How cool is this?!
Reconfiguring MongoDB Replication
You might have noticed that even if you kill all mongod clients, the next time you execute them the replica set will still be there. Why is this? Do you remember the output when we executed the command rs.initiate(rsconf)? It read something like:"info" : "Config now saved locally. Should come online in about a minute."
Yes, the configuration was saved! Therefore, every time you execute mongod clients using the --replSet and the same replica set you used before, then that configuration will be applied for that replica set.Changing, removing, or adding a replica set member is not too hard. Connect to a mongo shell for the primary mongod client, and set a new newConfig object containing all new members, and then apply those changes with:rs.reconfig(newConfig)That’s it! Now you know all the basics of increasing the reliability of your web application by setting up MongoDB Replication!
Conclusion
A downside to MongoDB replication is that it does not matter if your database is replicated N times on N servers, it is still being centralized. Which means that you could bump into networking performance issues if you do not provide your server with enough resources (RAM, processor, hard drive space) to handle your data.But! Even in spite of the issue mentioned above, we still believe that MongoDB Replication is a good option when you need a server that is reliable, easy to configure, and fast to set up. If you want to learn more about this technique and squeeze all the power of MongoDB Replication, we recommend you read the official documentation to see all the configuration options you can use. For instance, you could set a hidden replica member to create a backup file of your database and upload it to an AWS S3 bucket!Thanks for reading! Let us know what you think in the comments.M.