Menu

Creating a Mongo replicaset using docker: Mongo replicaset + Nodejs + Docker Compose

9 Comments



In this video I am going to create a simple
mongo replica set in docker for this I am going to instantiate three mongo containers,
a container for setting up the replica set, another one for adminmongo and a last one
for a express application, a modified version of the one I created for the last video that
connects to the replica set. Finally I am going to experiment a little
bit with the replica set. A replica set is a group of mongod instances
that maintain the same data set for providing redundancy and high availability. When a replica set is configured, a Primary
is elected while the other instances will remain as secondaries. The primary will be the instance to which
you can send write and by default read requests and the secondaries will be kept in sync with
the primary. Whenever a Primary is down a secondary can
start the process of becoming a primary so that the service is restored. Let’s start with our three mongo containers,
for this I am going to use mongo’s official container called, perhaps not surprisingly,
mongo. Let’s create a mongo configuration file
called mongo.conf and then add the rs0 as the replica set name and 1024 as the oplog size
Let’s create a folder for the first mongo container, for this one I am going to create
a simple Dockerfile that uses the base mongo image mongo, I am going to define /usr/src/configs
as the default working directory, copy the mongo.conf into the container using the copy
instruction, expose the default port for a mongo database and finally just add a cmd
instruction with the configuration file so that mongo uses it when starting up. Let’s add a data subfolder, this one will
be the folder for our volume, which will allow us to keep the data even when we stop the
mongo container. Let’s also create a folder for the other
two containers each with its own data subfolder This is a good moment to create the docker-compose.yml
file, define the version at the top and add our three services, the first one is going
to point to the dockerfile inside the mongo-rs0-1 folder, expose locally the same port as it
does inside the container, map the volume from our first data subfolder to “/data/db”
which is the database folder inside the container. For the second and third containers, given
that it is such a simple configuration let’s just use the base image without creating a
custom Dockerfile. Let’s define the image that we are using,
in this case “mongo”, the same configurations that we defined for the replica set and the
oplog on the first container and let’s add one to the mapped port on the local machine
so we can use the next available one. Let’s just copy and paste it for the third
container and do the same for the mapped port. Until now we have only created the necessary
configurations for getting up and running our mongo containers. A replica set has to be explicitly configured
before it can be used, for that I am going to create another container. This container will be short lived, but it
will wait until our three mongo databases are up and ready and then it will initialize
the replica set for us. Let’s create a folder for the setup container
and then a script file that we can execute remotely using the mongo shell, in this file
I will create an object for the configuration with “rs0” as the id and we will add the
three members for this replicaset using the names for the services that we defined in
the docker-compose file, then initiate and show the configurations for the replica set. Create a new file called setup.sh that will
wait for a couple of seconds until hopefully the three mongo containers are fully up and
then will connect to the first container using the mongo shell and execute the script we
created earlier for initiating the replica set. Let’s create a dockerfile for this container,
we will need to also use the mongo image as a base so we can use the mongo shell, as I
did for the first image, I will set the workdir as /usr/src/configs, copy the replica Set.js
and setup.sh into the working directory using the copy instruction, and finally I will instruct
docker to call the setup.sh file using the command instruction. Go back to the docker-compose.yml file and
add this container, I will use the name setup-rs and will define this to be also the name for
the image that will get created, define the folder for the dockerfile and will define
that this service requires that the first mongo container is up and running, I will
also add to that first container a dependency on the other two mongo containers just to
make sure that they start before everything else. Finally I am going to add an adminmongo container
so we can explore our new replica set without problems. For that I will set the name of the service
as adminmongo, add the image to mrvautin/adminmongo, map the local port 1234 to the same one exposed
by the adminmongo container… and… that’s it. Now let’s play a little bit with our newly
created replica set, first let’s only start the mongo instances
and adminmongo In another window let’s query for the current
state of the replica set, first let’s see which containers are up
Let’s use the first container to query the replica set for its status Let’s use docker exec mongo replica set and, well, the name for that container bash -c ‘mongo –eval “rs.status();”‘ As we can see here the instances are waiting
for a replica set configuration and given that none of the instances is set to be a
primary, we won’t be able to do anything yet. Let’s now start all of our services and
wait for the setup container to run Query again for the status of the replica
set as we did before Now we can see a different result, the replica
set has been configured and if we go through the members of the replica set we will see
that in my case it is the first instance the one that was elected primary Before we continue if you are not sure what’s
the ip for your containers just type docker-machine ip and it will give you the ip that you can use
for your containers in my case it is 192.168.99.100 Let’s configure the connection for the three
mongo containers in adminmongo, open the ip address for your container plus the 1234 port
in a browser And add one connection per mongodb instance
using the name for the service in the docker compose file Let’s try adding a new database and adding
a collection and some documents to the primary, add a new document, add an entry… ok it works!!! If we try to simply query or do anything in
any mongo container but the primary it will complaint and will not allow us to do it that
easily Let’s just be mean and bring down the primary, let’s use docker stop and the name of the container After a couple of seconds a new primary will
be elected, let’s find out which one, by querying for the status again. But using this time the second container
If we scroll a little bit we will find that in my case the third instance was the one
elected primary this time let’s query that instance in adminmongo,
ok, now we can see everything there Let’s add a nodejs + express app and connect
to the replica set, for this I am going to use the application created for the previous
video, let’s copy the folder here and make some little changes. First on the database.js file I changed the
connection string so it has the list of mongo instances for the replica set then I added
a disconnect call before trying to connect to the db so that in case we were already
connected we could easily reset the connection And then from the index.js I removed the db
initialization from the beginning of the file and added it to the first route so it was
easier to reestablish a connection to the db if needed Go into the docker-compose file and add a
service for our test app, let’s call the service web-app, set a name for for the image,
the folder in which the dockerfile file is so an image can be built from it, map the
3000 port, set the volumes the same way we did in the previous video and finally set
a dependency to the setup-rs container which in turn will end up depending on everything
starting up. The dependency won’t give us any assurances
into the state of the replica set but at least we know that the configuration is in progress. Let’s docker-compose up our containers and
give the setup container a couple of seconds to configure the replica set as we did before
Now let’s open the ip address for your container plus 3000 for the port so that we can initialize
our db connection, then let’s add to the url /testFind to try to query the replica
set, nice!!! It works!! Awesome! Congratulations and thanks for watching!

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

9 thoughts on “Creating a Mongo replicaset using docker: Mongo replicaset + Nodejs + Docker Compose”

  1. RGPankO says:

    Moooree!! Please! More!
    It's insane how well you explain things and how well you give visual feedback when you refer different parts of the code.
    Thanks for the hardwork on these clips, please stop doing whatever you are doing and record a few more 🙂

  2. TuncaySonmez says:

    Thanks for the great tutor.

  3. Leonardo Bascope says:

    I wish I had seen this before… Great video!! Thank you!

  4. Bhargav Patel says:

    I went through with this tutorial.. getting below exception while running setup.sh script

    Building setup-rs
    Traceback (most recent call last):
    File "site-packages/docker/utils/build.py", line 96, in create_archive
    PermissionError: [Errno 13] Permission denied: '/home/user/Desktop/mongo-rs0-2/data/diagnostic.data/metrics.interim'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File "bin/docker-compose", line 6, in <module>
    File "compose/cli/main.py", line 71, in main
    File "compose/cli/main.py", line 127, in perform_command
    File "compose/cli/main.py", line 1052, in up
    File "compose/cli/main.py", line 1048, in up
    File "compose/project.py", line 471, in up
    File "compose/service.py", line 352, in ensure_image_exists
    File "compose/service.py", line 1067, in build
    File "site-packages/docker/api/build.py", line 154, in build
    File "site-packages/docker/utils/build.py", line 31, in tar
    File "site-packages/docker/utils/build.py", line 100, in create_archive
    OSError: Can not read file in context: /home/user/Desktop/mongo-rs0-2/data/diagnostic.data/metrics.interim
    [12690] Failed to execute script docker-compose

  5. Just Watch says:

    setup-rs_1 | /usr/local/bin/docker-entrypoint.sh: /usr/src/configs/setup.sh: /bin/bash^M: bad interpreter: No such file or directory
    setup-rs_1 | /usr/local/bin/docker-entrypoint.sh: line 354: /usr/src/configs/setup.sh: Success

  6. Luuk th says:

    Great video, excellent explanation!!

  7. Tara Gurung says:

    Great Post, can you also make a video on Incremental backup and restoration.

  8. Jon L says:

    Great tutorial!! Thank you for the knowledge!!
    I have a question about that. When i make a new docker-compose and i run the containers other time, I have an error. You can see the next in the terminal:

    setup-rs_1 | MongoDB shell version v4.0.10
    setup-rs_1 | connecting to: mongodb://mongo-rs0-1:27017/?gssapiServiceName=mongodb
    setup-rs_1 | 2019-07-25T18:10:35.648+0000 E QUERY [js] Error: couldn't connect to server mongo-rs0-1:27017, connection attempt failed: SocketException: Error connecting to mongo-rs0-1:27017 (172.27.0.5:27017) :: caused by :: Connection refused :
    mongodb-replicaset_setup-rs_1 exited with code 1.

    After that i could see that my database doesn't exist. Could you help me please?

    Thank you for all and we are waiting for more tutorials like this.

  9. Deepal Jayasekara says:

    This is super helpful! Short and Sweet. Thanks.

Leave a Reply

Your email address will not be published. Required fields are marked *