amazon ec2 - How good is this MongoDB/EC2 setup?

itemprop="text">

Hi, />I wondered what was a good setup for AWS/MongoDB in terms of machines and the sizes
of their disks.

Current
setup

• 3
  micro machines for the config servers, 1 mongos and arbiters. The 8Gb limit is almost
  reached. (and I ran the arbiters with --nojournal)


• per
  shard : a replica set of 2 machines m1.large with 8Gb for system + 20Gb for
  data


• everything is on
  EBS.

Questions

1. is
   20Gb too big or too small ? Should I go with 100Gb for example
   ?


2. Am I supposed to inform mongodb about the 20Gb (or
   other) disk limit ?



3. Do you see anything wrong
   that I don't see ? Im new to mongodb and aws but Im an ok experienced
   SWE

Plan of
use

My database should allow a 100
qps (mostly writes), and should grow up to 1Tb over the next 3 years. The plan is to add
as many shards as needed, more or less manually (with scripts), when we see that more
memory is needed on the database.

We will also
run a few mapreduce over this and have some scripts that do aggregates with the data
over the past 15 minutes, every 15 minutes.

We
are a very small company, spending up to a few hundred $ per month on our servers would
be ok but we can't go crazy on
cash.

We hope that we won't have to
manually take care of too many machine failures, manually taking care of things once a
month would be fine.

Thanks for telling me what
you think about
that.

Thomas

Answer

First your specific
questions:

is 20Gb too big or too small ? Should I go with 100Gb for example
?

This completely
depends on your data requirements and how many documents you intend to insert. If you
intend to have 5GB of documents then you should be fine, even with overheads for
replication (oplog is 5% of free space) and storage (there is always an empty file
pre-allocated for each database). If you plan to have 10-12GB of data (and remember you
have to store indexes, journal, logs as well) then I would go for a larger disk.

Since you say you plan to grow to 1TB in a year
then you will probably exceed 20GB inside a month and need to increase disk anyway,
hence it will probably be easier to go for 100GB immediately. At 1TB in a year, assuming
constant growth, that will only give you about 1 month of room (1TB per year ~= 83GB per
month).

Am I
supposed to inform mongodb about the 20Gb (or other) disk limit
?

No,
there have been improvements in how it handles the situation but MongoDB will
currently just use all available space until there is none left - you need to monitor
your disk space independently.

Do you see
anything wrong that I don't see ? Im new to mongodb and aws but Im an ok experienced
SWE

Never use
micro instances for anything in production - in particular do not use them for config
servers. Your config servers are critical for the operation of a sharded cluster. But,
no need to take my word for it - see page 6 of the href="http://info.10gen.com/rs/10gen/images/AWS_NoSQL_MongoDB.pdf" rel="nofollow
noreferrer">updated Amazon
whitepaper:

T1.micro instances are not recommended for production MongoDB deployments,
including arbiters, config servers, and mongos shard
managers.

Generally
I would recommend reading through the whitepaper and following the guidelines therein -
you'll find recommendations for Linux settings (readahead, hugepages etc.), storage,
pIOPS and more. Also worth checking out are the href="http://docs.mongodb.org/manual/administration/production-notes/#production-notes"
rel="nofollow noreferrer">Production Notes - some duplication, but it's
updated more often than a whitepaper.

Finally,
get some idea of your href="http://docs.mongodb.org/manual/faq/storage/#what-is-the-working-set" rel="nofollow
noreferrer">working set size for your database (per shard) - that will
dictate how much RAM you need, which is really the key to selecting instance size on EC2
for MongoDB. You may have enough with 8GB, but if not you will see significant
performance hits for hitting disk.