Report on Compute and Storage Clouds
As a starting point we need to include at least the following offerings:
- Amazon S3 and EC2
- EngineYard
- google file system
A thorough executive brief that covers at least the following elements:
- Reliability ... do they provide performance guarantees?
- Scaling ... what is required to add/remove resources? what communication overhead is involved across nodes in the cluster/cloud?
- Cost
- Flexibility
- Openness ... is the architecture an open standard? how locked in to the technology would we be?
- Risk ... how financially sound is the company? how many providers of the service are there? could we jump to a different provider if needed?
- Technology ... what is the gist of the architecture?
Contents
Usage Stories
- We're having some sort of character set problem with our database ... say we want to write a little program to assess the extent of the problem ... and assume that are database is all in s3 ... and is 25 times bigger than it is now ... so we write our little program and run it on as many computers as need be to get some statistics in 15 minutes
- We have some special entry published on the wiki and our traffic doubles in a day. We want to add new resources to the system to incorporate this load change.
- We have four time as many requests per seconds as we have currently in the system. What would be the response time of the storage for every request.
- Time required to retrieve a wiki page out of an external storage. How would this time vary if we use some internal storage.
- Bandwidth consumed for retrieving all the requests in a day.
AcceptanceTest
- Report includes a thorough comparison on each of the dimensions mentioned above
- Report includes recommendations for next steps
- Report garners the approval of Ward and Brandon
Hadoop (HDFS and MapReduce)
- Reliability
- The distributed file system maintains replicas of stored data.
- A single master servers holds the file system metadata. Potential single point of failure. A workaround could be to snapshot master's state periodically and recover in case of failure.
- Scalability
- Yahoo is running a Hadoop cluster of 900 nodes which houses 3 million files (150TB of data).
- Last.fm has deployed a 25 node cluster with 1TB data per node.
- Cost
- Hadoop is open source and free of cost.
- Hadoop can run on commodity machines. No need to buy high end servers.
- Maintenance costs can be potential problem.
- Openness
- Open source.
- API to access DFS and run MapReduce programs available only in Java.
- C and C++ APIs in the work.
- Parallel computation must be programmed using the MapReduce style library. Some problems don't fit the paradigm.
- Risk
- Hadoop is supported by Yahoo Inc.
- Hadoop in production user is still a question mark.
- Technology
- Hadoop is a library which provides a Scatter-Gather style parallel processing facility. It also provides a distributed file system.
- Follows the same paradigm as Google's MapReduce system.
- It is written in Java
- References
Amazon S3 and EC2
- Reliability
- S3 ensures 99.99% availability.
- S# streams the data to multiple storage servers in multiple datacenters to ensure that the data remains available in case of network/server failures.
- Amazon claims that any performance bottleneck can be fixed by simply adding nodes to the system.
- Cost(in dollars)
- Storage 0.15 per GB-Month of storage used
- Data Transfer
- 0.10 per GB - all data transfer in
- 0.18 per GB - first 10 TB / month data transfer out
- 0.16 per GB - next 40 TB / month data transfer out
- 0.13 per GB - data transfer out / month over 50 TB
- Requests
- 0.01 per 1,000 PUT or LIST requests
- 0.01 per 10,000 GET and all other requests
- Instances of ec2
- 0.10 per instance-hour consumed (or part of an hour consumed)
- Data transfer between between s3 and ec2 is free.
- Openness
- Requests to access data can be made using REST and SOAP protocol.
- The API is available in Java, Perl, PHP, C#, Python and Ruby.
- Internal details of how the data is stored on amazon servers is hidden.
- Risk
- Amazon is using s3 as a storage service for its own web services.
- S3 is also used by Smartsheet.com, G.ho.st, 37signals, Webmail.us, YouOS, SmugMug, Altexa, ElephantDrive, Jungle Disk, MediaSilo, MyOwnDB, adaptiveblue
- Technology
- Amazon S3 is a simple key-based object store.
- Object size can vary from 1 byte – 5GB and one can have unlimited number of objects.
- Objects can of any type.
- References
- Statistics
- Time required to upload 36MB file = 25mins
- Time required to download 36MB file = 11mins
- Time required to download an object of size 21K = 2 seconds 126768 microseconds, 1 seconds 941934 microseconds, 1 seconds 516516 microseconds, 1 seconds 570831 microseconds
- Time required to upload an object of size = 1 seconds 608922 microseconds, 1 seconds 916495 microseconds, 1 seconds 660754 microseconds, 1 seconds 570831 microseconds
EngineYard
- Reliability
- EngineYard lets you add redundant slices(load balancers, servers, disks) so that you dont have any single point of failure.
- The database is replicated/mirrored.
- They take offsite backup of data every 5 hours.
- Scalability
- EngineYard lets you run your application in a cluster and you can add as many slices as you want to. So application can be scaled up by adding more number of slices.
- You can add multiple slices to balance the load among a set of virtual servers that they provide.
- Openness
- They are focused on hosting rails applications.
- Risk
- Their customers include Idealab (www.idealab.com), Kongregate (www.kongregate.com), Bonnaroo Music Festival (www.bonnaroo.com), Confabb (www.confabb.com)
- Technology
- Each VPS or slice runs ruby, rails, your gems, mongrel_cluster, load balancer to balance between mongrels.
- They store data on Coraid SAN and the mount the slice storage space as a GFS mount of the SAN.
- References
