February 2009 Archive

Introducing Memcache As A Service!

Posted by Nathan Kaiser about about 1 year ago...

First Memcache as a Service from Gear6 Now Available!

At Blue Box Group, we are a partner in your business. As such, we continually strive to provide you with the best tools to give your site an edge. Nine months ago we partnered with Gear6 to implement their “memcached”:http://www.gear6.com distribution solution as a new way to scale Rails apps and see better site performance overall. The results seen by early adopters have been stellar.

Memcached is a popular, open-source distributed caching system that has gained tremendous traction in data center environments. Dynamic, high-growth web sites face never-ending challenges to maintain performance, support traffic growth, and offer better and deeper user experiences. With new Web 2.0 requirements, database and app loads can grow to levels that can’t be easily sustained. Memcached is designed to offload your database and application servers and, for this reason, is widely used by popular and dynamic sites like Facebook, Twitter and Flickr to scale out.

While you can get your own copy of memcached and roll with it, we know first hand that it can be challenging to enhance, manage and scale. Gear6 Memcache as a Service has taken the guess work out of using memcached. Gear6 has hardened their memcached distro, added replication and clustering, optimized memory usage for a smaller footprint, included security and networking controls. Because Gear6 Memcache as a Service serves database and application requests from DRAM, web site performance can be boosted by 100x or more, and read-intensive database loads are slashed. This enhanced, highly-efficient, highly-available Gear6 memcached service means that more – or even all – of your objects can reside in cache, further boosting performance, controlling your costs, enhancing your uptime and pleasing your users.

The Gear6 Web Cache service is fully integrated with the Blue Box Group platform, so you can have seamless management and monitoring abilities in a single offering. Give it a try and let us know what you think!

Seattle Start-up Event with nPost

Posted by Nathan Kaiser about about 1 year ago...

One of the things I love about Blue Box Group is our location near Seattle, WA. As an internet geek with a passion for all things electronic, I can’t help but feel at home in one of America’s bastions of technology. Often, I stop at a coffee shop on the way to work and overhear a conversation about Ruby on Rails or as I stand in line at the supermarket the guy behind me rants about Microsoft to his friend. I can’t help but get that warm fuzzy feeling inside. It’s reasons like this that make Seattle a breeding ground for innovative ideas.

Like me, Blue Box Group also has a passion for new technology. We’re always looking for that new and improved feature which makes our customer’s lives just a little bit easier. Plus, being located near Seattle, we get a front row seat for these new companies and their products. Often this means that we hear about the latest and greatest when they’re just starting, and we get to see their growth and development first hand. Our friends over at “nPost”:http://www.npost.com know what we’re talking about. They eat, sleep and breath technology start-ups. By providing great resources like industry experts and interviews with CEO’s from start-ups around the world, nPost helps foster the start-up spirit we all have inside. This is why we here at Blue Box Group are excited to sponsor the next nPost Seattle Start-up Event.

This event, which will be held on March 31st at the Columbia City Theater, seeks to bring together start-ups and everything a start-up could need together in one place. With time to give demos, talk about new start-ups, and to meet, greet and mingle with the VC’s, these events from nPost are great places to hear new ideas and get excited for the future.

If you’re looking to register or you just want more information about the event, you can check out the nPost “site”:http://www.npost.com and their “wiki”:http://www.wiki.npost.com/index.php/NPostWiki:Seattle-event-mar09. We’re excited to get a chance to meet with all of the start-ups that will be in attendance and maybe we’ll even see you there too!

– Joe Clark
Blue Box Group

Scaling up in a hurry: A practical example

Posted by Nathan Kaiser about about 1 year ago...

When I tell people I work in the server hosting business, and that along with Ruby on Rails, we specialize in helping businesses scale their applications from those which see little traffic and sit comfortably on a shared-hosting server, all the way up to sites which see tens of thousands of unique visitors per hour and require large clusters of powerful machines to meet the demand, one of the questions they frequently ask is, “How do you do that?”

There is no simple answer to this question. I usually start by telling people, “It depends on the customer’s application,” which is true, but not very satisfying. A more precise (if no more satisfying) answer would be to say, “We do it by discovering and alleviating bottlenecks.” Doing this generally requires intimate knowledge of the application in question and the underlying subsystems on which it relies– which can vary a lot from application to application. Having a good team of smart, experienced technical people who are quick on their feet always helps. Sometimes we can help stream-line logical processes within an application or adjust configurations so that major hardware upgrades are unnecessary; Sometimes scaling up the hardware is the best solution to the problem. Really, it depends.

Still, we’d like to be able to give people a better idea about how the process might be done from a practical perspective. And since we had the opportunity to do this yesterday for one of our customers, we thought it would be a good opportunity to write about how we did it in this case.

(As a side note, I should say that we generally have a bit more time to go about the scaling up process– we keep trend graphs and otherwise monitor our customers’ machines specifically so that we do have a window of warning to deal with scaling issues before they become critical. Occasionally, though, an event can happen which can throw an otherwise moderate-to-low usage web site into a situation where it sees 10 to 20 times its usual traffic load overnight. It is this exact scenario that we bumped into yesterday.)

Some background on the “normal” state of affairs for this website:

• Moderate site traffic: Generally about 5-6 Mbps of outbound traffic at peak on most days, and less than a thousand unique website visitors in an hour.
• Server has 2 CPU cores, 2GB of RAM, and 160GB of SCSI disk space in a hardware RAID-1
• Website is apache+php, including SSL components and is mainly driven by a Wordpress back-end along with a custom PHP engine.
• The site is primarily an e-commerce site but contains much other general content outside the e-commerce site.

And here’s what happened today:

Our monitoring system first warned us that the site was starting to time-out on web page loads, that database query times were above a slow-query threshold, and the server load was starting to rise rapidly. Other monitoring checks start timing out. Our technicians start to look into the problem, a few minutes into this, the server effectively becomes unresponsive (it’s not down, just taking minutes to execute simple commands). A quick check of the traffic and trend graphs confirm that the machine is seeing a traffic spike unlike anything it’s seen before. Since the site is effectively down anyway, we initiate a reboot of the server to clear out the processing load and allow our technicians access to make changes to the server configuration.

As soon as the server is up, it becomes apparent that the machine is running itself into the ground by spawning too many apache children, thus using too much memory and thrashing swap on the disks. Whenever swap is being actively used on a server, it is always the performance bottleneck. We estimate the amount of RAM each apache child is using and adjust the apache configuration such that it will never spawn so many children that it runs into swap again. We also turn off “KeepAlive” in the server configuration so that apache children aren’t tied up waiting for idle clients to respond. This stabilizes the server so that it doesn’t become unresponsive, and for a while the site handles the load again. However, page load times are being adversely affected by the traffic spike. We start investigating the site code, check to see whether the site itself is under a denial of service attack, and initiate contact with the customer to keep them informed about what we’re seeing and to see whether they’re aware of what’s going on.

As page load times become steadily worse, we get confirmation from the customer that an event has happened which has triggered the sudden increase in traffic, and with that we know the site is not being DOSed. Turning on server-status pages for apache, we’re able to confirm that apache has spawned its configured maximum number of child processes. At this point, the number of apache child processes is the performance bottleneck– however, we can’t increase this without running the server into swap again, so we start looking into alternate solutions. Site traffic is plateauing at about 8 Mbps outbound– about twice the server’s normal load. Most of the traffic appears to be hits to either very specific graphics, or Wordpress pages.

We decide to deploy nginx as a proxy layer in front of apache. Nginx is a very fast, lightweight web server which excels at handling a large number of end-user connections. In our opinion, it is also one of the best web proxies available today. Further, it’s threaded, so it doesn’t usually need to spawn that many children in order to handle a large number of clients. (Technically, apache can run in threaded mode too– which might enable it to return performance numbers that rival nginx’s. However, not all the php modules our customer’s application is using are thread-safe, so that isn’t an option for us.)

Our nginx installation checks to see if the request is a static file on disk (say a style sheet, image, or javascript file). If it is, nginx serves the file out directly. If it is not, nginx then passes the request onto apache. Apache generates the dynamic content and then sends the data back to nginx. Nginx can then spoon feed that content out to users without tying up additional connections within Apache.

After deploying nginx, site load times are much improved, but still not as fast as they should be. Site traffic is now plateauing at around 11 Mbps outbound. Further investigation shows that the number of apache children is still being maxed out in many cases, but not all the time. It turns out the CPU is now 100% busy and has become the bottleneck. And, traffic is still ramping up, so page load times are getting steadily longer.

Looking closer at the application code and the hits the site is receiving, we discover that the Wordpress content is now the largest dynamic module being accessed and the processing of each Wordpress page is causing the CPU load to shoot through the rough. We worked with the customer to quickly install the Super Cache plugin into Wordpress. This plugin allows for creation of a static cache that scales back the number of times the server needs to create certain pages and hopefully alleviates the load on the CPU. The server is now pushing about 15-20 Mbps outbound and it’s still climbing and at an exponential rate as the lunch rush approaches. Page load times are in the 2-5 second range: Acceptable given the circumstances but not good. (And yes– a common trend with many websites we see is a large spike in traffic around 12:00pm PST as presumably a lot of workers around the country and especially apparently on the west coast all go on their lunch breaks and start surfing the web. We kid you not.)

Watching CPU utilization, we notice that while the CPU load has been mostly alleviated, we are still seeing occasional periods of 100% usage. Further, looking at the number of utilized apache children, while on average the usage is somewhat lower, we occasionally still see the spikes of all apache children in use. We posit that although the static cache is helping, the overhead of loading the Apache listeners to read the cache is still enough to be the bottleneck.

So, we alter nginx’s configuration to access the static cache directly where appropriate without loading the wordpress PHP code. This modification involved the implementation of some fancy nginx rewrite rules, as the Super Cache plugin uses a bit of a unique file path for its cache directories. At this point, our performance issues are mostly resolved– page load times are reduced to sub-second intervals. The lunch rush hits and site traffic jumps to 90+ Mbps outbound (it’s also possible that sudden TV publicity of the event which triggered this whole load spike in general has just occurred and people are going online to read about it. This is a pretty typical traffic pattern for TV publicity. But we don’t have confirmation on exactly what caused the sudden jump to 90+ Mbps: It probably wasn’t just the optimizations we’ve made or the lunch rush.) But we’re not quite at the last bottleneck: This server has a 100 Mbps uplink to the internet and we’re closing in on it fast.

So, we configure a gigabit uplink for the server and move it to this link almost instantly. The server itself saw a peak of around 115 Mbps of outbound traffic (sustained over the 5-minute polling interval), and continues to push 20+ Mbps well into the evening. And all this, on just 2GB of memory and 2 CPU cores.

Now, we’re not saying that we can wave our hands and magically make every website suddenly handle 20 times its load, and all that without upgrading the hardware. This was in many ways a special case which lent itself to a proxy / caching solution– and this approach probably wouldn’t have been applicable to the vast majority of the websites we host (because these usually already have a proxy / caching layer in them).

However, what we are saying is that we can help businesses scale their websites effectively. And we’re the people to call on if suddenly you’re seeing traffic many times what you ever expected.

I should also mention that in handling today’s load spike crisis for this customer, we didn’t need to pull out some of the more versatile tools in our toolbox. Some of these include:

• Virtualization: With this, we have lot more flexibility in what we can do, from quickly and easily cloning an existing server, to doing live-migrations of the server to more powerful hardware, to dynamically scaling the capabilities of any server up or down depending on its needs– all with little to no down time of your application.
• Load balancing: When you can’t squeeze any more juice out of a single machine, the ultimate solution to the ramping-up problem is to scale horizontally. And running behind our capable F5 hardware load balancers makes adding and removing servers from your application cluster seamless.
• Content Delivery Network (CDN): Blue Box Group is in final beta testing of a CDN partnership with Internap in order to provide CDN services to our customers. If a significant portion of your website content can be cached (as is usually the case with most web applications we encounter), then it makes a lot of sense to off-load the work of serving it to a large array of capable caching servers distributed all over the globe. In this scenario above, being able to use our CDN to cache both images and Wordpress content from the get go should have prevented the site from ever really seeing an outage.
  Anyway, there you have it: An example of one customer that needed to scale up quickly. And didn’t even have to contact us because we were already on the problem. Really, this is the sort of thing we get to do all the time. Which is why I love working at Blue Box Group so much. –Stephen Balukoff Blue Box Group (Additional content contributed by Jesse Proudman)

JSConf 2009

Posted by Nathan Kaiser about about 1 year ago...

It’s probably a little strange seeing a post about a JavaScript conference come from a Ruby on Rails host. It also may surprise you, but Blue Box Group has customers running applications not only written in Rails, but in Java, PHP and many other languages as well. Regardless of which language the primary application has been developed in, JavaScript provides that extra pop we’re all looking for in our apps. Since JavaScript is beneficial to anyone in the web development community and many of our customers utilize it across their sites, we wanted to share some information about an upcoming conference organized by one of our customers with you.

JSConf 2009 is the first conference dedicated to JavaScript developers in all forms and it is shaping up to be one amazing conference. The conference brings together some of the greatest visionaries and developers within the JS community including John Resig creator of jQuery, Jason Huggins creator of Selenium testing environment, and Stefan Stoyanov from YUI. This is an incredible opportunity for anyone doing web based development since the conference material will show you how to use JavaScript in the browser, on the server, and on mobile handsets.

For anyone currently hosting their mobile apps with us or for those of you who’ve been considering writing a mobile application, this conference provides a wonderful opportunity to learn from some of the best in the business. With platforms like the iPhone and Android becoming more and more popular, now is the time to go mobile. Check out the conference and Blue Box Group’s Mobile Application Hosting to jump start your mobile app today!

If you are interested in going, we recommend getting your tickets soon at http://jsconf2009.eventbrite.com as they are selling very quickly. For more information or other speakers, check the website at http://www.jsconf2009.com

Affordable Content Distribution Network Services

Posted by Nathan Kaiser about about 1 year ago...

Blue Box Group is happy to announce it’s newest addition to our superb line-up of Rails hosting services: Bundled Content Delivery Network Services. We’re proud to offer services and features not found at other providers and designed to make your business operate more smoothly and effectively. We have historically been at the forefront of Rails hosting technology (Carbon Neutral Web Hosting, Ruby on Rails Hosting Service Level Agreement, IPv6 Services) and now we’re one of the first to make a Content Delivery Network an affordable part of every day rails hosting.

So what is a CDN and why does it benefit your business? More than likely, your site has thousands of visitors not only from the US, but from across the world. By caching your site’s static content on both our national and global network of caching servers, your visitors will experience faster load times regardless of location. Powered by Internap’s powerful network backbone, the Blue Box Group CDN will automatically determine the closest content server to each visitor and serve your content from that point. Since electronic information is transmitted at roughly the speed of light (think fiber optics), placing your largest data closer to your end users is the most fundamental way to speed up your site.

Increasing the speed of your site isn’t the only benefit of the Blue Box Group CDN. In addition to bringing your largest files closer to your visitors (typically videos, images, stylesheets and javascripts), many clients will see their server loads remain stable paired with an INCREASE in traffic. Curious how this happens? As your site’s static content gets cached and redistributed around the globe, the CDN begins to serve files from its cache instead of directly from your site. Any request for a piece of content that already exists on the CDN will now be served from the closest server to your customer as opposed to your site’s main server. All of the requests processed by the CDN are requests that your hardware doesn’t have to worry about, freeing up system resources for more important non-cached requests. By moving the burden of serving the hundreds of media files your site has to external hardware, your site will find itself with an increased ability to handle site traffic while seeing a comparably lower server load.

A client of Blue Box Group located in Australia recently decided to utilize the CDN with their content heavy site which had visitors from all over the world. Before utilizing the CDN, this client was experiencing slower response times on some of its pages due to the long distance those images must travel (Washington State to Australia). After configuring their site to use the CDN (less than a few hours work), we saw some pretty exciting results:

As you can see from the above graphs, not only did our client see a dramatic reduction in ping response times to the asset servers after implementing the CDN, but the total amount of traffic that the site was able to serve went up as well. This was largely due to the CDN improving usage of server resources by keeping server resources available for dynamic content. Any site will see some improvement in speed and reliability after utilizing the CDN, plus our platform extremely easy to get started on!

Content Distribution Networks are used by some of the largest internet content providers on the web today: Google, Amazon, Apple and more! Now your site can utilize the same technologies as these internet superstars, at a price anyone can afford. With minimal startup fees and pricing inline with traditional bandwidth rates, there really is no reason to avoid using this type of platform. It’s possible to implement not only for large enterprise clusters, but for shared hosting and virtual and dedicated platforms as well. To get more information about this great opportunity to improve your site’s functionality, give us a call today!

-—Joe Clark
Blue Box Group

VPS Blow-out Coming to an End?

Posted by Nathan Kaiser about about 1 year ago...

So… you’re one of those “procrastinator” types, aren’t ya? You hear about a great new service or an upgrade to your favorite product and what do you do? You wait.

If this sounds like you then pay close attention to what I’m about to tell you. The Blue Box Group VPS sale isn’t going to last much longer. That’s right, the sale you’ve been following for the past month but have put off acting on is coming to an end. How much longer do you have? Well, not long. The VPS sale, offering 40% discounts on any new virtual server purchased from Blue Box Group, is going to conclude at the end of the month. So, don’t miss out on this great deal! If you haven’t already, checkout our “VPS offerings”:/services/virtual_servers today and be sure to get yours before it’s too late!

Passenger / Mod_rails Community Sponsorship Campaign

Posted by Nathan Kaiser about about 1 year ago...

The great team over at Phusion who make Passenger (aka mod_rails) have done an amazing job on the latest round of mod_rails updates. They’re about ready to release version 2.1 into production form which has some great updates, but they are asking for community support. Here’s there post from the passenger mailing list:

Blue Box Group has been very satisfied with passenger and so we feel it to be important to support the project. We were initial donors during their first campaign, and we’re happy to donate again. This time, we’ve contributed $750 to the campaign, and we encourage our customers to donate as well.

We can’t wait for 2.1 to be released!

• Jesse Proudman
  

Blue Box Group