Proposing a new HTTP request header: max_timeout

I'm certainly not going to confess that my sites sometimes have technical problems. But a... friend...of mine tells me that servers can get bolloxed up. With a site like OvationTix (er, I mean: OshmationShmix), the most likely source of a site-wide problem will be congestion in the database. In such instances, the site will not be completely inaccessible, but it will be seriously slow. Users will say "the site's down," while engineers will say, "it's not down, exactly..." Situations like this can be frustrating, and become especially problematic when dealing with SLAs -- if pages that usually load in 2 seconds are instead loading in 2 minutes, is that "downtime," requiring remediation according to an SLA? I've seen some contracts stipulating that all pages must respond in under 3 seconds. That's good, but still too coarse-grained; what about a page (like a big report) which typically takes 20 seconds -- and which users expect will take a long time?

For this and other situations, I'm proposing a new HTTP request header: max_timeout (in milliseconds). After the timeout is reached, the browser will treat it as failed and inform the user accordingly. For users, this resolves ambiguity: currently, if a page loads slower than they're used to, they don't know whether it's likely to load in another second or two, or if it's stalled forever. For servers, this timeout can be used to abort processing of requests that exceed the threshold. (If the user isn't waiting for the response, there's no reason to keep executing the request.) Servers could also use this parameter to prioritize incoming requests: those that expect a subsecond response will go to the top of the pile, and those that say "hey, no rush" can be delayed.

How would the timeout be specified? We need to allow different requests to specify different timeouts. For the first request to a site, the timeout could be delivered via DNS (not saying this is easy to implement, but...): a response from a DNS server would contain the destination IP as well as the recommended timeout header to request. For subsequent pages, a parameter on an <a> tag or Ajax call would instruct the browser on how to construct the request header.

If a critical mass of sites/browsers/servers implement this request header, users will come to trust that the Internet won't leave them hanging -- if a server is taking longer to respond than it's supposed to, we'll time out the response so they can go about their business. Or, we could offer a prompt: "Your request has been cancelled because the website did not respond in a timely manner. Would you like to try again with no time constraint? [Yes|No]" Either way, having the confidence of a time limit would discourage users from abandoning early, or hitting "refresh" because the site feels slow. To that end, browsers could even refuse to re-issue a command until the timeout was reached. (Hang on, we're working on it...)

Defaulting on performance

In creating a highly performant Web environment, I've frequently been hampered by the very conservative default limits that exist in many technologies. In fact, I'd say that at least 4 out of every 5 performance bottlenecks I've encountered have come not from the system being actually overwhelmed, but rather from a self-imposed limit. Systems have numerous in-built constraints on concurrency, memory usage, and many other ways to prevent themselves from utilizing all of their available resources and realizing their true potential. Servers, you need a life coach. I'm here for you.

Of course, these limits were placed there by very wise people with very good reasons. For example, on *NIX operating systems, the designers were considering the needs of a multi-user situation: in a university CS lab where dozens of naive and/or mischievous undergrads are sharing clock cycles of a single CPU, it's important that no one user is allowed to chew up all the resources and bring down the system. But a Web server typically runs just one user, and has much more capacity than it is configured to use by default. When you're chasing down an international superspy in your Bugatti Veyron (who hasn't been there?), you've gotta pop out the electronic speed limiter and go for it. Likewise, when your NCIS slash fiction goes viral (who hasn't been there?), you need to goose the config and let those servers fly.

So, here is an incomplete list of some of the configuration-imposed constraints I've encountered over the years that are typically easy to relieve -- if you do it before the crowds arrive:

• Linux: iptables' ip_conntrack default database size is too small [see racker hacker]
• MySQL: max_connections default is low [see electric toolbox]
• Apache: prefork module is single-threaded [see serverfault]
• MySQL: query cache is disabled by default [it's not always a good idea to turn it on, but when it's good, it can be very good. see mysqlperformanceblog]
• Tomcat: the JDBC default connection pool size is small [see tomcat 6 docs, but also, tomcat 7's new hotness]
• Java: default JVM memory settings don't take advantage of available memory, and default garbage collection can create long "stop-the-world" pauses [this is a deep topic, but here's an intro]
• Linux: default max open files is too low [see stackoverflow]
• MySQL: back_log default is low [see MySQL documentation]
• MySQL: innodb_thread_concurrency default is low [see stackexchange -- though be aware, setting it to "0" (infinite) might be too much]
• Linux: net.core.somaxconn, net.core.netdev_max_backlog are too low [see nginx.org]

Share the load

For load-testing OvationTix, we've tried a few approaches over the years. The first time around, we used HP LoadRunner, which is an enterprise-level tool with a price to match. It was pretty easy to use, and we got the data we needed, but it was too expensive to become a part of our ongoing development process. Ideally, we'll load-test every release before deploying, and I don't want cost concerns to intimidate us into holding back from deploying good code when it's ready to go.

So we moved to jmeter, running in Amazon EC2 cloud instances, which of course was cheaper. I set up some (admittedly clunky) Windows instances -- a controller and some generators -- and went to work. Again, we got the data we needed, but now the workflow was cumbersome. We had to launch the generators, hope they booted correctly, figure out their IPs, copy those back to the controller, then fire up the scripts, and then we had problems with the test data saturating the connection between the generators and the controller. It was fair, but not great.

For this year, we made it our goal to have a smoothly automated system -- still based around jmeter, which we like. First, we tried BlazeMeter. It's a jmeter PaaS, which is a really cool idea and promises to take care of the infrastructure so we could focus on writing the tests. It's not bad at all, and I think we may use it in the future, but for now, the costs were higher than we wanted, there were too many limitations on usage (the price tiers control things like ramp-up time, max users etc.), and the reporting wasn't as transparent as we wanted.

Finally, we found jmeter-ec2, which is a wrapper around Amazon's API that automates launching linux micro instances, deploying resources to those instances, firing up the test, and aggregating results. It's a lightweight script that runs in a shell and eliminates the need for a dedicated controller -- instead, each generator controls its own virtual users, and the condensed results are sent back to the shell, which makes for much less traffic between the instances (therefore, no saturation). The data collected isn't as deep as with the other approaches, but for our purposes, that's okay. We're mostly interested in simply finding out how many users we can throw at the site before it crashes. Since our plan is to take over the world, our target for concurrent users is currently 7,057,131,972. Wish us luck.