Defaulting on performance

In creating a highly performant Web environment, I've frequently been hampered by the very conservative default limits that exist in many technologies. In fact, I'd say that at least 4 out of every 5 performance bottlenecks I've encountered have come not from the system being actually overwhelmed, but rather from a self-imposed limit. Systems have numerous in-built constraints on concurrency, memory usage, and many other ways to prevent themselves from utilizing all of their available resources and realizing their true potential. Servers, you need a life coach. I'm here for you.

Of course, these limits were placed there by very wise people with very good reasons. For example, on *NIX operating systems, the designers were considering the needs of a multi-user situation: in a university CS lab where dozens of naive and/or mischievous undergrads are sharing clock cycles of a single CPU, it's important that no one user is allowed to chew up all the resources and bring down the system. But a Web server typically runs just one user, and has much more capacity than it is configured to use by default. When you're chasing down an international superspy in your Bugatti Veyron (who hasn't been there?), you've gotta pop out the electronic speed limiter and go for it. Likewise, when your NCIS slash fiction goes viral (who hasn't been there?), you need to goose the config and let those servers fly.

So, here is an incomplete list of some of the configuration-imposed constraints I've encountered over the years that are typically easy to relieve -- if you do it before the crowds arrive:

• Linux: iptables' ip_conntrack default database size is too small [see racker hacker]
• MySQL: max_connections default is low [see electric toolbox]
• Apache: prefork module is single-threaded [see serverfault]
• MySQL: query cache is disabled by default [it's not always a good idea to turn it on, but when it's good, it can be very good. see mysqlperformanceblog]
• Tomcat: the JDBC default connection pool size is small [see tomcat 6 docs, but also, tomcat 7's new hotness]
• Java: default JVM memory settings don't take advantage of available memory, and default garbage collection can create long "stop-the-world" pauses [this is a deep topic, but here's an intro]
• Linux: default max open files is too low [see stackoverflow]
• MySQL: back_log default is low [see MySQL documentation]
• MySQL: innodb_thread_concurrency default is low [see stackexchange -- though be aware, setting it to "0" (infinite) might be too much]
• Linux: net.core.somaxconn, net.core.netdev_max_backlog are too low [see nginx.org]