If you have a webserver and find you have high IO/lagginess MySQL is one of the first things to check. It turns out MySQL was my problem and it was creating a high load on my server, especially for IO.
Add this anywhere under [mysqld]
log-slow-queries = /var/log/mysql/mysqlslowqueries.log
long_query_time = 1
You can change the log path to whatever you like. You can also chanege the threshold for slow queries, I have it sent on 1 second. Once you do this and restart mysqld you'll start seeing the user, database and exact query that is slowing things down.
I was finding queries that were taking as long as 42 seconds long! That was 42 seconds of slow IO performance that was wreaking havoc on my webserver.
I thought about creating a MySQL cluster but it requires several servers and takes some time to setup and test. I also thought about a hardware upgrade, getting faster RPM HDD's and maybe going SSD, but this takes time and money even if I wanted to do it now.
What can be done immediately? The answer for me and a lot of people will be MySQL Query Caching.
Under [mysqld] add:
#increase performance with caching
#128MB of memory cache in KB
#maximum size of individual query that can be cached in KB (about 1 gig although note the cache total size is restricted to 128MB now)
After doing this my performance issues were solved. A quick note about Query Caching is that it only works for the EXACT same SELECT statements when data has not changed, if data changes the cache becomes irrelevant until the next request. Also note that if the SELECT statement expliclity specifies NOCACHE (can't remember the syntax) then the cache will not be used.
For many database driven sites this is still good and at least it can only improve performance. I've found it to make a huge difference.
mysql, enable, query, logging, cache, mysqld, io, cpu, usageif, webserver, lagginess, creating, server, queries, vi, etc, cnf, var, mysqlslowqueries, long_query_time, chanege, threshold, restart, ll, user, database, slowing, wreaking, havoc, cluster, requires, servers, hardware, upgrade, rpm, hdd, ssd, caching, improve, mb, kb, query_cache_size, query_cache_type, maximum, individual, cached, restricted, query_cache_limit, select, statements, irrelevant, expliclity, specifies, nocache, syntax, sites, ve,