Because of the new mechanism built into Alfresco 4.0 that keeps track of failed transformations and limits the number retries (which is a good thing), we can't just reject the requests when we're busy. Currently there is a waiting period of 240 secs per request, and only when that limit is exceeded (no free slot available during that time) is the call rejected, which will cause an exception in the Alfresco logfile.
I think the mechanism in Alfresco should be extended to handle an overload situation as a special case. Since the transformation did not abort because the file is corrupt or too large, such an attempt should not be registered as a failed attempt. This could be done by having a special Exception type (e.g. TransformerBusyException) that will not trigger the setting of the ContentModel.ASPECT_FAILED_THUMBNAIL_SOURCE aspect on a node, which would allow us to fail fast and eliminate these problems.