In the comments to my OR/M Smackdown post Adam Tybor noted:  

To which Ted Neward has replied:

Which reminds me of a conversation that I had with Udi Dahan recently, which we concluded with this great quote from him:

All of which brings me to the following conclusion, performance tuning in the microseconds is a waste of time until you have profiling data in place, but that doesn't meant that you should think about performance from the get go. You can code your way out of a lot of things, but an archtiecture that is inherently slow (for instance, chatty on the network) is going to be very hard to modify later on.

Udi had an example of a Customer that has millions of orders, in this case, the performance consideration has a direct affect on the domain model (Yes, I know about filters). From a desing perspective, it basically means that the entity contains too much data and need to be broken. From a performance perspective, that is making it explicit that a potentially very costly call is made (and obviously filtered for the need).

A good rule of the thumb for performance is that you should consider an order of magnitude increase in the number of users/transactions before you need to consider a significant change in the archtiecture of the application.

That is absolutely not to say that you should consider everything in advance, and I had my greatest performance success by simply batching a few thousands remote calls into a single one, but architecutre matters, and that should be considered in advance, and built accordingly. (And no, it doesn't necessiate a Big Architecture Up Front either, although where I would need to scale very high I would spent a while thinking about the way I am going to build the app in advance. Probably with some IT/DBA/Network guys as well, to get a good overview.

Oh, nevertheless, premature optimization...