Much of what Rails provides to get your apps up and running isn't optimized for performance. It's crafted to be more efficient for developers, not more efficent at runtime. before_filter callbacks on your RESTful controllers to get the current object? That's an extra database call. All those nifty plugins you are using to kickstart your app? They probably generate far more SQL (and slower SQL) than if you coded the same functionality ad-hoc. ActiveRecord itself is slow compared to raw SQL and object instantiation.
If your project grows huge -- 10's of millions of PV/day huge -- you're gonna have to revisit some of that stuff. Some of it you can compensate for with smart caching techniques and more hardware. And some of it you will have to throw away and rewrite. If you get really huge, you're going to have pay back some of the technical debt you incurred by choosing a tool like Rails.
Yes, using a tool like Rails incurs some debts. But just like the rest of the world, there are good debts and bad debts. If you're smart about the kind of debt you take on, you can build far more, and build it faster.
Many of your projects will never reach the level where you need to "pay back" for all that developer productivity you enjoyed on the front end. That means you can try more ideas, and (hopefully) fail fast at the more speculative ones. If some of your projects do need to scale radically (beyond the basics of better caching, more hardware, etc), you have an incredible amount of upfront productivity you can leverage against that work of -- say -- optimizing some key queries by hand. As long as you go in with eyes wide open and realistic expectations, then I say that's a smart kind of technical debt.