You should pay attention to is. Whether you like it or not, make some ways to speed up your site better than the usual. If you can't afford a dedicated server to support your site, it will be a big loss for you. Wasting money is not a big deal. Page load time usability factor directly affects sales and conversions, so its worth the pay.