Have you tested load times from different geographical locations? Who is hosting your site it may be that you can achieve fast load times where you are at and are testing from. Also, clear your browser cookies before testing load time just in case. From a look at your site, maybe you could change the layout into tiles and allow the viewer to get a look at more posts/images/headlines right away. If something doesn't catch their eye when it loads they are going to be less inclined to search around for something that does.

Hope this helps!

LadiesLife