"For Gods sake" was possibly my most common thought during my earlier meditation sessions. And I used to think this after I realised that instead of thinking about my breath, I had been thinking about last night's game/breakfast/lunch/dinner/an argument I had, for the past ten minutes.
It really helped to think of any meditation session as having three phases, and to understand that my mind wandering wasn't me "doing it wrong" - it was part of the whole process . The three phases are as follows:
The first phase is when we are fully focussed on our breath, and immersed in the present moment.
The second phase is when we notice that our minds have wandered. We often don't see this coming, we just suddenly realise that we are no longer thinking about our breath. Instead of seeing this as a moment of failure, we can frame this as a success. Yes, our mind wandered, but we were able to notice, and we can bring it back to the breath now.
The third phase is where we return our attention to our breath. We don't need to be annoyed or chastise ourselves because our mind has wandered - that wandering is part of our meditation session!
Source: A Monk's Guide to Happiness - Gelong Thubten