It's important to understand the difference between sitting down to meditate, where you are setting aside a time for "non-doing", and other times where you are procrastinating or whiling away time, which could be counted as doing nothing.
The key differences are intention and immersion in the present moment. When you are mindlessly scrolling through facebook, or sitting on the couch with the TV on in the background, your mind is on autopilot. Whereas when you sit down to meditate, your mind is focussed on and immersed in the present moment. You aren't letting life pass you by, or thinking about the future, you're appreciating what's going on around you right now. And this is exactly what makes "non-doing" such a worthwhile activity.
Source: Adapted from a passage in - Wherever You Go, There You Are by Jon Kabat-Zinn