Let's do the math.
If you spend 5 minutes per day playing your drum, that adds up to 35 minutes per week, 2.3 hrs per month, 27.6 hours per year.
What if you spent 10 minutes a day? That's 70 minutes per week, 4.6 hrs per month, 56 hrs per year...
The ability to foresee the cumulative effects of your actions is an important skill to learn both in music and in life. Begin to look at the efficiency of your daily routine and how much time you spend on things that are important to you. Pay attention to those things you could do without, and replace them with actions that will help you to progress.