You can delay family travel for many reasons. But do not delay it simply because you are planning to take your child travelling once they reach a certain age. There is no such thing as a perfect age for children to start travelling. Every age group from baby, toddler, school age and grown up kids will have its own pros and cons.
Waiting to start family travel until children reach school age will mean more expensive holidays. You will be paying full airfares and you are limited to travel during school holiday (high) seasons. Accommodations will be more expensive as you need bigger rooms. And there will be less chance of discounted entry fees to many sites. Children at this age will also more likely to complain of being bored, and have more opinions with what they want to see and do.
Later in their teenage years you may no longer be fit enough to keep up with them. They may even prefer to stay home and hang out with their friends.
There might be extra bit of organisation, slightly heavier baggage, occasional tantrums and nappy changes when travelling with little ones. Remember that you will come home with a lifetime of memories that you create from the experience.