My explanation will be more clear if you pause the video @ 12:31 in the following link.
Just mark noon and midnight on the said dates.
Ok, let's assume the difference is 6 hours per year, to make things more obvious.
After 365 days calendar counts one year.
Days are still 86400 seconds each.
Solar noon is still when our meridian is directly facing the Sun.
Seasons shifted
by 6 hours.
After 730 days calendar counts two years.
Days are still 86400 seconds each.
Solar noon is still when our meridian is directly facing the Sun.
Seasons shifted
by 12 hours.
After 1095 days calendar counts three years.
Days are still 86400 seconds each.
Solar noon is still when our meridian is directly facing the Sun.
Seasons shifted
by 18 hours.
After 1460 days calendar counts four years.
Days are still 86400 seconds each.
Solar noon is still when our meridian is directly facing the Sun.
Seasons shifted
by 24 hours.
After 4 years we count one period between two solar noons as extra day.
It is irrelevant where in the year we do it.
One extra day compensates for seasons shift.
Without compensation, after 100 years our calendar would, at the beginning of spring, show April 14.
After 100 more years it would show May 9.
(Spring starts when Sun is above Equator and going north. It is based on tilt between Sun, and Earth's axis.)
Soon it would become useless for agriculture, and for few other activities.
EDIT:
I think that Arabic calendar is based on Moon.
Ramadan was in August in 2009 and in 2018 will be in May.
Their year is simply shorter and shifts pretty quickly through seasons.
They don't pay much attention to it.
But when they do business, they use Gregorian calendar.
I also think that Chinese calendar is based on Moon.
In 2016 Chinese new year was on Feb 9, in 2018 is on Feb 16.
They every several years add one month, so their year shifts only back and forth, not through all four seasons.
For business they also use Gregorian calendar.
All together we count hours based on time between two solar noons and days and hours can't shift from each other.