Consumer confidence fell by five percent in March, as Australians worry about rising interest rates and falling employment levels.