We had similar question already
Visits per day -- should we be worried?
I dont think you have to worry that this site get closed. SE team knows, that user number will grow with ongoing time.
Interesting to compare
http://stackexchange.com/sites?sort=newengagedusers&expand=true
http://area51.stackexchange.com/proposals/18413/german-language-usage
http://area51.stackexchange.com/proposals/7526/japanese-language-usage
http://area51.stackexchange.com/proposals/29480/french-language-usage
So imho our stats look OKish to me
http://area51.stackexchange.com/proposals/14456/spanish-language-usage
Spanish proposal will soon start and probably make SE as a language site much more popular, Japanese, French, German are likely still minorities in the language spectrum of SE users.
good opportunity to advertize
http://area51.stackexchange.com/proposals/21061/latin-language-usage
one more time ;) Please vote questinos with less than 10 votes!