Don't ignore your community but be ready to moderate
This is a very hard one with lots of bots / art spammers targeting small creators, you don't know who is genuinely starting a conversation and who only wants to spam you with their art or various promotions. Check their profile first, what they post, is it even relevant to your content? You need to make the decision here and not waste your time on spammers. I spent way too much time responding to every single hi and running rounds of small talk by the time they finally blurted out they wanted to sell their art to me. You don't have to reply to every single DM, use your time wisely, do a bit of profile research and then reply to the ones who are genuine viewers and are truly part of your community.
And if someone comes to your stream just to promote themselves or posts comments on your videos which criticise you in a not constructive way, just delete the comment or ban them. Be ready to protect yourself early on before it gets out of hand. If you have to pause your stream, just let your viewers know, go to your dashbboard and find the mod tools. Big channels have moderators for this, you'll only have yourself. And your true viewers will appreciate seeing that you are looking out for them and work on providing a safe and good experience for them.
You can't just stream and expect to grow
This is one I learnt the hard way. You have to start being active on social media, post about what you do and create other content like short videos which will help you get discovered, because just from streaming no one will find you. Youtube is a bit better with discoverability since past streams show up on your channel, but you won't get discovered just by going live on Twitch. Your channel will be at the very bottom of the page of the game you are streaming. The higher viewer count someone has, the higher their channel will come up when someone searches for a game.
Network with other creators you follow, introduce yourself on their discord if you are a member, speak to their community, find friends and you'll find new viewers as well. Try it with creators your size, they will more likely be approachable and willing to help since you are in the same boat. You might even get a nice collab out of it one day :) I have discovered new creators by being thrown into their channel with a raid, so maybe your creator friends will trust you with their community and you get new people watching you after you get a raid.