I have made a standalone arduino board using atmega328. I am using it on my home automation project with a custom relay board and a hc 05 bluetooth module. From the first itself i am getting problems. The board resets after sometime or after some serial transmission. Or sometime it stucks. Why is it happening and what is the solution for it?
No idea since we have no idea whatsoever how your system is connected.
But actually you did mention one clue - you said "custom relay board". My bet is on that as the problem.