Well then I guess this project should probably use a relatively thick-walled glass bottle or jar, especially non-colored glass. While there is still an issue of some creature accidentally eating it, a glass vessel if broken up would eventually be turned into small pieces of glass. Common non-colored glass is almost three quarters silica (SiO2), similar to a lot of the natural sands, along with some sodium oxide (Na2O) and lime (CaO). So even though it probably won't break chemically, if it does there's nothing involved that would be innately poisonous in small amounts. Also unlike polymers, even small piece of glass wouldn't stay suspended in the water column indefinitely. Instead it will act much like small rocks or sand, depending upon the size.
I'm certainly not in favor of adding to the oceanic pollution problem, but if this device is never found again it looks like a glass container would pose less long-term problems than a plastic one. Of course that's not taking into account whatever is inside the container...