I'm using a Tl082 op amp.
Not a good choice. The data sheet shows +/- 5V as the lowest you can use. With just a singe 5V supply in effect you are running this from +/- 2.5V. Also you can only get to within 2.5V of either supply rail.
Use an op amp that is designed for a lower voltage, like the MCP602. There are lots of others to choose from as well.