To make the regulator function properly, you actually need two fixed ADJ resistors, referenced to the negative side of the battery (ground). Currently there is no reference, the way you've got it wired.
Also, you should probably have a current limiting resistor in series with the LED.
It's not really clear what your battery pack source voltage is (it appears to be a two-cell AAA pack, but I can't tell.) For this circuit, with only two series AAA's you will on have 3.0V from the battery pack. The linear regulator dropout is probably anywhere from 1.5V - 2.0V. This means that to generate the required 3.5V output you would need an input voltage of at least 5.0V - 5.5V to the regulator. This won't work with the two-cell pack you have there (3.0V input). Moreover, you need to current limit the LED so you actually need an output voltage higher than the required LED forward voltage drop, and then you would limit the current with the series resistor.
I recommend reading the datasheet for the LM317:
http://www.ti.com/lit/ds/symlink/lm317-n.pdf
Also, you should probably have a current limiting resistor in series with the LED.
It's not really clear what your battery pack source voltage is (it appears to be a two-cell AAA pack, but I can't tell.) For this circuit, with only two series AAA's you will on have 3.0V from the battery pack. The linear regulator dropout is probably anywhere from 1.5V - 2.0V. This means that to generate the required 3.5V output you would need an input voltage of at least 5.0V - 5.5V to the regulator. This won't work with the two-cell pack you have there (3.0V input). Moreover, you need to current limit the LED so you actually need an output voltage higher than the required LED forward voltage drop, and then you would limit the current with the series resistor.
I recommend reading the datasheet for the LM317:
http://www.ti.com/lit/ds/symlink/lm317-n.pdf