The O's went two up inside the first ten minutes of the second half through Paul Comley and Tom Hutton but in an inspired turn around, the Wiltshire side levelled through Steve Seals and Lewis Graham before Jake Herd scored the winner with nine minutes to go.
The O's had started by far the better side and both Ben Harris and Scott hendy tested the Warminster keeper early on. Then, debutant Jordan Greenwood and Paul Comley also had chances to score before half time.
The second half started in great fashion for the O's. Comley again went close but four minutes in, they finally took the lead their play had deserved. Harris broke down the right flank and crossed for Comley to finish well at the far post.
And just seven minutes further into the game, Oldland doubled their lead. Comley laid the ball back from Tom Hutton to smash a shot home from 25 yards out for his sixth goal of the season, a trademark piledriver.
At this point, it seemed the hosts were odds on for their first victory in two months but the next twenty minutes was a horror show. Steve Seals pulled a goal back on the hour mark and a long ball out of defence by Warminster found Lewis Pearce and he rifled home the equaliser on 68 minutes.
The comeback was completed with nine minutes left when Jake Herd scored direct from a free kick to cap a disappointing debut for new manager Dale Dempsey.