Anytime a normal hill graphic would obstruct the cliff face from being seen, it won't draw the cliff face. Since the ladder acts as a hill, it doesn't render it, presumably to save a few draw commands as it expects it won't be needed (as in the case of its default uses: stairs and hills).
The goal was to make it maximally open ended and flexible, sort of like lockpicks or piercing crystals with no magical equivalent, or an indoors, limited, Orb of Thralni. Any barrier you absolutely don't want players to climb just needs to be >1 height difference. Or have *any* terrain on the base of the cliff, including little mushrooms or grass sprouts.
Alternatively you can change where it looks for an open square (terrain 0) to look for your custom ladder friendly terrain. Just don't forget to change it to leave your terrain behind when the ladder is removed. Then all you have to do is put your custom terrains where you want ladders used, preferably only in towns with matching cliff graphics.
Next idea... Breakable window that allerts enemies or angers towns people, passable once broken. Or openable window that is potentially trapped. Maybe a three way option, like locked doors, Break, Open, Leave.