gitk: Fix some corner cases in computing vrowmod and displayorder