Great Idea! Thank you.

I've been doing exactly that - have siamese network that takes image on each side and concatenate output. I've initially used two mobilenets that later become one at the dense layer but now use just my own very small net.

Initially I used two similar but separate nets and then the RGB did a little better.

Although I did not do that, I think you could perhaps then increase training sample by swapping images in a single class rather than use the same (aligned) rgb and rgn image for each input.

Augmentation on RGN is tricky, and the size differ quite a bit - just some practical issues you need to consider when you do this.