It could be a good occasion to try this: Integrated Circuits - Memory 
Otherwise, if you wanna try to work with what you got, you can try this:
1 - if you want to cut mem usage by 2 right away, use char instead of int.
2 - you can use an easy algorithm, if you observe the pattern a bit more attentively.
for your 4 pin example, you have:
(ledN ranging from 0 to 11)
// (ledN ranging from 0 to 11)
pinL = ledN/3+1
tmp = ledN % 3 + 1
if (tmp < pinL) {
pinH = tmp
}else{
pinH = tmp + 1
}
I guess, if you use the same kind of wiring for 18 pins, you get:
// (ledN ranging from 0 to 305)
pinL = ledN/17 + 1
tmp = ledN % 17 + 1
if (tmp < pinL) {
pinH = tmp
}else{
pinH = tmp + 1
}
I suspect tho, that manually wiring 306 leds for charliplexing is gonna be quite a pain in the ass.
I would have rather multiplexed them using shift registers and preassembled 8x8 led matrices.
I've a friend who actually manually built a 16x16 led array and I can tell that after placing and soldering 200 of those, he wasn't that excited about the project anymore.
But if you still do it, that would be cool, I fully support you!