Language processing is monumentally complex and so it slows down visual processing of things like this. Even if it was a set of symbols unknown to a human test subject (basic shapes, even), the subject would impose some meaning on them in order to identify them and then would order them based on their meaning, so that processing would be directed to the language centers of the brain. For chimps it's easy because it's more like ordering based on colour for them. Even ordering based on colour would be difficult for us, but definitely easier than based on number. The less the icons activate our language centers, the easier it would be for us.