Will you be using the built in microphones, or will you want to connect your own? If you want to connect your own mics, you will need a recorder that has ¼” or XLR inputs. Be sure to select a recorder with phantom power if you are using condenser mics as most require phantom power.
Will you listen to your recordings on the device using headphones or speakers, or do you want a recorder with built in speakers for playback? (Note that the built in speakers are not loud or of high quality, given how small they are to fit on the device.)
Do you want to just record to stereo, or do you want the ability to multi-track, or record more than just stereo?
If you plan on recording rehearsals or concerts, it may be important to have the ability to mount the recorder on a mic stand and start or stop playback from a remote instead of moving the recorder up or ...