Mercury is a very polished and reasonably powerful system. FTD has been using it to replace their entry-level and more advanced POS systems (incl Advantage, which was the RTI competitor).
RTI is a very powerful POS system used by larger shops. It's an older system, but still sold by TF as their upper tier system, while Dove is their entry-level platform.
RTI will be the bigger investment in terms of dollars and learning curve for you and staff.
Have you looked into non-wire service POS systems?