The main difference between rotary and your more standard combustion engines is that rotary engines don’t use pistons. Essentially, one or more triangular rotors spin in a circular chamber to combust the fuel/air mixture, which is a lot less complex of a solution than using pistons. Other advantages include less mechanical stress, less vibration, higher power-to-weight ratio than a piston engine, higher compression, the ability to use a wider range of fuels, a wider torque band, and better thermodynamic efficiency.
It all sounds wonderful when described as above, but the only successful rotary engine design to be used in cars is the Wankel, most famously found in the Mazda RX-7 and RX-8. The problems come with fuel efficiency and emissions and the habit of the rotary seals to fail as they are exposed to the fuel and varying heat levels. They also tend to consume a lot of oil. However, when the pros outweigh the cons, it can be a brilliant engine. It’s also a great candidate for engine swaps to give a car a completely different character in its performance. Engine swaps are not the easiest of projects, and a rotary has a particularly high level of difficulty, but that hasn’t stopped some adventurous car owners.