This all depends on how far and fast you want to ride! There are street bikes called "pedal forward" bikes:
These are not as fast nor as comfortable for long distances, but something like this might be just right for you. The sixthreezero bikes come with various transmissions as well as single speed (no gears). Not having any gears would make riding uphill harder, but if you live in a flat area, might be fine.
If you want to keep your present bike, the way to stop is to stop pedaling with the pedals vertical, gently apply the brakes and when the bike is almost stopped move forward off the saddle, lean to the side with the upper foot and put that foot down. You can dab along for a couple of feet with that toe if you haven't quite stopped.