Saving someone quite some pain:
flyctl proxy 15432:5432 -s -a <app_name>
The -s flag is crucial! Otherwise your proxy won't boot up.
-s