Saving someone quite some pain:

flyctl proxy 15432:5432 -s -a <app_name>

The -s flag is crucial! Otherwise your proxy won't boot up.