Troubleshooting BlueMix aps when they fail to start
- categories: cloud
- tags: ibm, bluemix, cloudfoundry, troubleshooting
It is more difficult to troubleshoot apps running in BlueMix to start with but it is still possible to do naive things like output to stdout/stderr and view the startup logs via the BlueMix space console and viewing the application logs. This is a challenge to the paradigm shift as you can no longer pop over to a remote session to the machine and poke around. Thankfully CloudFoundry provides the
cf command with a host of goodies to help us out here, let’s take a look at a few.
So we are minding our own business we push an app change and splat… it fails to start with no obvious error…
swilbur@SWILBUR ~/workspaces/harbinger (swilbur_cloudant-test) $ cf push Using manifest file c:\Users\swilbur\workspaces\harbinger\manifest.yml Creating app unleashthecloud3 in org email@example.com / space Prototype as firstname.lastname@example.org... OK Creating route unleashthecloud3.mybluemix.net... OK Binding unleashthecloud3.mybluemix.net to unleashthecloud3... OK Uploading unleashthecloud3... Uploading app files from: c:\Users\swilbur\workspaces\harbinger Uploading 93K, 25 files Done uploading OK Starting app unleashthecloud3 in org email@example.com / space Prototype as firstname.lastname@example.org... 0 of 1 instances running, 1 starting 0 of 1 instances running, 1 down 0 of 1 instances running, 1 down 0 of 1 instances running, 1 down 0 of 1 instances running, 1 down 0 of 1 instances running, 1 down
The problem is when an application fails to start there are no logs to inspect! So what to do then?
One good first option, if you are also developing locally is to ensure you don’t have any silly syntax issues that are causing the runtime to just not run, that can produce non-obvious results when the container is built and starts as nothing is logged or output since you program simple does not start. This is not always possible for non-trivial setups or fully online development teams where they may not have a need for any local development environment and just have all environments setup in differnet orgs and spaces.
So let’ start troublehsooting, we get ourselves back to the
cf command line, logged into the appropriate org/space and ready to view the app in question, if you are trying to push as in my case you are already there, but if you had started from the delivery pipeline or a push made by a build tool or something you will enter here to look at the current as-is system.
swilbur@SWILBUR ~/workspaces/harbinger (swilbur_cloudant-test) $ cf api https://api.ng.bluemix.net Setting api endpoint to https://api.ng.bluemix.net... OK API endpoint: https://api.ng.bluemix.net (API version: 2.14.0) Not logged in. Use 'cf.exe login' to log in. swilbur@SWILBUR ~/workspaces/harbinger (swilbur_spyke) $ cf login -u email@example.com -o 'firstname.lastname@example.org' -s Prototype API endpoint: https://api.ng.bluemix.net Password> Authenticating... OK Targeted org email@example.com Targeted space Prototype API endpoint: https://api.ng.bluemix.net (API version: 2.14.0) User: firstname.lastname@example.org Org: email@example.com Space: Prototype
Now let’s inspect out broken app, here are some suggested commands:
See if anything has changed recently:
cf events <app_name>
See what the current environment settings are, this will show you things like the current VCAP_SERVICES and VCAP_APPLICATIONS, as well as any custom variables you may have set with
cf env <app_name>
For an application that is still running, this is where your stdout and stderr can be found as in console output from your application, starting when you issue the command, no history is shown here only new outputs.
cf logs <app_name>
If you application is not starting, this is how you look at recent messages to the logs, useful when and app is not running and you are trying to if it output anything useful when it died or failed to start.
cf logs <app_name> --recent
One less snazzy way to debug this is to try and bring it up in a local environment that you do have access to, sometimes duck typed languages don’t complain when you have simple syntax isseus that will bring your Node.js app down and leaves you very little to go on in the logs or output from Cloud Foundry.