Phusion Passenger provides a set of tools, which are useful for system analysis, maintenance and troubleshooting. === Inspecting memory usage === Process inspection tools such as `ps` and `top` are useful, but they link:http://groups.google.com/group/phusion-passenger/msg/1fd1c233456d3180[rarely show the correct memory usage]. The real memory usage is usually lower than what `ps` and `top` report. There are many technical reasons why this is so, but an explanation is beyond the scope of this Users Guide. We kindly refer the interested reader to operating systems literature about 'virtual memory' and 'copy-on-write'. The tool `passenger-memory-stats` allows one to easily analyze Phusion Passenger's and Apache's real memory usage. For example: ------------------------------------------------------- [bash@localhost root]# passenger-memory-stats ------------- Apache processes --------------. PID PPID Threads VMSize Private Name ---------------------------------------------. 5947 1 9 90.6 MB 0.5 MB /usr/sbin/apache2 -k start 5948 5947 1 18.9 MB 0.7 MB /usr/sbin/fcgi-pm -k start 6029 5947 1 42.7 MB 0.5 MB /usr/sbin/apache2 -k start 6030 5947 1 42.7 MB 0.5 MB /usr/sbin/apache2 -k start 6031 5947 1 42.5 MB 0.3 MB /usr/sbin/apache2 -k start 6033 5947 1 42.5 MB 0.4 MB /usr/sbin/apache2 -k start 6034 5947 1 50.5 MB 0.4 MB /usr/sbin/apache2 -k start 23482 5947 1 82.6 MB 0.4 MB /usr/sbin/apache2 -k start ### Processes: 8 ### Total private dirty RSS: 3.50 MB --------- Passenger processes ---------. PID Threads VMSize Private Name ---------------------------------------. 6026 1 10.9 MB 4.7 MB Passenger spawn server 23481 1 26.7 MB 3.0 MB Passenger FrameworkSpawner: 2.0.2 23791 1 26.8 MB 2.9 MB Passenger ApplicationSpawner: /var/www/projects/app1-foobar 23793 1 26.9 MB 17.1 MB Rails: /var/www/projects/app1-foobar ### Processes: 4 ### Total private dirty RSS: 27.76 M ------------------------------------------------------- The 'Private' or 'private dirty RSS' field shows the *real* memory usage of processes. Here, we see that all the Apache worker processes only take less than 1 MB memory each. This is a lot less than the 50 MB-ish memory usage as shown in the 'VMSize' column (which is what a lot of people think is the real memory usage, but is actually not). NOTE: Private dirty RSS reporting only works on Linux. Unfortunately other operating systems don't provide facilities for determining processes' private dirty RSS. On non-Linux systems, the Resident Set Size is reported instead. === Inspecting Phusion Passenger's internal status === One can inspect Phusion Passenger's internal status with the tool `passenger-status`. This tool must typically be run as root. For example: -------------------------------------------------- [bash@localhost root]# passenger-status ----------- General information ----------- max = 6 count = 1 active = 0 inactive = 1 ----------- Domains ----------- /var/www/projects/app1-foobar: PID: 9617 Sessions: 0 Processed: 7 Uptime: 2m 23s -------------------------------------------------- The 'general information' section shows the following information: max:: The maximum number of application instances that Phusion Passenger will spawn. This equals the value given for <> (Apache) or <> (Nginx). count:: The number of application instances that are currently alive. This value is always less than or equal to 'max'. active:: The number of application instances that are currently processing requests. This value is always less than or equal to 'count'. inactive:: The number of application instances that are currently *not* processing requests, i.e. are idle. Idle application instances will be shutdown after a while, as can be specified with <>/<> (unless this value is set to 0, in which case application instances are never shut down via idle time). The value of 'inactive' equals `count - active`. The 'domains' section shows, for each application directory, information about running application instances: Sessions:: Shows how many HTTP client are currently in the queue of that application Instance, waiting to be processed. Processed:: Indicates how many requests the instance has served until now. *Tip:* it's possible to limit this number with the <> configuration directive. Uptime:: Shows for how long the application instance has been running. Since Phusion Passenger uses fair load balancing by default, the number of sessions for the application instances should be fairly close to each other. For example, this is fairly normal: -------------------------------- PID: 4281 Sessions: 2 Processed: 7 Uptime: 5m 11s PID: 4268 Sessions: 0 Processed: 5 Uptime: 4m 52s PID: 4265 Sessions: 1 Processed: 6 Uptime: 5m 38s PID: 4275 Sessions: 1 Processed: 7 Uptime: 3m 14s -------------------------------- But if you see a "spike", i.e. an application instance has an unusually high number of sessions compared to the others, then there might be a problem: -------------------------------- PID: 4281 Sessions: 2 Processed: 7 Uptime: 5m 11s PID: 17468 Sessions: 8 <-+ Processed: 2 Uptime: 4m 47s PID: 4265 Sessions: 1 | Processed: 6 Uptime: 5m 38s PID: 4275 Sessions: 1 | Processed: 7 Uptime: 3m 14s | +---- "spike" -------------------------------- Possible reasons why spikes can occur: . Your application is busy processing a request that takes a very long time. If this is the case, then you might want to turn <> on. . Your application is frozen, i.e. has stopped responding. See <> for tips. [[debugging_frozen]] === Debugging frozen applications === If one of your application instances is frozen (stopped responding), then you can figure out where it is frozen by killing it with 'SIGABRT'. This will cause the application to raise an exception, with a backtrace. The exception (with full backtrace information) is normally logged into the Apache error log. But if your application or if its web framework has its own exception logging routines, then exceptions might be logged into the application's log files instead. This is the case with Ruby on Rails. So if you kill a Ruby on Rails application with 'SIGABRT', please check the application's 'production.log' first (assuming that you're running it in a 'production' environment). If you don't see a backtrace there, check the Apache error log. NOTE: It is safe to kill application instances, even in live environments. Phusion Passenger will restart killed application instances, as if nothing bad happened. === Accessing individual application processes === When a request is sent to the web server, Phusion Passenger will automatically forward the request to the most suitable application process, but sometimes it is desirable to be able to directly access the individual application processes. Use cases include, but are not limited to: * One wants to debug a memory leak or memory bloat problem that only seems to appear on certain URIs. One can send a request to a specific process to see whether that request causes the process's memory usage to rise. * The application caches data in local memory, and one wants to tell a specific application process to clear that local data. * Other debugging use cases. All individual application processes are accessible via HTTP, so you can use standard HTTP tools like 'curl'. The exact addresses can be obtained with the command `passenger-status --verbose`. These sockets are all bound to 127.0.0.1, but the port number is dynamically assigned. As a security measure, the sockets are also protected with a process-specific random password, which you can see in the `passenger-status --verbose` output. This password must be sent through the ``X-Passenger-Connect-Password'' HTTP header. Example: [listing] ...................................... bash# passenger-status --verbose ----------- General information ----------- max = 6 count = 2 active = 0 inactive = 2 Waiting on global queue: 0 ----------- Application groups ----------- /Users/hongli/Sites/rack.test: App root: /Users/hongli/Sites/rack.test * PID: 24235 Sessions: 0 Processed: 7 Uptime: 17s URL : http://127.0.0.1:58122 Password: nFfVOX1F8LjZ90HJh28Sd_htJOsgRsNne2QXKf8NIXw * PID: 24250 Sessions: 0 Processed: 4 Uptime: 1s URL : http://127.0.0.1:57933 Password: _RGXlQ9EGDGJKLevQ_qflUtF1KmxEo2UiRzMwIE1sBY ...................................... Here we see that the web application 'rack.test' has two processes. Process 24235 is accessible via http://127.0.0.1:58122, and process 24250 is accessible via http://127.0.0.1:57933. To access 24235 we must send its password, 'nFfVOX1F8LjZ90HJh28Sd_htJOsgRsNne2QXKf8NIXw', through the 'X-Passenger-Connect-Password' HTTP header, like this: ------------------------------------------- bash# curl -H "X-Passenger-Connect-Password: nFfVOX1F8LjZ90HJh28Sd_htJOsgRsNne2QXKf8NIXw" http://127.0.0.1:58122/ ------------------------------------------- === Attaching an IRB console to an application process === **Available in Phusion Passenger Enterprise since version 3.0.0.** You can attach an IRB console to any application process and inspect its state by executing arbitrary Ruby code. Do this by invoking `passenger-irb ` where '' is the PID of the application process you wish to inspect. Note that the IRB console is currently only available for Ruby apps, not for apps in any other languages.