Welcome to /proc

What is /proc? It is a pseudo directory present in all Linux systems which provides information about the system. It was originally designed to convey process information (hence the name), but now conveys a wide array of system information.

Note that I referred to /proc as a “pseudo directory”. It doesn’t actually exist (on disk). Instead, the Linux kernel creates this pseudo filesystem in memory, so that we can access it as if it were a directory on disk.

Let’s take a look.

$ ls /proc
1     13401  2      25     34    4478  50    5336  61   674  75   818   88   98         device-tree  ioports      locks         slabinfo       tty
10    14     21     26     35    45    5037  5337  62   676  77   844   89   asound     devices      irq          meminfo       softirqs       uptime
1084  142    22     26222  3611  450   51    54    63   679  774  85    9    buddyinfo  diskstats    kallsyms     misc          stat           vc-cma
1088  144    223    27     434   46    5136  5464  64   68   775  8580  90   bus        driver       key-users    modules       swaps          version
11    15     228    28     435   47    5183  55    65   69   78   8588  91   cgroups    execdomains  keys         mounts        sys            vmallocinfo
1105  152    229    29     438   48    52    56    66   7    79   8593  92   cmdline    fb           kmsg         net           sysrq-trigger  vmstat
1127  17     23     3      439   481   53    57    67   70   8    86    93   consoles   filesystems  kpagecgroup  pagetypeinfo  sysvipc        zoneinfo
1163  18     23307  31     44    485   5322  58    670  71   80   87    939  cpu        fs           kpagecount   partitions    thread-self
1238  18779  237    32     442   49    5323  59    671  72   801  870   96   cpuinfo    interrupts   kpageflags   sched_debug   timer_list
13    19     24     33     4477  5     5332  60    672  73   810  8719  97   crypto     iomem        loadavg      self          timer_stats

As you notice, there’s a whole slew of numeric directories. These contain information about the process with the corresponding id. There’s also a wide variety of other information, such as memory information (meminfo), load average statistics (loadavg), kernel version, gcc version, and Linux distribution (version), virtual memory statistics (vmstat), and a wide variety of other kernel and hardware information.

So, what’s in a process? Let’s take a peek.

$ ls /proc/$(pgrep -f -n tmux)
attr     auxv        clear_refs  comm       cpuset     environ     fd      gid_map  limits         map_files    mem     mounts     net               numa_maps  oom_score
pagemap  projid_map  sched       sessionid  smaps      stat        status  task     uid_map        autogroup    cgroup  cmdline    coredump_filter   cwd        exe
fdinfo   io          loginuid    maps       mountinfo  mountstats  ns      oom_adj  oom_score_adj  personality  root    schedstat  setgroups  stack  statm      syscall
timers   wchan

You can check out the status of a process in status.

$ cat /proc/$(pgrep -f -n tmux)/status
Name:   tmux
State:  S (sleeping)
Tgid:   4143
Ngid:   0
Pid:    4143
PPid:   1
TracerPid:      0
Uid:    1001    1001    1001    1001
Gid:    1001    1001    1001    1001
FDSize: 64
Groups: 10 1001 
VmPeak:   130100 kB
VmSize:   129840 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:      6136 kB
VmRSS:      5516 kB
RssAnon:            4220 kB
RssFile:            1296 kB
RssShmem:              0 kB
VmData:     4404 kB
VmStk:       140 kB
VmExe:       448 kB
VmLib:      2700 kB
VmPTE:        80 kB
VmSwap:      160 kB
Threads:        1
SigQ:   0/127679
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000081802
SigCgt: 0000000188034201
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000001fffffffff
CapAmb: 0000000000000000
Seccomp:        0
Cpus_allowed:   ff
Cpus_allowed_list:      0-7
Mems_allowed:   00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list:      0
voluntary_ctxt_switches:        1246008
nonvoluntary_ctxt_switches:     10550

In status you can see the state (sleeping), the process id (4143), the parent process id (1), ownership (user/group 1001), memory sizes, the number of threads, and many other details. It contains much of what is in stat and mem (status and memory information respectively), but in more human readable form.

You can also see what files the process is referencing in the fd directory. Here are the open file descriptors for my Plex Media Server.

$ cd /proc/$(pgrep -f -n 'Plex Media Server')
$ sudo ls -l fd
total 0
lr-x------ 1 plex nogroup 64 Apr 28 00:44 0 -> /dev/null
l-wx------ 1 plex nogroup 64 Apr 28 00:44 1 -> /dev/null
l-wx------ 1 plex nogroup 64 Apr 28 00:44 10 -> /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log
lrwx------ 1 plex nogroup 64 Apr 28 00:44 11 -> /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.dlna.db-shm
lrwx------ 1 plex nogroup 64 Apr 28 00:44 12 -> socket:[11739]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 13 -> socket:[11740]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 14 -> socket:[11741]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 15 -> socket:[11742]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 16 -> socket:[11743]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 17 -> socket:[11744]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 18 -> socket:[11746]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 19 -> socket:[11747]
l-wx------ 1 plex nogroup 64 Apr 28 00:44 2 -> /dev/null
lrwx------ 1 plex nogroup 64 Apr 28 00:44 20 -> socket:[11748]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 21 -> socket:[11749]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 22 -> socket:[11750]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 23 -> socket:[11751]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 24 -> socket:[11752]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 25 -> socket:[11753]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 26 -> socket:[11754]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 27 -> socket:[11755]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 28 -> socket:[11756]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 29 -> socket:[11757]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 3 -> /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Logs/Plex DLNA Server Neptune.log
lrwx------ 1 plex nogroup 64 Apr 28 00:44 30 -> socket:[11758]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 31 -> socket:[11759]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 32 -> socket:[11760]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 33 -> socket:[65881]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 34 -> socket:[65882]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 35 -> socket:[65883]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 4 -> anon_inode:[eventfd]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 5 -> anon_inode:[eventpoll]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 56 -> socket:[10045]
lr-x------ 1 plex nogroup 64 Apr 28 00:44 57 -> pipe:[10038]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 58 -> socket:[10044]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 59 -> socket:[10050]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 6 -> anon_inode:[timerfd]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 60 -> socket:[10053]
l-wx------ 1 plex nogroup 64 Apr 28 00:44 61 -> pipe:[10039]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 62 -> socket:[11730]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 63 -> socket:[10058]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 64 -> socket:[11731]
l-wx------ 1 plex nogroup 64 Apr 28 00:44 7 -> /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Logs/Plex DLNA Server.log
lr-x------ 1 plex nogroup 64 Apr 28 00:44 72 -> /dev/urandom
lrwx------ 1 plex nogroup 64 Apr 28 00:44 8 -> /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.dlna.db
lrwx------ 1 plex nogroup 64 Apr 28 00:44 82 -> socket:[10072]
lrwx------ 1 plex nogroup 64 Apr 28 00:44 9 -> /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.dlna.db-wal

Using the proc filesystem, we can even access and recover files. Let’s get the tail of the log for file descriptor 10 (Plex Media Server.log).

$ sudo cat fd/10 | tail
Apr 28, 2017 00:20:19.835 [0x731ff400] DEBUG - NAT: UPnP, not an IGD: <http://192.168.0.1:49152/wps_device.xml>.
Apr 28, 2017 00:20:19.837 [0x731ff400] DEBUG - NAT: UPnP, found device <http://192.168.0.1:49152/wps_device.xml> with private address <192.168.0.101>
Apr 28, 2017 00:20:19.838 [0x731ff400] DEBUG - NAT: UPnP, not an IGD: <http://192.168.0.1:49152/wps_device.xml>.
Apr 28, 2017 00:20:19.840 [0x731ff400] DEBUG - NAT: UPnP, found device <http://192.168.0.1:1900/igd.xml> with private address <192.168.0.101>
Apr 28, 2017 00:20:19.845 [0x731ff400] DEBUG - NAT: UPnP, usable device <http://192.168.0.1:1900/igd.xml> with private address <192.168.0.101>.
Apr 28, 2017 00:20:19.847 [0x731ff400] DEBUG - NAT: UPnP, public address is XXX.XXX.XXX.XXX
Apr 28, 2017 00:20:19.850 [0x680fb400] DEBUG - PublicAddressManager: Mapping succeeded for 192.168.0.101:23979.
Apr 28, 2017 00:20:19.854 [0x731ff400] DEBUG - MyPlex: Last published value didn't change, we're done.
Apr 28, 2017 00:20:19.855 [0x680fb400] DEBUG - MyPlex: Last published value didn't change, we're done.
Apr 28, 2017 00:50:17.467 [0x680fb400] DEBUG - Sync: uploadStatus

We can also check out the environment of the process via environ.

$ sudo cat environ
INFINALITY_FT_AUTOHINT_HORIZONTAL_STEM_DARKEN_STRENGTH=10^@MAIL=/var/mail/plex^@USER=plex^@INFINALITY_FT_FRINGE_FILTER_STRENGTH=0^@SHLVL=1^@LD_LIBRARY_PATH=/usr/lib/plexmediaserver^@HOME=/var/lib/plexmediaserver^@PLEX_MEDIA_SERVER_TMPDIR=/tmp^@OLDPWD=/var/lib/plexmediaserver^@INFINALITY_FT_CONTRAST=0^@INFINALIT    Y_FT_USE_VARIOUS_TWEAKS=true^@INFINALITY_FT_GAMMA_CORRECTION=0 100^@INFINALITY_FT_WINDOWS_STYLE_SHARPENING_STRENGTH=10^@INFINALITY_FT_STEM_ALIGNMENT_STRENGTH=25^@TMPDIR=/tmp^@LOGNAME=plex^@_=/usr/sbin/start_pms^@XDG_SESSION_ID=c1^@INFINALITY_FT_AUTOHINT_VERTICAL_STEM_DARKEN_STRENGTH=25^@PATH=/usr/local/sbin:/us    r/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games^@XDG_RUNTIME_DIR=/run/user/117^@PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR=/var/lib/plexmediaserver/Library/Application Support^@INFINALITY_FT_GLOBAL_EMBOLDEN_X_VALUE=0^@LANG=en_GB.UTF-8^@INFINALITY_FT_AUTOHINT_SNAP_STEM_HEIGHT=100^@INFINALITY    _FT_GLOBAL_EMBOLDEN_Y_VALUE=0^@INFINALITY_FT_CHROMEOS_STYLE_SHARPENING_STRENGTH=0^@SHELL=/bin/bash^@PLEX_MEDIA_SERVER_MAX_STACK_SIZE=3000^@INFINALITY_FT_BRIGHTNESS=0^@INFINALITY_FT_STEM_SNAPPING_SLIDING_SCALE=40^@INFINALITY_FT_STEM_FITTING_STRENGTH=25^@INFINALITY_FT_GRAYSCALE_FILTER_STRENGTH=0^@INFINALITY_FT_US    E_KNOWN_SETTINGS_ON_SELECTED_FONTS=true^@PWD=/usr/lib/plexmediaserver^@INFINALITY_FT_AUTOHINT_INCREASE_GLYPH_HEIGHTS=true^@PLEX_MEDIA_SERVER_HOME=/usr/lib/plexmediaserver^@INFINALITY_FT_BOLD_EMBOLDEN_X_VALUE=0^@INFINALITY_FT_FILTER_PARAMS=11 22 38 22 11^@INFINALITY_FT_BOLD_EMBOLDEN_Y_VALUE=0^@PLEX_MEDIA_SERVER_    MAX_PLUGIN_PROCS=6^@

We can inspect the command with which the process was launched via cmdline.

$ sudo cat cmdline
./Plex Media Server

We can also take a look at the various kernel level limits for a process in limits.

$ sudo cat limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             7336                 7336                 processes 
Max open files            65536                65536                files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       7336                 7336                 signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us

Wrapping up

As you can see, there’s all kinds of cool information in /proc!

Many utilities (ps, top, lspci, etc.) get their information from the /proc filesystem.

Let’s take a look at the files ps opens when it runs on my Raspberry Pi.

$ strace -etrace=open ps
open("/etc/ld.so.preload", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib/arm-linux-gnueabihf/libarmmem.so", O_RDONLY|O_CLOEXEC) = 3
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib/arm-linux-gnueabihf/libprocps.so.3", O_RDONLY|O_CLOEXEC) = 3
open("/lib/arm-linux-gnueabihf/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/lib/arm-linux-gnueabihf/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) = 3
open("/proc/self/stat", O_RDONLY)       = 3
open("/proc/uptime", O_RDONLY)          = 3
open("/proc/sys/kernel/pid_max", O_RDONLY) = 4
open("/proc/meminfo", O_RDONLY)         = 4
open("/proc/1/stat", O_RDONLY)          = 6
open("/proc/1/status", O_RDONLY)        = 6
open("/proc/2/stat", O_RDONLY)          = 6
open("/proc/2/status", O_RDONLY)        = 6
open("/proc/3/stat", O_RDONLY)          = 6
open("/proc/3/status", O_RDONLY)        = 6
...

Nifty, huh? strace is another one of those cool Linux utilites. It can be used to trace system calls (the essential interface between programs and the operating system) in a running program. More on that in a future post. :)