Native file system watcher for Linux
April 15th, 2010 by Roman ShevchenkoIf you’re an IntelliJ user working under Linux you’ve probably seen that boring “Synchronizing files…” spinning icon in a left corner of a status bar. It is there because for an intelligent IDE it is a must to be in the know about any external changes in files it working with - e.g. changes made by VCS, or build tools, or code generators etc. On Windows and Mac OS X native file system watchers used to facilitate this task but on Linux the only option was to recursively scan directory tree. Now you’re welcome to give a try to native file system watcher for Linux.
Prerequisites
File system watcher requires inotify(7) facility. It is in mainstream kernel for more than two years (since 2.6.13, and in glibc since 2.4) so chances are your distribution don’t missing it. The sign of inotify availability in a system is a presence of /proc/sys/fs/inotify/ directory.
Download and setup
File system watcher is a single binary executable (fsnotifier) and can be downloaded directly from our Git repository. It should be named ‘fsnotifier’, placed into bin/ directory of your IDE and granted execution rights. Inotify requires a “watch handle” to be set for each directory it monitors. Unfortunately, the default limit of watch handles may not be enough for reasonably sized projects (e.g. IntelliJ IDEA sources contain 30000+ directories). The current limit can be verified by executing:
cat /proc/sys/fs/inotify/max_user_watches
It can be raised by adding following line to the /etc/sysctl.conf file:
fs.inotify.max_user_watches = 524288
… and issuing this command to apply the change:
sudo sysctl -p
64-bit systems
If your system is strictly 64-bit (i.e. doesn’t contains 32-bit runtime libraries in /lib32 directory) you should download 64-bit version here. Rename downloaded file to “fsnotifier”, rest of setup is same.
Tags: Linux
April 15th, 2010 at 2:33 pm
How do I know if Idea is using fsnotifier ?
April 15th, 2010 at 2:50 pm
Great work!
April 15th, 2010 at 4:01 pm
I’ve build a 64bit version from the sources, seems to work.
April 15th, 2010 at 5:06 pm
@Dragisa You can check idea.log.
@Stefan It runs in a separate process so there is no need for it to be 64bit.
April 16th, 2010 at 2:32 am
@Dragisa
Try using:
ps axf
IDEA Java process should have subprocess fsnotifier.
Regards,
Oleg
April 16th, 2010 at 3:58 am
I did exactly what you wrote, downloaded it, placed it in idea’s bin/ dir and gave it +x permissions. But it doesn’t seem to start, ps aux doesn’t show any child processes for idea’s java process, and idea.log says:
–snip–
2010-04-16 10:51:58,198 [ 16139] INFO - api.vfs.impl.local.FileWatcher - Native file watcher failed to startup.
–snip–
Something’s wrong…
April 16th, 2010 at 7:43 am
Does it work with the latest RubyMine build as well?
April 16th, 2010 at 8:11 am
I had the right max_user_watches set already, and if I run “ps axf”, I don’t see any subprocess for IntelliJ.
Carl
April 16th, 2010 at 1:35 pm
Fantastic. It’s great to see you folks paying attention to improving performance like this.
April 18th, 2010 at 12:52 pm
Awesome! Is this targetted to 9.0.2?
April 19th, 2010 at 5:01 am
This is exactly what I need. Unfortunately it is not working.
I have the lib installed and I’ve ensure the file is executable. The only thing I get in the log is this:
2010-04-19 10:57:08,338 [ 6781] INFO - api.vfs.impl.local.FileWatcher - Native file watcher failed to startup.
Any ideas?
April 20th, 2010 at 2:59 am
How can I get more information about why it failed?
2010-04-20 09:18:50,985 [ 10281] INFO - api.vfs.impl.local.FileWatcher - Native file watcher failed to startup.
Running:
Linux 2.6.30.10-105.2.23.fc11.x86_64 #1 SMP Thu Feb 11 07:06:34 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux (Fedora 11)
java version “1.6.0_18″
Java(TM) SE Runtime Environment (build 1.6.0_18-b07)
Java HotSpot(TM) 64-Bit Server VM (build 16.0-b13, mixed mode)
April 21st, 2010 at 4:27 am
@marp, @Carl, @Nick, @WarnerJan debugging instructions is here: http://confluence.jetbrains.net/display/~roman.shevchenko/Debugging+native+file+system+watcher+for+Linux
@Gleb yes, it works with any product based on IntelliJ platform.
@Gus it should work with previous versions too.
April 21st, 2010 at 7:36 am
@trespasserw: I followed the debugging instructions, but I don’t get any logging output in any file below /var/log. Even when using level “debug”.
April 21st, 2010 at 10:04 am
@trespasserw: +1 for marp’s comment.
April 21st, 2010 at 10:19 am
Debugging instructions and watcher itself were updated.
April 21st, 2010 at 1:13 pm
I don’t know if it matters, but I named the file fsnotifier in the bin directory…not the original name it downloaded as.
I can see it running as a subprocess…..
April 21st, 2010 at 4:45 pm
Well, I also named the file ‘fsnotifier’ in the bin directory. But like the others, I don’t get an fsnotifier subprocess under neither IDEA 9.0.1 nor 9.0.2
April 21st, 2010 at 4:54 pm
OK, forget my last comment. Thought I had given fsnotifier execute permissions, but that was not the case. After running chmod +x fsnotifier it is started by IDEA.
April 22nd, 2010 at 2:18 am
Great, now I got it working (in IDEA 9.0.2). The key was to rename the downloaded file to “fsnotfier”, otherwise it doesn’t launch.
@trespasserw: Perhaps you can name the downloaded artifact correctly per default, so the user doesn’t have to rename it.
April 22nd, 2010 at 2:19 am
Regarding my last comment: of course it’s “fsnotifier”…
April 22nd, 2010 at 5:56 am
Great work guys, when will this be included in the default product download ?
April 23rd, 2010 at 3:40 am
Hmm, still no luck. If I run this on the commandline I get:
[root@pcwj bin]# ./fsnotifier
bash: ./fsnotifier: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
Apparently this does not play nice with 64 bit Linux. Any thoughts on how to fix this?
April 23rd, 2010 at 4:25 am
@WarnerJan It should work well. But may be you system misses 32-bit glibc. What do “file ./fsnotifier” and “ldd ./fsnotifier” say?
April 27th, 2010 at 8:31 am
[root@pcwj bin]# ./fsnotifier
bash: ./fsnotifier: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
[root@pcwj bin]# ldd fsnotifier
not a dynamic executable
April 27th, 2010 at 1:09 pm
@WarnerJan I’ve updated the post to include link on a 64-bit version.
May 5th, 2010 at 4:38 am
The fsnotifier –selftest says that my home directory is unwatchable. Why ?
May 12th, 2010 at 9:29 am
Sorry, been on holiday for a while… Just notifiying (no pun intended :)) that it works.
Thanks a lot!
5649 2010-05-12 16:27:52,196 [ 2602] INFO - api.vfs.impl.local.FileWatcher - Native file watcher is operational.
May 21st, 2010 at 5:49 am
@Hugo (sorry for the late answer) when it will be considered stable enough
@jeanpaul if there are not enough watch descriptors for a tree behind it, or if it is on network file system - it is hard to guess without full self-test log
June 30th, 2010 at 2:05 am
My files are located on a network file system.
Is this a limitation in inotify?
June 30th, 2010 at 3:01 am
@jeanpaul
It is. Network file systems don’t produce file change notifications. May be not all but at least all of those in wide use.
June 30th, 2010 at 5:52 am
Too bad. It takes “forever” for IDEA to check all files for changes before running my application.
June 30th, 2010 at 6:36 am
@jeanpaul
You can disable automatic synchronization (File | Settings | General).
July 15th, 2010 at 3:42 pm
I am having an odd situation with fsnotifier, here is my scenario:
my source code is located on an SSD HardDrive formatted using ext3 filesystem
my IntelliJ9.0.2 is running on another harddrive alongside my operating system also formatted to ext3
I am using subversion as my VCS and when I run an update from the command line, it doesn’t update the file contents inside of intelliJ. Instead it puts a little yellow bar at the top of the file that says “Outdated version… Show Diff | Update Project”
Then when I try to make changes to the file, it comes up with an alert stating the the file system version is different than the version in memory, do I want to load the FS version or keep the memory version.
My question is: Why are these files not automatically updating in IntelliJ when I do the update from the command line.
I checked if the fsnotifier was running and it is. I also did the test on fsnotifier to see if it was running correctly and it is as well.
Any suggestion?
July 15th, 2010 at 3:56 pm
Also to note, that when I click on the ‘Update Project’ link from the yellow bar at the top of the file, it sits for a while, svn updating the already updated projects, then after its done, the file STILL doesnt have the changes from the file system.
The only method I found that works is to close and reopen intelliJ
July 23rd, 2010 at 8:44 am
@Ray please run a short session with fsnotifier log level set to DEBUG and then attach a log to the abovementioned page.
July 26th, 2010 at 1:14 pm
I attached the log to http://confluence.jetbrains.net/display/~roman.shevchenko/Debugging+native+file+system+watcher+for+Linux
July 27th, 2010 at 3:20 am
@Ray I’m sorry but it’s impossible to say what’s wrong from short log. Can you please make a log with the level set to DEBUG?
July 30th, 2010 at 7:10 pm
Hi trespasserw, this sounds ilke a great feature, but I’ve tried it a few times, and it just doesn’t want to play ball.
Does it only watch the content roots of my project, or everything it can find under some location, such as where my project files are? The log below suggests it might be the latter, which is a problem - I have many Gbytes of stuff under there it should not look at. Shouldn’t it strictly look only at my content roots?
Also, how do I know if it’s working? It is running, but when I edit a file by hand it does not update in the IDE as expected. I have synchronization set to manual in the IDE - do I have to change that setting?
If I run it at INFO level, I see this in the log: Should I see something at INFO level when I edit a file? I don’t see anything. Also, there’s no mention in the log of any of my content roots, though the proj dir sits above them. Should I see the content roots listed here?
Repeated messages about dirs I don’t care about, eg,
Jul 30 23:41:57 marsbar fsnotifier[1082]: unwatchable: /proc
Jul 30 23:41:57 marsbar fsnotifier[1082]: unwatchable: /sys
Jul 30 23:41:57 marsbar fsnotifier[1082]: unwatchable: /sys/fs/fuse/connections
Strange sequences like this:
Jul 30 23:42:06 marsbar fsnotifier[1082]: unregistering root: /tmpfs/intellij9/system
Jul 30 23:42:06 marsbar fsnotifier[1082]: registering root: /tmpfs/intellij9/system
(IntelliJ sys dir - why does it even look here?)
Registration of some of my library classes like this:
Jul 30 23:42:06 marsbar fsnotifier[1082]: registering root: /usr/local/mwr/intellic-cache/marsbar/libs/mylib/classes
Then it unregisters the same thing:
Jul 30 23:42:06 marsbar fsnotifier[1082]: updating roots (curr:2, new:6)
Jul 30 23:42:06 marsbar fsnotifier[1082]: unregistering root: /usr/local/mwr/intellic-cache/marsbar/libs/mylib/classes (library classes)
More stuff that looks weird, again, why is it looking at system dir?:
Jul 30 23:42:06 marsbar fsnotifier[1082]: unregistering root: /tmpfs/intellij9/system
Jul 30 23:42:06 marsbar fsnotifier[1082]: registering root: /tmpfs/intellij9/system
My project dir, where the project files are:
Jul 30 23:42:06 marsbar fsnotifier[1082]: registering root: /usr/local/mwr/myproj
Intellij’s own build output location (why?):
Jul 30 23:42:06 marsbar fsnotifier[1082]: registering root: /usr/local/mwr/intellic-cache/marsbar/intellij_classes
My libs again:
Jul 30 23:42:06 marsbar fsnotifier[1082]: registering root: /usr/local/mwr/intellic-cache/marsbar/libs/mylib/src
JDK
Jul 30 23:42:06 marsbar fsnotifier[1082]: registering root: /opt/jdk1.6.0_20_32bit
More apparent redundancy:
Jul 30 23:42:06 marsbar fsnotifier[1082]: registering root: /usr/local/mwr/intellic-cache/marsbar/libs/mylib/classes
And a whole lot more stuff I shouldn’t care about:
Jul 30 23:42:06 marsbar fsnotifier[1082]: unwatchable: /proc
Jul 30 23:42:06 marsbar fsnotifier[1082]: unwatchable: /sys
Jul 30 23:42:06 marsbar fsnotifier[1082]: unwatchable: /sys/fs/fuse/connections
Jul 30 23:42:06 marsbar fsnotifier[1082]: unwatchable: /sys/kernel/debug
Jul 30 23:42:06 marsbar fsnotifier[1082]: unwatchable: /sys/kernel/security
etc…
August 1st, 2010 at 1:53 am
@Mark
Yes, you have to enable “Synchronize files on frame activation” option.
You’ve seen all that messages in log - it means the thing is working. In order to be absolutely sure you can just repeat quick test with editing file outside IDEA.
File modification events are only reported at DEBUG level.
IDEA keeps lots of important stuff in it’s system dir and wants to be sure noone touched it. As for external (outside of project root) libs and JDK - one can upgrade any of this externally and IDEA should be aware of changes to provide e.g. correct completion.
Unwatchable dirs are usually mount points for special and network file systems which don’t generate file modification events.
Registering/unregistering sequence is a sign of suboptimal algorithm, have to do some optimization.
August 2nd, 2010 at 4:15 pm
Confirmed - it works with sync on frame activation. But we already had that option, so what does this buy us? Is it just faster/more efficient to detect changed files? The biggest cost I notice is indexing, but every little bit helps, for sure.
Thanks for the great answer, much appreciated.
August 3rd, 2010 at 1:01 am
@Mark
It is faster, especially on big projects. Without native file watcher IDEA deep scans directory tree when looking for changes.