Wunschkonzert, Ponyhof und Abenteuerspielplatz

It's amazing how far Linux has come without providing for proper file locking that works and is usable from userspace. A little overview why file locking is still in a very sad state:

To begin with, there's a plethora of APIs, and all of them are awful:

POSIX File locking as available with fcntl(F_SET_LK): the POSIX locking API is the most portable one and in theory works across NFS. It can do byte-range locking. So much on the good side. On the bad side there's a lot more however: locks are bound to processes, not file descriptors. That means that this logic cannot be used in threaded environments unless combined with a process-local mutex. This is hard to get right, especially in libraries that do not know the environment they are run in, i.e. whether they are used in threaded environments or not. The worst part however is that POSIX locks are automatically released if a process calls close() on any (!) of its open file descriptors for that file. That means that when one part of a program locks a file and another by coincidence accesses it too for a short time, the first part's lock will be broken and it won't be notified about that. Modern software tends to load big frameworks (such as Gtk+ or Qt) into memory as well as arbitrary modules via mechanisms such as NSS, PAM, gvfs, GTK_MODULES, Apache modules, GStreamer modules where one module seldom can control what another module in the same process does or accesses. The effect of this is that POSIX locks are unusable in any non-trivial program where it cannot be ensured that a file that is locked is never accessed by any other part of the process at the same time. Example: a user managing daemon wants to write /etc/passwd and locks the file for that. At the same time in another thread (or from a stack frame further down) something calls getpwuid() which internally accesses /etc/passwd and causes the lock to be released, the first thread (or stack frame) not knowing that. Furthermore should two threads use the locking fcntl()s on the same file they will interfere with each other's locks and reset the locking ranges and flags of each other. On top of that locking cannot be used on any file that is publicly accessible (i.e. has the R bit set for groups/others, i.e. more access bits on than 0600), because that would otherwise effectively give arbitrary users a way to indefinitely block execution of any process (regardless of the UID it is running under) that wants to access and lock the file. This is generally not an acceptable security risk. Finally, while POSIX file locks are supposedly NFS-safe they not always really are as there are still many NFS implementations around where locking is not properly implemented, and NFS tends to be used in heterogenous networks. The biggest problem about this is that there is no way to properly detect whether file locking works on a specific NFS mount (or any mount) or not.
The other API for POSIX file locks: lockf() is another API for the same mechanism and suffers by the same problems. One wonders why there are two APIs for the same messed up interface.
BSD locking based on flock(). The semantics of this kind of locking are much nicer than for POSIX locking: locks are bound to file descriptors, not processes. This kind of locking can hence be used safely between threads and can even be inherited across fork() and exec(). Locks are only automatically broken on the close() call for the one file descriptor they were created with (or the last duplicate of it). On the other hand this kind of locking does not offer byte-range locking and suffers by the same security problems as POSIX locking, and works on even less cases on NFS than POSIX locking (i.e. on BSD and Linux < 2.6.12 they were NOPs returning success). And since BSD locking is not as portable as POSIX locking this is sometimes an unsafe choice. Some OSes even find it funny to make flock() and fcntl(F_SET_LK) control the same locks. Linux treats them independently -- except for the cases where it doesn't: on Linux NFS they are transparently converted to POSIX locks, too now. What a chaos!
Mandatory locking is available too. It's based on the POSIX locking API but not portable in itself. It's dangerous business and should generally be avoided in cleanly written software.
Traditional lock file based file locking. This is how things where done traditionally, based around known atomicity guarantees of certain basic file system operations. It's a cumbersome thing, and requires polling of the file system to get notifications when a lock is released. Also, On Linux NFS < 2.6.5 it doesn't work properly, since O_EXCL isn't atomic there. And of course the client cannot really know what the server is running, so again this brokeness is not detectable.

The Disappointing Summary

File locking on Linux is just broken. The broken semantics of POSIX locking show that the designers of this API apparently never have tried to actually use it in real software. It smells a lot like an interface that kernel people thought makes sense but in reality doesn't when you try to use it from userspace.

Here's a list of places where you shouldn't use file locking due to the problems shown above: If you want to lock a file in $HOME, forget about it as $HOME might be NFS and locks generally are not reliable there. The same applies to every other file system that might be shared across the network. If the file you want to lock is accessible to more than your own user (i.e. an access mode > 0700), forget about locking, it would allow others to block your application indefinitely. If your program is non-trivial or threaded or uses a framework such as Gtk+ or Qt or any of the module-based APIs such as NSS, PAM, ... forget about about POSIX locking. If you care about portability, don't use file locking.

Or to turn this around, the only case where it is kind of safe to use file locking is in trivial applications where portability is not key and by using BSD locking on a file system where you can rely that it is local and on files inaccessible to others. Of course, that doesn't leave much, except for private files in /tmp for trivial user applications.

Or in one sentence: in its current state Linux file locking is unusable.

And that is a shame.

Update: Check out the follow-up story on this topic.

posted at: 19:38 | path: /projects | permanent link to this entry | 21 comments

Posted by gw666 at Sat Jun 26 20:07:47 2010
Google Chrome tries to lock a file in $HOME and fails immediately when using an AFS home.

Posted by John at Sat Jun 26 20:09:48 2010
You've obviously thought a great deal about this problem. Do you conceive of a better API?

Posted by Olivier at Sat Jun 26 20:47:23 2010
Then fix it! You could add an API to discover if BSD locking is available for example.. and add some way for the file owner to break a lock... But that probably requires implementing something like revoke(2).

Posted by Lennart at Sat Jun 26 20:48:33 2010
John: well, the BSD locking semantics are mostly OK. If this API would get extended to fix the security problem, to add byte range locking and to make it detectable whether locking works on a specific FS (i.e. via correct trustable error codes and via fpathconf(), where a reply "DONTKNOW" might be acceptable for the cases where even the kernel doesn't know) then a lot would already be won.

Posted by Mark at Sat Jun 26 22:14:38 2010
It's too bad linux switched to broken posix thread implementation.

Things were so much better in the initial implementation where PID=TID.

IMO

Posted by David at Sat Jun 26 22:15:22 2010
gw666: Actually, that's not why it fails. I believe it locks via the "traditional file-based locking" setup by creating a symlink (SingletonLock) whose target tries to identifies the process. The reason it fails on AFS is because it also creates a socket to communicate with that process and AFS does not allow socket files. I'm going to see if I can move the socket itself elsewhere sometime this summer.

Posted by Tor Lillqvist at Sat Jun 26 22:17:32 2010
You forgot one type of locking: The "share mode" (also known as the complement name, "deny mode") kind, as known on Windows and thus in the SMB protocol. Something similar exists in BSD, and also not very well documented (and possibly not that well implemented, and not intended to be used from normal code?) in Linux (see LOCK_MAND in bits/fcntl.h, and /usr/src/linux/fs/locks.c).

This is a totally different kind of locking, as it applies to the opening of a file. There is naturally support for this in libsamba, but unfortunately no way to get at that then in GIO.

Posted by Lennart at Sat Jun 26 22:35:54 2010
Tor, indeed. I wasn't aware of those. However, to me they read as just another kind of mandatory locking. Which means it is probably something to stay away from...

Posted by Anonymous at Sun Jun 27 11:26:56 2010
Or, you could write off people who put key filesystems on broken NFS servers, assume that NFS has working lock support, proceed accordingly, and reassign any resulting bug report to the NFS server. Bugs should get fixed.

Posted by Lennart at Sun Jun 27 15:33:29 2010
Anonymous: well, I am not against telling people with broken NFS implementations to fix their stuff, but for that we'd need to be able to detect this case. And currently we can't. Things will just sometimes work, sometimes not, and issues might become visible only long after the actual problem happened.

Posted by Kjetil Matheussen at Mon Jun 28 13:07:24 2010
Are you sure there are no general way to create
a file lock, which don't require atomic O_EXCL?

For instance, I'm pretty sure the code below
should be pretty safe, but there are maybe
better ways, which also includes some kind of
formal proof.

void unlock(char *filename){
void *handler = open(filename);
write_access_num(handler,0);
close(handler);
}

bool trylock(char *filename){
void *handler = open(filename);

int access_num=get_number_from_handler(handler);

if(access_num != 0){
close(handler);
return false;
}

for(int i=1;i<1000;i++){
write_number_to_handler(handler,i);
if(i%2==0)
usleep(rand()%100);
if(read_number_from_handler(handler)!=i)
return false;
}

return true;
}

void lock(char *filename){
while(trylock(filename)==false)
usleep(500);
}

Posted by Kjetil at Mon Jun 28 13:14:58 2010
Oops, that for loop probably works better like this:

for(int i=1;i<1000;i++){

int random_number=rand();

write_number_to_handler(handler,rand);

if(i%2==0)

usleep(rand()%100);

if(read_number_from_handler(handler)!=rand)

return false;

}

Well, most likely there's other things wrong with it too. :-)

Posted by Jeremy Allison at Mon Jun 28 18:46:53 2010
Some history:

http://www.samba.org/samba/news/articles/low_point/tale_two_stds_os2.html

Read the "First Implementation Past the Post" section for an explanation of why POSIX locking is broken in this way. No, it's not a Linux issue...

Jeremy.

Posted by Lennart at Mon Jun 28 19:14:26 2010
Jeremy, very interesting read indeed. Thanks for the pointer!

Posted by Karel Zak at Tue Jun 29 09:47:39 2010
The another problem is a timeout for waiting on the lock. If I good remember than for example fcntl(F_SETLKW) could be interrupted by a signal only...(e.g. SIGALRM) ... it does not seem like an elegant solution especially for shared libraries.

Posted by gdamjan at Wed Jun 30 02:55:26 2010
What's the problem with Linux mandatory locking?

Posted by Nix at Thu Jul 1 18:12:16 2010
Mandatory locking in general allows anyone to DoS-attack anyone else (unless it is constrained to only affect the same uid, which is so rare I've never heard of it actually being implemented). Read locking is particularly bad (you want to read this? you got to wait). Worst of all is when these locks affect root, because now you can DoS-attack your sysadmin's backups.

Windows 'solves' this by having an entire parallel set of file-manipulation APIs (the Backup API) whose sole real purpose is to allow access to files bypassing ACLs, ownerships, and locking (since then it has had other things hurled into it, e.g. it is the only way to create hardlinks in Windows). This is really not an approach we should countenance in Linux.

(Oh, and Lennart? you are too charitable. There is one situation in which it is acceptable to use dotlocking/whateverlocking in $HOME, though: if you've tested at runtime to make sure this locking works. This is really difficult and basically nobody ever does it. procmail at least tests to find a working locking mechanism, but it tests at compile-time: not so terribly useful in these days of distributions.)

Posted by Lennart at Thu Jul 1 19:35:09 2010
Nix: such an auto-detection cannot work, see the follow-up story on this story: basically it is impossible to guarantee that everybody who accesses $HOME has the same feature set available, but that is very often not the case, for example, when one side uses NFS and the other accesses the ext3 partition natively. One of the points I wanted to point out above is the fact that it is not possible to auto-detect anything.

Posted by Bruce Fields at Mon Jul 5 06:23:50 2010
Also, Linux's implementation of mandatory locking isn't really correct: if I remember correctly, it checks for the presence of a conflicting mandatory lock once before doing the io, but doesn't hold any lock over the io operation; so a race between the io and a new incoming lock request is theoretically possible.

Posted by Lennie at Sat Jul 10 02:22:00 2010
This might be really stupid or smart, but I ones had to make sure I had only one process running and had some problems with locking. So what I did instead was create a Unix-domain-socket and try to connect to it in a non-blocking fashion. If I could connect to it, without a failure (connection refused, etc.) then something was already running. Otherwise I could just continue with starting my process and try to create a socket-server on that same socket.

It was probably really stupid and possible would get into some kind of race condition, but it worked better then the locking. Which is pretty sad really. :-(

Posted by Jason at Tue Sep 14 04:04:38 2010
Add into the mix Perl's lacking and confusing implementation of file locking. Perl's flock function will attempt to use these system calls in order, resorting to the next until one is supported: flock2, fcntl2, lockfd3. Talk about portability issues. They reference a flag that can used when building Perl, Ud_flock... But this is poorly documented and error prone. Not to metion, it requires a rebuild of Perl!?!

Leave a Comment:

It should be obvious but in case it isn't: the opinions reflected here are my own. They are not the views of my employer, or Ronald McDonald, or anyone else.

Please note that I take the liberty to delete any comments posted here that I deem inappropriate, off-topic, or insulting. And I excercise this liberty quite agressively. So yes, if you comment here, I might censor you. If you don't want to be censored your are welcome to comment on your own blog instead.

レナート Wunschkonzert, Ponyhof und Abenteuerspielplatz ﻟﻴﻨﺎﺭﺕ

Sat, 26 Jun 2010

On the Brokenness of File Locking

The Disappointing Summary