2

A 1994 book called 'The Design and Implementation of a Log-structured file system' contains the claim "We have implemented a prototype log-structured file system called Sprite LFS; it outperforms current Unix file systems by an order of magnitude for small-file writes while matching or exceeding Unix performance for reads and large writes." That would seem on the face of it to constitute a compelling case to start using the file system in question, which raises the question of why this did not happen.

The Unix Haters Handbook claims this is because Sprite LFS not only was not compatible with the Unix API, but could not be; that user-visible aspects of the Unix API preclude high performance in any file system that retains compatibility. Granted that this book disclaims impartiality by its very title, I'm still interested in whether they are right.

Did the classic Unix API indeed preclude high-performance file systems like Sprite LFS? If so, what features of the API were responsible for this? (And how was the problem eventually resolved? I haven't seen anyone claim modern Linux file systems lack performance, and I'm under the impression Linux retains a high degree of backward compatibility.)

Improve this question
3
  • 2
    I don't know whatproblem the UHH was talking about, but it should be noted that ZFS (which I believe manages similar performance to Sprite) and was in use in Solaris (which, being a System-V derivative, is much closer to traditional Unix than Linux ever has been) starting from only a handful of years after the UHH was published, so whatever the issue was, it must have been easy to resolve.
    – Jules
    Commented Mar 20, 2018 at 11:12
  • Without a clear definition what "Performance" means in this context, and in which setup it is to be measured, what defines a filesystem in this context and what parts are considered, no clear answer to the question and regarding any of the mentioned claims can be given.
    – Raffzahn
    Commented Mar 20, 2018 at 11:23
  • 2
    It should be noted that "current" filesystems in 1994 were mostly careful-write and didn't include logging or journalling: other than JFS1, I can't think of any mainstream filesystems at that time which included features like that. XFS and UFS didn't come out until 1994, possibly after the book was written.
    – ErikF
    Commented Mar 20, 2018 at 23:04

1 Answer 1

Reset to default
4

The Sprite LFS paper explains some of the performance issues its designers were trying to solve:

  • lack of locality (even if FFS tries to keep individual files’ blocks contiguous on disk, they end up separated from the inode, and often separate from related files);
  • synchronous writes.

The first issue is indeed driven by Unix-like semantics: Unix-like file systems always end up with concepts representing directories, files, and inodes. The second issue has seen common practice move back and forth over the years, and isn’t intricately tied to Unix-like semantics (although the definitions of sync operations do impose some constraints).

A variety of techniques have been used to address locality concerns, starting with block groups in ext2, more “intelligent” pre-allocation (and additional APIs allowing programs to tell the file systems about their allocation patterns), and moving to tree-structured file systems more recently.

Write performance has been addressed using a number of techniques too: for example, dropping fully-synchronous operations in favour of barriers, and using techniques from log-based file systems, re-implemented as the journal in journalling file systems.

As proof that logging file systems aren’t inherently incompatible with Unix-like semantics, you can look at real log-structured file systems which implement POSIX semantics, such as NILFS2. I’m not actually convinced that Sprite LFS itself is incompatible with Unix-like semantics; the paper doesn’t support this, and the UHH doesn’t give enough detail...

Improve this answer
7
  • 4
    Actually the synchronous IO is very much a Posix thing. On other operating systems, (I'm thinking mainly of VMS and NT) you can issue an IO request and then go off and do other things while it completes.
    – JeremyP
    Commented Mar 21, 2018 at 9:54
  • 2
    No it's not the same. Non blocking reads and write still operate synchronously, they just return prematurely if the IO operation can't be done.
    – JeremyP
    Commented Mar 21, 2018 at 10:26
  • 3
    @JeremyP ah yes, you’re right, the real equivalent would be AIO. That said, the synchronicity issues with regards to performance, especially for writes, have been largely dealt with by caches etc. (which is one of the reasons why POSIX AIO never got anywhere).
    – Stephen Kitt
    Commented Mar 21, 2018 at 10:36
  • 4
    And the synchronous model is simpler for the programmer. However, I don't see anything in the paper that precludes implementing the Posix IO model on top of an asynchronous API.
    – JeremyP
    Commented Mar 21, 2018 at 11:00
  • 2
    All modern OSes (NT, VMS, Unix etc.) support both async and synchronous IO. But Unix is synchronous by default, and it permeates the libraries, standard utilities, etc. VMS, by comparison, is largely async by default. As I understand it, the guys behind Unix Haters felt that this was a major design error for scalability.
    – RETRAC
    Commented Dec 15, 2020 at 23:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .