File I/O and the Unix Philosophy of Universality
A practical guide to File I/O in unix.
File I/O and the Unix Philosophy of Universality
One of the most important ideas in Unix is:
Many things can be treated as files.
Regular files, devices, terminals, pipes, sockets, and other resources can often be accessed through the same basic file I/O interface.
This idea is called the universality of the file I/O interface.
1. File Descriptors
When a process opens a resource, the kernel returns a file descriptor (FD).
A file descriptor is simply a small integer that identifies an open resource inside a process.
For example:
1
int fd = open("hello.txt", O_RDONLY);
If open() returns 3, then 3 is the file descriptor for that open file.
Unix processes normally start with:
1
2
3
0 → stdin
1 → stdout
2 → stderr
For example:
1
2
read(0, buffer, 100);
write(1, buffer, 100);
These operations work with the terminal, but the same functions can operate on regular files, pipes, and many other resources.
2. The Basic File I/O Functions
The core Unix file-I/O system calls are:
| Function | Purpose |
|---|---|
open() | Open or create a file |
read() | Read bytes |
write() | Write bytes |
lseek() | Change the file offset |
close() | Close the file descriptor |
A typical sequence is:
1
2
3
4
5
open()
↓
read() / write() / lseek()
↓
close()
For example:
1
2
3
4
5
6
7
8
9
int fd = open("file.txt", O_RDWR);
read(fd, buffer, 100);
lseek(fd, 0, SEEK_SET);
write(fd, buffer, 100);
close(fd);
These are system calls: the program requests that the kernel perform an operation on its behalf.
3. Why Unix File I/O Is Universal
Consider:
1
2
3
4
5
Regular file
Terminal
Pipe
Device
Socket
Many of these resources can be represented by a file descriptor.
Therefore, programs can use a common interface:
1
2
3
4
5
6
7
8
File Descriptor
│
┌──────────┼──────────┐
↓ ↓ ↓
File Terminal Pipe
│ │ │
read() read() read()
write() write() write()
For example, the following program doesn’t necessarily care whether fd refers to a file or a pipe:
1
2
3
4
5
6
char buffer[4096];
ssize_t n = read(fd, buffer, sizeof(buffer));
if (n > 0)
write(1, buffer, n);
This is one reason Unix tools can be composed so easily.
For example:
1
cat file.txt | grep hello
cat writes to its standard output, while grep reads from its standard input.
Both sides communicate through the same file-descriptor-based I/O interface.
4. The File Offset
An open file has a current file offset.
Suppose we have:
1
2
3
4
file.txt
0123456789
abcdefghij
Initially:
1
offset = 0
After:
1
read(fd, buffer, 3);
the offset becomes:
1
offset = 3
The next read() starts from byte 3.
You can change the offset using lseek():
1
lseek(fd, 0, SEEK_SET);
This moves the offset back to the beginning.
Other common forms are:
1
2
3
4
5
lseek(fd, 10, SEEK_SET); // offset = 10
lseek(fd, 5, SEEK_CUR); // move 5 bytes forward
lseek(fd, -5, SEEK_END); // 5 bytes before EOF
5. File Offset Is Not the Same as File Size
This is an important distinction.
Suppose:
1
2
file size = 5 bytes
offset = 5
You can do:
1
lseek(fd, 1000, SEEK_SET);
Now:
1
2
file size = 5
offset = 1000
lseek() changed the offset, not the file size.
If you then read:
1
read(fd, buffer, 10);
the result is:
1
0
meaning EOF was reached.
But if you write:
1
write(fd, "X", 1);
the file becomes much larger:
1
2
3
4
5
logical file:
0 1000
|-------------------|X
hole
The new file size is:
1
1001 bytes
6. Sparse Files and Holes
The region between the old EOF and the new data is called a hole.
For example:
1
2
lseek(fd, 1000000, SEEK_SET);
write(fd, "X", 1);
The logical file size becomes:
1
1,000,001 bytes
but the filesystem doesn’t necessarily allocate physical disk blocks for the million-byte gap.
Conceptually:
1
2
3
4
Logical file:
[allocated data][ hole ][data]
ABC 1 MB of zeros XYZ
Reading the hole returns zero bytes:
1
read(hole) → 00 00 00 00 ...
However, those zeros don’t necessarily exist physically on disk.
This is extremely useful for things such as virtual machine disk images.
7. Holes vs. Real Zero Data
A subtle but important question is:
How can we distinguish a hole from actual zero bytes?
Normally, we can’t do it with read().
Suppose these two files logically contain:
1
2
3
4
5
File A:
ABC 00 00 00 XYZ
File B:
ABC [hole] XYZ
A normal read() sees:
1
ABC 00 00 00 XYZ
in both cases.
The distinction is physical storage, not logical contents.
On Linux/filesystems that support it, lseek() provides:
1
2
SEEK_DATA
SEEK_HOLE
For example:
1
off_t hole = lseek(fd, offset, SEEK_HOLE);
This asks the filesystem for the next hole.
Similarly:
1
off_t data = lseek(fd, offset, SEEK_DATA);
asks for the next data region.
This allows programs such as cp to preserve sparse files instead of converting holes into huge sequences of physically stored zeroes.
8. read() and write() Are Not Guaranteed to Process Everything
Another important property of Unix I/O is that:
1
write(fd, buffer, 1000);
does not mean you should always assume 1000 bytes were written.
write() returns the number of bytes actually written:
1
ssize_t n = write(fd, buffer, 1000);
It might return:
1
1000
or potentially less.
Therefore, programs that need to write all bytes generally use a loop:
1
2
3
4
5
6
7
8
while (written < count) {
ssize_t n = write(fd,
buffer + written,
count - written);
if (n > 0)
written += n;
}
The same idea applies to read(): its return value tells you how many bytes were actually obtained.
9. The Important Mental Model
It is useful to separate three concepts:
File descriptor
Identifies an open resource:
1
fd = 3
File offset
Determines where the next I/O operation occurs:
1
offset = 100
File size
Determines the logical size of the file:
1
size = 1000
They are related, but they are not the same thing.
For example:
1
2
3
4
5
6
File
│
┌──────┴──────┐
│ │
file size file offset
1000 500
lseek() normally changes the offset.
write() can change both the contents and the file size.
read() normally changes the offset but does not change the file size.
10. The Unix I/O Philosophy
The power of Unix file I/O comes from providing a small set of general operations:
1
2
3
4
5
open
read
write
lseek
close
instead of requiring a completely different API for every kind of resource.
This enables the Unix philosophy of building small programs that can be connected together:
1
2
3
4
5
6
7
8
9
program A
│
│ stdout
↓
pipe
│
│ stdin
↓
program B
The programs don’t need to know exactly what is on the other side of the file descriptor.
This simple abstraction is one of the foundations of Unix and Linux system programming.