C: Handling large files (>2G) with standard c
Articles may may have files attached at the end of the post
Once upon a time, file size was capped to 2G because of 2^31 - 1 = 2G. Has such, open will fail on files bigger than 2G.
this tutorial will show how to get standard C to handle large file.
As per man open:
O_LARGEFILE
(LFS) Allow files whose sizes cannot be represented in an off_t (but can be represented in an off64_t) to be opened. The _LARGEFILE64_SOURCE macro must be defined
in order to obtain this definition. Setting the _FILE_OFFSET_BITS feature test macro to 64 (rather than using O_LARGEFILE) is the preferred method of obtaining
method of accessing large files on 32-bit systems (see feature_test_macros(7)).
thus, we will need to compile out code with -D_FILE_OFFSET_BITS=64 in order to get open to succeed.
Let's create a 2.1G file:
$ dd if=/dev/zero of=2.1Gfile bs=1M count=2150
2150+0 records in
2150+0 records out
2254438400 bytes (2.3 GB) copied, 79.7822 s, 28.3 MB/s
And a code sample to open this file:
- #include <stdio.h>
- #include <fcntl.h>
- int main(int argc, char **argv){
- char *fname;
- int fd, hasread;
- if(argc > 1){
- fname = argv[1];
- fprintf(stdout, "Filename is %s\n", fname);
- }
- fd = open(fname, O_RDONLY);
- if(fd == -1 ){
- perror("Could not open file");
- return 1;
- }
- fprintf(stdout, "File %s was open success fully\n", fname);
- close(fd);
- return 0;
- }
Now, compile it the old good way and try to open 2.1Gfile:
$ gcc largefiles.c
$ ./a.out 2.1Gfile
Filename is 2.1Gfile
Could not open file: Value too large for defined data type
and, when compiling it with -D_FILE_OFFSET_BITS=64 :
$ gcc -D_FILE_OFFSET_BITS=64 largefiles.c
$ ./a.out 2.1Gfile
Filename is 2.1Gfile
File 2.1Gfile was open success fully
| Attachment | Size |
|---|---|
| largefiles.c | 387 bytes |












