C: Handling large files (>2G) with standard c

Articles may may have files attached at the end of the post

Once upon a time, file size was capped to 2G because of 2^31 - 1 = 2G. Has such, open will fail on files bigger than 2G.

this tutorial will show how to get standard C to handle large file.

As per man open:

(LFS) Allow files whose sizes cannot be represented in an off_t (but can be represented in an off64_t) to be opened. The _LARGEFILE64_SOURCE macro must be defined
in order to obtain this definition. Setting the _FILE_OFFSET_BITS feature test macro to 64 (rather than using O_LARGEFILE) is the preferred method of obtaining
method of accessing large files on 32-bit systems (see feature_test_macros(7)).

thus, we will need to compile out code with -D_FILE_OFFSET_BITS=64 in order to get open to succeed.

Let's create a 2.1G file:

$ dd if=/dev/zero of=2.1Gfile bs=1M count=2150
2150+0 records in
2150+0 records out
2254438400 bytes (2.3 GB) copied, 79.7822 s, 28.3 MB/s

And a code sample to open this file:

  1. #include <stdio.h>
  2. #include <fcntl.h>
  5. int main(int argc, char **argv){
  6.   char *fname;
  7.   int fd, hasread;
  8.   if(argc > 1){
  9.     fname = argv[1];
  10.     fprintf(stdout, "Filename is %s\n", fname);
  11.   }
  12.   fd = open(fname, O_RDONLY);
  13.   if(fd == -1 ){
  14.     perror("Could not open file");
  15.     return 1;
  16.   }
  17.   fprintf(stdout, "File %s was open success fully\n", fname);
  18.   close(fd);
  19.   return 0;
  20. }

Now, compile it the old good way and try to open 2.1Gfile:

$ gcc largefiles.c
$ ./a.out 2.1Gfile
Filename is 2.1Gfile
Could not open file: Value too large for defined data type

and, when compiling it with -D_FILE_OFFSET_BITS=64 :

$ gcc -D_FILE_OFFSET_BITS=64 largefiles.c
$ ./a.out 2.1Gfile
Filename is 2.1Gfile
File 2.1Gfile was open success fully

largefiles.c387 bytes