Handling large files on S3 with Laravel

How to upload files

To upload file to S3 bucket we just need to create a new Filesystem instance representing our S3 disk storage; define the path relative to our bucket, where we would like to store desired file; and then upload the file using put() method.

$targetFile = 'hello.txt';          // path where data should be stored
$sourceFile = '/var/log/syslog';    // local path to file to be uploaded

$disk= Storage::disk('s3');
$disk->put($targetFile, file_get_contents($sourceFile));
// or
$disk->getDriver()->putStream($targetFile, file_get_contents($sourceFile));

Method put() of the IlluminateContractsFilesystemFilesystem interface accepts 3 parameters:

• Destination file path,
• Content to be stored in this file,
• and options

Third parameters used to set visibility of the file and it is optional. But it could be used as:

$disk->put($targetFile, file_get_contents($sourceFile), ['visibility' => 'private']);
// or just
$disk->put($targetFile, file_get_contents($sourceFile), 'private');

This is a good way to handle small files, but you should note that file_get_contents will load the entire file into RAM before sending it to S3 storage. This could be an issue for large files. To prevent errors or exceptions for large files, we will use streams to upload and download files.

What is stream

A stream is a resource object which exhibits streamable behavior. It can be read from or written to in a linear fashion.

How to upload using stream

To proper upload big files you should use PHP streams. That's how you could do it:

$disk = Storage::disk('s3');
$disk->put($targetFile, fopen($sourceFile, 'r+'));

PHP will only require a few MB of RAM even if you upload a file of several GB.

How to read stream

To download a file from S3 storage to the local file system you can use readStream() method. This method returns a resource or boolean false, so it is handy to check results of execution before reading stream. Locally file from the stream can be stored as shown below:

$sourceFileOnS3 = 'hello.txt';

$disk = Storage::disk('s3');
$stream = $disk
    ->getDriver()
    ->readStream($sourceFileOnS3);

is_resource($stream) && file_put_contents($targetFile, stream_get_contents($stream), FILE_APPEND);

You can even use streams to copy file from one disk to another without touching the local filesystem:

$s3 = Storage::disk('s3');
$sftp = Storage::disk('sftp');

$stream = $s3->getDriver()->readStream($sourceFile);

is_resource($stream) && $sftp->put($targetFile, $stream);

TL;DR: Conclusion

To upload files with the large size you should always rely on streams usage, otherwise, your RAM could be exhausted.