Direct Upload to S3 (with a little help from jQuery) [Updated]

Direct Upload to S3 (with a little help from jQuery) [Updated]

   91 Comments   
http://bit.ly/1O6WQ5E

The contents of this article has been replaced by a PHP Composer package, hope you find it useful.

  View on Github

Whist working on a new web app recently, running on Heroku, we wanted users to be able to upload large files (I’m talking 50mb-150mb) onto our site. We started by having the standard html form post the data to our back-end code, which processed it, etc, etc. This was fine for small files, but large files would hit the maximum request time of 30 seconds for any request on heroku, which then in turn messed up the dyno it was running on.

The solution was to upload the file directly to Amazon’s S3 (Simple Storage) and not to the back-end code at all. This method comes with advantages and disadvantages of course. One of the disadvantages is the small matter of tracking the file once it’s been uploaded. How do you add a file/row to your database to track the file if you aren’t calling any back-end code? We’ll take a look at these problem and how we can bypass it in the tutorial.

Edit 1st Feb 14: Complete code update thanks to comments by Matt Painter and Ian Mackay.

HTML

To upload a file directly to S3 and have the ability to then track it within your app the front-end must have two separate forms (they can look like they’re one, but we need two action locations).

Edit 18th Feb 14: Form action attribute changed from //<?php echo $bucket; ?>.s3.amazonaws.com to //s3-eu-west-1.amazonaws.com/<?php echo $bucket; ?>/ due to a comment by Matt and it not working over HTTPS. Make sure, however, that your S3 bucket is in fact based in: s3-eu-west-1 (or change appropriately).

<!-- Direct Upload to S3 -->
<!-- URL prefix (//) means either HTTP or HTTPS (depending on which is being currently used) -->
<form action="//s3-eu-west-1.amazonaws.com/<?php echo $bucket; ?>/" method="POST" enctype="multipart/form-data" class="direct-upload">

    <!-- We'll specify these variables with PHP -->
    <!-- Note: Order of these is Important -->
    <input type="hidden" name="key" value="${filename}">
    <input type="hidden" name="AWSAccessKeyId" value="<?php echo $accesskey; ?>">
    <input type="hidden" name="acl" value="private">
    <input type="hidden" name="success_action_status" value="201">
    <input type="hidden" name="policy" value="<?php echo $base64Policy; ?>">
    <input type="hidden" name="signature" value="<?php echo $signature; ?>">

    <input type="file" name="file" />

    <!-- Progress Bar to show upload completion percentage -->
    <div class="progress"><div class="bar"></div></div>

</form>

<!-- Used to Track Upload within our App -->
<form action="process-form-data.php" method="POST">
    <input type="hidden" name="upload_original_name" />

    <label for="upload_custom_name">Name:</label><br />
    <input type="text" name="upload_custom_name" id="upload_custom_name" /><br />

    <input type="submit" value="Save" />
</form>

The important bit of this code is in the first form within the hidden input parameters.

key This is the name given to it when saving it on S3, by default it will be ${filename} which will take name it has on upload, but you can give it unique name by hashing it or adding in a prefix folder name.
AWSAccessKeyId This is your AWS access key, given to you through the S3 interface.
acl This is the access control list, which can be private (only giving the owner rights), public-read (giving AllUsers read capabilities), public-read-write (giving AllUsers both read and write capabilities). For a full list see the AWS documentation.
success_action_status This tells S3 what to do on success, it’s possible to also redirect on success but we’ll be using AJAX and XHR requests so we’ll be wanting it to return a successful HTTP status code.
policy This is the upload policy, we’ll talk about a little bit more about later on, but is a base 64 encoded string containing important information related to the upload.
signature The signature is an encoded string which includes both your S3 secret and the upload policy.

PHP

Although the file will upload directly to S3, we need to use a little bit of server-side code (PHP in this case) to generate some necessary hidden input parameters. For any direct upload to succeed on S3, a policy has to be specified — which contains information like the request expiry, acl (whether it’s private or public) and what happens on success.

So with PHP we’ll both specify a base64 encoded policy and add our S3 secret (also encoded).

For this to work you’ll need to change the $bucket variable to an already created S3 bucket; the $accesskey to your S3 key you will have been given; and the $secret variable to your AWS secret. Make sure to not expose your secret to the public.

Edit 10th Sept 14: Update expiry to include timezone & increase to 6 hours, thanks to S. Kristoff’s comment.

<?php

// Fill These In!
$bucket = "";
$accesskey = "";
$secret = "";

$policy = json_encode(array(
    'expiration' => date('Y-m-d\TG:i:s\Z', strtotime('+6 hours')),
    'conditions' => array(
        array(
            'bucket' => $bucket
        ),
        array(
            'acl' => 'private'
        ),
        array(
            'starts-with',
            '$key',
            ''
        ),
        array(
            'success_action_status' => '201'
        )
    )
));
$base64Policy = base64_encode($policy);
$signature = base64_encode(hash_hmac("sha1", $base64Policy, $secret, $raw_output = true));
?>

Edit 7th Jan 15: It’s possible to expand the policy to limit the file size being uploaded, using content-length-range. You can also specify the content-type of the upload file, if you’re sticking to a single file type. These are explained in Daniel Katzan’s comment.


S3 Configuration – Allow CORS

All modern browsers now allow cross origin resource sharing (CORS). This means that by default you can’t post random data from your form to another website. It is possible however to explicitly allow it, so the next step is to go into your S3 account, find the bucket you are working with and open it’s properties. In the properties you’ll see an ‘Add CORS Configuration’ button. Within this you can paste XML allowing your bucket to receive POST data from your domain.

    <CORSConfiguration>
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>POST</AllowedMethod>
<AllowedMethod>PUT</AllowedMethod>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>

jQuery

Now we’ll just want to add the finishing touches by using jQuery, jQuery UI and the jQuery File Upload plugin (which you’ll need to download). The iframe-transport-js and fileupload.js should be the only files needed.

This jQuery was created with help from a tutorial by Pierre Jambet which is geared towards Ruby development.

HTML

<script src="//code.jquery.com/jquery-1.11.2.min.js"></script>
<script src="//code.jquery.com/ui/1.10.4/jquery-ui.js"></script>
<script src="fileupload/jquery.fileupload.js"></script>

JS

$(document).ready( function() {
    $('.direct-upload').each( function() {
        var form = $(this);

        form.fileupload({
            url: form.attr('action'),
            type: 'POST',
            datatype: 'xml',
            add: function (event, data) {

                // Message on unLoad.
                window.onbeforeunload = function() {
                    return 'You have unsaved changes.';
                };

                // Submit
                data.submit();
            },
            send: function(e, data) {
                // onSend
            },
            progress: function(e, data){
                // This is what makes everything really cool, thanks to that callback
                // you can now update the progress bar based on the upload progress.
                var percent = Math.round((data.loaded / data.total) * 100);
                $('.bar').css('width', percent + '%');
            },
            fail: function(e, data) {
                // Remove 'unsaved changes' message.
                window.onbeforeunload = null;
            },
            success: function(data) {
                // onSuccess
            },
            done: function (event, data) {
                // Fill the name field with the file's name.
                $('#upload_original_name').val(data.originalFiles[0].name);
                $('#upload_custom_name').val(data.originalFiles[0].name);
            },
        });
    });
});

CSS (Optional)

We’ll also add in a little CSS to style a custom made progress bar (it’s nothing special) which we’ll interact with through jQuery.

.progress {
    position: relative;
    width: 100%;
    height: 15px;
    background: #C7DA9F;
    border-radius: 10px;
    overflow: hidden;
}

.bar {
    position: absolute;
    top: 0;
    left: 0;
    width: 0;
    height: 15px;
    background: #85C220;
}

This article has been update to use the AWS Signature V4 within a different post.
It’s recommended you use the code from the new article instead.

View Shiny New Article

91 responses to “Direct Upload to S3 (with a little help from jQuery) [Updated]

  1. In general situations where AJAX isn’t being employed, you can remove the two form kludge to identify the uploaded file. If the ‘success_action_redirect’ field is present, the uploaded key is included as a request parameter.

    Also, you can get away with just specifying ‘POST’ in the CORS for the purpose of file uploads.

  2. There’s a typo in the ‘progress’ function in the JS – the ‘loaded’ and ‘total’ properties are members of ‘data’, not ‘e’.

    Also, I’m assuming that ‘#real_file_url ‘ referenced in the JS should be ‘#document_original_name’ (and an ID added to the field in the markup).

    As I just wanted a single button drag and drop upload form, I reduced my secondary kludge form to this single hidden field alone and used jQuery to submit the form after the upload completed. All works beautifully!

    1. Ahh Thanks for that Matt! I’ve changed the e variables to data now. There are many ways to upload to S3, this is just one. It would be great to see some other examples. I’d like to sit down and improve this code sometime too.

  3. I run in chrome but error came:
    XMLHttpRequest cannot load No ‘Access-Control-Allow-Origin’.

    In Firefox its ok, do you know why?

    1. Hi Tran. You’ll need to make sure on your S3 account that you’re either allowing all origins (with *) or adding your individual hosts. This could be one reason why. Can you post an example?

  4. Thanks for the example!

    Just wanted to note that if you’re using PHP 5.1.2 or newer, you don’t need the hmacsha1() or hex2b64() functions – built-ins can get you the same signature in one line:

    $signature = base64_encode(hash_hmac("sha1", $base64Policy, $secret, $raw_output=true));
  5. Hi Edd

    Great piece of coding. I just wanted to let you know about an issue I came across. I my dev machine, all was working well but the application fell over when it went live. The only difference between the two was the live environment was accessible over SSL. In the action part of the form you have:

    <form action="//.s3.amazonaws.com" method="POST" enctype="multipart/form-data" class="direct-upload">

    With // at the start of the URL allowing a switch between protocols, the URL on my live environment became “https://bucket_name.s3.amazonaws.com”. Unfortunately this kept coming back as “Cancelled”. The reason for this was Amazon would have to have an SSL certificate for this subdomain, and all the thousands of others for every bucket they are hosting. The solution to this was to switch the position of the bucket name in the URL to:

    <form action="//s3.amazonaws.com//" method="POST" enctype="multipart/form-data" class="direct-upload">

    Hope this helps anyone else having issues with this.

    1. Hi Edd/Matt,

      I have integrated my form in facebook app. as facebook always loads with https, I have changed my form action as but still I am getting 403 error. can you tell me what might be the wrong.

      with out https I can upload perfectly

  6. Hey Edd, really great article!

    Just wanted to mention the policy should include Content-Type and Content-Disposition. That way when the file is downloaded it has a correct filename, mime type, and set to open in browser or as an attachment.

  7. Real nice tutorial. I’m trying to implement this. But I still have a question. Do I need to add the generated policy everytime to the bucket, and if so how do I do this?

    I tried:

    $this->s3->putBucketPolicy(array(
        // Bucket is required
        'Bucket' => $bucket,
        // Policy is required
        'Policy' => $policy
    ));
    

    The $policy is the same as in your example.

    Thanks for any help 🙂

    1. The policy is uploaded along with your file (after being base64 encoded) and will be unique to that upload.

      It’s uploaded by being added to a hidden input name="policy". If that answers your question?

  8. Hi, great job.
    I’m trying to change a bit jquery file uploader this way.
    Is it possible for you going from to ?
    Thanks a lot

  9. Hello, Thanks for the tutorial, it helps a lot!
    I am using safari and have run into this error, when the page loads.
    TypeError: ‘undefined’ is not a function (evaluating ‘form.fileupload’)

  10. Thanks for your article Edd, that was exactly what I was looking for 🙂

    I only got an issue when uploading larger files (>200MB):
    Here is the Response of my POST Request:

    RequestTimeoutYour socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.

    Any idea how to solve this is welcome!

    1. Hi Thorsten! Unfortunately, after a long google-session I haven’t come up with anything — I haven’t come across this problem myself (but then again, I don’t think I’ve tried uploading 200mb+).

      If you do find a solution, please get let me know and I’ll add it to the post.

  11. Hi Edd, this is a great article. Thank you! Unfortunately I have an Access-Control-Allow-Origin issue. I of course set a corresponding cors policy for my bucket but I always get No ‘Access-Control-Allow-Origin’ header is present on the requested resource. Any ideas ? I am not using cloudfront. I read that this might cause problems because it might cache an old header.

  12. Hi Edd, its working perfectly now. There must have been some mistake in the script I put together. Thank you a lot, I really appreciate it.

  13. This example is great. How can i return the full image s3 path of the uploaded file on the done callback?

  14. Hi, is it also possible to upload two files at the same time ? If yes, what changes are necessary to achieve that ?
    Thanks a lot 😉

  15. I am getting [HTTP/1.1 400 Bad Request 0ms]

    InvalidPolicyDocumentInvalid Policy: Invalid ‘expiration’ value: ‘2014-08-27IST19:45:0319800’0350E3DF57BF9F6F8rJlnf3865R/WyPvT3gBo9a0WXeV9I7SoSneXYD7p+vfrNE61YuMdxBr5PkJR7uP

      1. I had the same problem as Mangal. I changed the expiration so it was in the following format:

        2015-01-01T12:00:00.000Z

        Here is PHP to generate that time stamp format:

        $expire = date("Y-m-d",  strtotime('+6 hours', $now)) . "T" . date("G:i:s.Z",  strtotime('+6 hours', $now)) . "Z";

        You’ll have to play around with the ‘+6 hours’ bits but keep in mind the return will be ‘invalid’ if the format is wrong and ‘expired’ if the value is wrong.

        Thanks for the tutorial Edd!

        -Sam K

          1. Thanks, great article. You can also use the gmdate function to get the date AWS likes:

            gmdate('Y-m-dTG:i:sZ', strtotime('+45 minutes'))
  16. I’m getting a 403 forbidden error on upload, any ideas why this would be happening?

    Ben, what did you do to resolve the 403 you had?

    (I setup an eu bucket to test, and added my domain to the CORS policy)

    1. Found my problem, so just in case anyone else runs into the same error.

      My bucket ACL is set to private, the form ACL hidden field is set to private. After changing $policy array to match it worked.

      'acl' => 'public-read'

      To:

      'acl' => 'private'
  17. I have tried the tutorial but unable to do so I am getting:

    AccessDenied Invalid according to Policy: Policy Condition failed: ["eq", "$acl", "public-read"]

    Any help?

  18. For reference, we were still getting 403 errors when trying to upload as acl public-read, even after following all the advice above. This may have been due to the config on our AWS account. The solution for us was to add a bucket policy for the bucket we wanted to write to, with the following action specified: "Action": "s3:PutObjectAcl"

    Hope this helps someone.

  19. Hi, Thx for the post
    I have a question,
    when the user chooses a file it automatically starts loading it to s3,
    when the uploads finishes, and the status bar is complete, and clicking on the save button
    i get the “you have unsaved changes’
    is it suppose to happen? doesnt suppose to show up only when trying to navigate while uploading?

  20. Hi Edd, I have another question, cause there is no server side here when uploading the files to s3,
    is there anyway I can validate the uploaded files?
    I have JavaScript on the page, making sure the file type uploaded is only zip, and limits his size
    but this could be easily changed on client side,
    is there any option to attach to the created policy additional parameters or somthing like that to limit the file types and sizez to be uploaded?
    btw, there is a little bug with your code I beleive, the policy you are creating has public read, while the form is private, they need to be the same, otherwise you get an error that the policy doesnt match for what you are trying to do.

    1. Hello again 😉 I have in the past used JS to check file types and size, before uploading – with the theory that if they avoid this client side, it isn’t critical. But to do this, or any file checking, with direct upload is an absolute pain… Let me know if you find an answer, I don’t know of one.

      Thanks, by the way. I’ve updated the tutorial to use private by default.

      1. Well, you can add to your policy this:

        array(
            'content-length-range', 0, 500000000
        )
        

        to set a limit for file size. Change this to limit the user to where you want him to upload the file

        array(
            'starts-with',
            '$key',
            'some_path'
        )
        

        And you can add this:

        array(
            "Content-Type" => "application/zip"  
        )
        

        to limit the file type the user can upload, you will need to add a hidden content type to your form with the appropriate type. But it still doesn’t answer my needs entirely, because it doesn’t prevent the user from uploading a non zip file.

  21. Thanks for the excellent work. I got some error and I’m unsure where it is from.

    Object not found!
    The requested URL was not found on this server. The link on the referring page seems to be wrong or outdated. Please inform the author of that page about the error.
    If you think this is a server error, please contact the webmaster.
    Error 404
    localhost
    Apache/2.4.10 (Win32) OpenSSL/1.0.1i PHP/5.6.3

  22. Thanks for such a quick reply. I just got started with php a month ago, so I am a very basic learner. I tried the example zip – inserted the AWS account key, bucket name, secret key, made sure the instance is in the same location – Ireland, and copy-pasted the CORS config. I uploaded the files to xampp and tried to run it but it seems to give me the error above. I suspect it’s because process-form-data.php file is not available for a re-direct. The file also doesn’t show up in the amazon bucket I created.

    One other minor problem is the progress bar goes green after I have selected a file but the tag “file not chosen” still appears.

    Sorry with all these basic questions. I appreciate your help very much! Thank YOU!

    1. Hi there, I’ve just updated the download zip with a slightly newer version – which I’ve just tested and can confirm it works.

      Hope it fixes your problem (keep me updated)

  23. I’m trying to create a website for web-users to upload pdf/word files to my amazon bucket and then allow these links to appear on the website

  24. absolutely brilliant post and codebase — precisely what I was looking for to begin figuring out how to tie in S3 for file storage in a new project Im tackling… question — is it possible to have the file saved to S3 have a unique filename? I’d just probably append epoch timestamp to the end of the file or something along those lines and Im assuming it (the rename) would need to happen immediately after the save is completed. do you have any example php that shows how to pull of the file rename?

    Thanks for the great resource on this!

    1. Hi tamak, I’ve done this in the past by adding a bit of code to the add function within the JS. I’ve also used CryptoJS to create a hash:

      var realName = data.files[0].name;
      var realEnding = realName.split('.').pop();
      var storageName = 'prefix-' + ((new Date).getTime()/1000) + '-' + CryptoJS.SHA1(realName).toString() + '.' + realEnding;
      form.find('input[name=key]').val(storageName);
      

      This is just an example of one way you can do it, the important part is to put it into the add function and change the input[name="key"] value.

  25. Awesome script for direct upload but i want to upload and convert into other format and also created thumbnail if you have any code regrading this very proud for you.

    Thanks

    1. This is tricky, by nature of direct upload, you can do it client-side (usually with a jQuery plugin) *before* uploading. Or you can do it server-side *after* the upload by pulling the file down and working on behind the scenes (background workers or async tasks).

  26. I’ve gotten everything working well, but *only* if I remove the second form object _and_ the reference to jquery.fileupload.js (which I copied locally from github).
    Obviously without the js file I lose the progress bar niftiness, and removing the second form object means I am simply uploading the file and unable to process on my end because after uploading I’m stuck on the S3 page; but otherwise the actual uploading is working.
    Can you point me to any reference that explains how having two form objects is working?

      1. Okay. I’m pretty sure the problem has something to do with the javascript, because the instant I select a file and close the file-selection dialog box, the progress bar changes to red and the file input displays “No file selected.”

        The only change I’ve made to your script is substituting my region (us-west-2) for the one in your script (s3-eu-west-1).

        1. PS: Well, that’s not the _only_ change. Obviously I used my own S3 details. And, like I said, I know that part is working because if I remove the second form object the upload works.

      1. Same thing happens in both Firefox and Chrome. Running locally, yes. I’ll try on a web server now, but I don’t know how it could work; problem seems to be with Jquery …

        1. Yep. Tried it on a web server with same issue. The instant I select a file in the file dialog modal, the file input displays “No file selected.” and the progress bar turns red.

        2. If I remove all the jQuery and javascript includes, and remove the second form, and add a submit button and a success_action_redirect tag to the form, it works fine.
          The only problem is that it redirects with the response as a $_GET instead of a $_POST, i.e. the values are returned in a querystring.
          I can parse the results of that, of course, but it feels fishy.

        3. Are you able to post an example url? I’m just wondering if it’s a CORS problem… because I don’t understand how having the second form there could affect it. And the JS should be straight forward

  27. I just wanted to drop a quick note here to say that I got everything working with Edd’s help. My problem was that I didn’t have POST as an AllowedMethod in my CORS configuration.
    Thanks for providing this awesome tutorial and a million thanks for the extra help.

  28. Hi Turtle,
    Thanks for your valuable code. You saved my time a lot in integrating S3 uploader in my application. It really works well for us.
    But only one customer is facing an issue in file upload.
    I checked the upload process with them, file upload is started and its upload status changing to 100% and then it is throwing error without sending response.
    Error in console: https://s3.amazonaws.com/mybucket/ net::ERR_EMPTY_RESPONSE Failed to load response data
    Please help me on this.

  29. I’m having a little problem with large files upload (~70mb). I’m always getting a 400 Bad Request response some of the time.

    I’ve tried setting the expiration date 6 hours, 48 hours and 100 hours from now but it doesn’t make any difference.
    Sometimes large files are uploaded without complications but that’s luck.

    I’ve also enabled chunking and set the max chunk filesize to 5mb. The error persists.

    Any idea how one could fix this?
    Many thanks!

  30. Thank you very much for this code!

    I was having issues uploading large files to s3 because they had to be uploaded to my websites server first.

    This solved it!

    However, I wanted to go and a step further and AJAX it.

    So using the theories in this post, I’ve managed to create a fully functioning AJAX variety will works beautifully, even with large files. The nice thing about it, it’s really clean, and actually really simple.

    If you like, I’d be happy to send you the code and some notes if you would like to share it 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *