Software Engineering in WordPress and Musings on the Deep Life

Importing CSV Files into WordPress: Visual Cues, Part 2

This is part three of a series for how to Import CSV Files into WordPress.

As mentioned in the previous post, one of the problems that comes with importing CSV files into WordPress – specifically large files – is giving the user feedback that something is happening while waiting for the process to complete.

But when your’e doing this within the context of a web application, it takes a little bit more work because everything happens in a single process. If you don’t show your users any feedback, then it looks as if the page is just sitting there.

Then again, because the process can take so long, you need to let the user know that something is happening.

Importing CSV Files into WordPress: Visual Cues

 

Within the context of web applications, you can do that by using client-side JavaScript to essentially poll the server and ask for status updates.

Giving Visual Cues in Importing CSV Files into WordPress

In the previous post, I walked through some pseudo-code for how to setup the server-side to store info on the progress of the import process, but – as mentioned – that’s only half of it. That is, we need to be able to have the browser asking for the an update from the server, then update an indicator (whatever type it is that you might chose) whenever the response is received.

Luckily, the standard Ajax API in WordPress makes this pretty easy (though the Heartbeat API is another route that you could take). So assuming that you’ve tracked with the series thus far, here’s how to setup your JavaScript source so that it can poll the server, handle the response, and then update the progress indicator.

1. The JavaScript is Enqueued

For this post, I’m assuming the following:

  • You have a file called `request.js`
  • The file is being enqueued via your theme or plugin

I’ll be showing specific code for what goes into the file, but I’m not going to take the time to walk through the process of creating a JavaScript file and then enqueuing it when it’s already so well documented.

2. Create The Files

Recall in the previous function, I provided a high level function hooked to an Ajax action called import_files. In short, the code is responsible for:

  1. Reading the most recent file that hasn’t been read
  2. Inserting information into the database
  3. Deleting the file that was just read
  4. Updating the number of processed files

One of the major components to this function the final step – that it updates the number of processed files – because, without that, we have no way of tracking the progress.

But before we get to that, we need to make sure that we kick off all of the above. This can be done using just a little bit of JavaScript. As with the rest of the code shared in this tutorial, this is primarily done at a high-level just to give an example of how to structure your code and the process around it.

This particular $.post alone is responsible for kicking off the import file process. When the request completes, it will receive the response value from the server after which we’re free to update the display however we see fit.

In this case, I’ve opted to set my progress bar to 100% and display a message on the screen that reads “Import Complete!”

3. Checking Progress

Of course, there’s activity happening from the time we kick off the import_files routine until the function has completed, right? And this is where the code responsible for updating the progress bar and actually giving visual cues comes into play.

The main idea behind all of this this:

  1. Every so many milliseconds (I’ll say 1000 for 1 second, just to make it easier to read), check the server for the status of the import
  2. Update the progress bar to the value returned from the server. Multiply it by 100 to get a percentage.
  3. If the progress has completed, then move the progress bar to 100% and display the success message.

Remember in the previous post, I shared a small function called get_import_status that will return a floating point decimal number (which is why we have to use JavaScript to multiply it by 100) based on how many files have been parsed out of the total number of files.

If the option responsible for saving the parsed file has been deleted, then it means the job is done. In this case, I’ll send a -1 back to the browser (since anything over 0 is indicative of progress).

Notice that I’ve opted to use $.get here. This doesn’t really matter that much but since I’m simply polling to, you know, get a value, it makes sense.

Note also that I store the value of the timer. This is so that it can be cleared whenever the progress is complete.

4. All Together Now

Putting all of the JavaScript together should look something like this:

Notice that it includes both the timer code for polling the server and the code responsible for kicking off the import.

Permitting that you’ve implemented everything correctly on the server-side, then you should see some type of indication that work is being completed.

Almost Done

At this point, the basic functionality is complete and you should have everything that you need in order to kick off an import process and track it’s progress.

The last thing that needs to happen is that any residual files and records that we’ve written to disk and/or to the database need to be cleaned up. In the final post, I’ll cover how to do that.

6 Comments

  1. Dq

    Thanks for sharing this.

  2. Pete

    So, is this correct?
    1. Your file creation (creating lots of CSV partials) happens via a traditional POST when the form is submitted with a new file.
    2. Your Javascript runs EVERY time the Import page is loaded, and uses AJAX to process any CSV partials that still exist.

    Which means that uploading a CSV and hitting submit essentially just uploads the file, creates the partials and refreshes the page.

    Is that right?

    • Pete

      One other question:

      It looks like you only call import_files() once. Does this mean that your import_files() function is looping through and processing all of the CSV files before returning a response? If so, the server could be engaged in the single import_files() call for a very long time, right? Couldn’t that cause a timeout?

      Or am I misunderstanding incorrectly? =)

      • Jan Beck

        I was wondering the same thing. I’m about to start a project so I am evaluating possibilities and that’s how I found this article. The way I understand it Tom is splitting one big file into smaller files and then imports those in a single php call. Wouldn’t this take more or less the same amount of time as importing the whole file would and cause timeouts? The only advantage being that it would potentially use less memory and avoids larger files.

        In a recent episode of the Apply Filters Podcast the topic was upgrade routines, which can also potentially cause timeouts http://applyfilters.fm/podcast/episode-33/. Quoting from the transcript:

        This script, the one that Thomas Griffin wrote, is very similar to what we did in EDD and what I’m going to do in…. It uses … combination of PHP and JavaScript. Basically each batch is triggered by ajax and by JavaScript, so it fires off an ajax request. That triggers a batch.

        Then the batch returns … which triggers the ajax to say, “Okay, we’re done. Let’s do it again,” and it just keeps looping and looping and looping. You don’t have to worry about redirect timeouts because, if you get a lot of servers, if you try to redirect, say, five or ten times, it will say, “Oh, you’re going back and forth to the same page again, and I’m done. I quit.” […]

        That’s how Migrate DB Pro works, actually. It uses ajax and just does one request, gets some kind of value back to start the next batch to know where to leave off the next batch. We’ve tested that solution quite a bit, and it’s worked quite well, so I think it’s the way to go.

        The script mentioned by Thomas Griffin is this: https://thomasgriffin.io/introducing-tgm-batch-updates-plugin-wordpress/

        So to put this all together, here is how I would do it:

        Splitting the huge file into smaller parts

        Write the import routine that takes one smaller part increments the overall progress

        The javascript kicks off the import routine at current progress. If it’s finished, it increments the overall progress and reports the step completed via AJAX.

        On the client side the browser reloads the page if the current step has been completed. The reload then triggers another import routine and continues until all files have been processed.

      • Khan

        I have the same question as its look like we are doing the same thing as before but now we have number of small files instead of one large file.

        • Tom

          The process should work the same. Rather than splitting one large file into a bunch of small files, try this:

          • Load all of the files located in the directory
          • Iterate through files
          • Perform the tasks as mentioned in the article

          From there, you should be good to go.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

© 2023 Tom McFarlin

Theme by Anders NorenUp ↑