2020-08-16 / Blog 6

This week's blog post will be a rather short one on importing and merging multiple data files. We have an API at NGIF, where we can access all the data in real time. However, when you download historic data, you might end up with multiple *.csv files. The code I will present here will save you from writing for loops.

Let's assume that you have a folder containing weekly data files of rainfall. Firstly, we create a vector with the location of the folder we have the data files. Then, the first function we will use is list.files(), and we will generate a vector with the names of the files in the folder.

weeklydata <- "D:/PLEXUS/AllData/Rainfall/"

dataname <- list.files(weeklydata)

We have 68 data files in the this folder, and now we will import them into R with the following function:

alldata <- lapply(1:68,function(x){

read.csv(paste0(weeklydata,dataname[x]),stringsAsFactors=F,header=T)})

This generates a large list named alldata with 68 elements in it. Now, what we need is the do.call function to merge these individual files. It is a simple but a very efficient function.

merged <- do.call(rbind,alldata)

The combination of list.files(), lapply() and do.call() is just magic. You do not need to write a for loop, import data files one by one, assign each of them a name, and then merge them. You can simply use these couple of lines of code and have your merged data frame imported into R.

keywords: data analysis; R; merge; do.call