Thread Subject:

Loading multiple files into a dataset

Hi,

I have a series of .txt files which contain data I would like to load into MATLAB and combine to form a dataset. Each file contains 2 columns: time and velocity.

Ideally I would like to have a dataset which has one column for time then a series of columns for the velocities of the various texts files e.g. attempt one velocity, attempt 2 velocity, attempt 3 velocity etc.

I have spent some time Googling and seen a few examples of loading multiple files but not into a dataset.

Please can anyone offer help?
Dan

> I have a series of .txt files which contain data I would like to load into
> MATLAB and combine to form a dataset. Each file contains 2 columns: time
> and velocity.

Dan, take a look at "help dataset/join". You might want something like

   join(d1,d2,'key','time')

-- Tom

Hi Tom,

The join function looks very good, I think that will be great for bringing the data together.

What would your recommend for loading multiple text files in to the workspace?

I would like to write a script that could complete the whole process once I have saved the .txt files into a directory of their own.

Thanks for your help
Dan

"Daniel Robbins" <d.w.e.robbins@gmail.com> wrote in message
news:jr7lfr$au4$1@newscl01ah.mathworks.com...
> Hi Tom,
>
> The join function looks very good, I think that will be great for bringing
> the data together.
> What would your recommend for loading multiple text files in to the
> workspace?

Question 12 in the Programming section of the FAQ describes several
techniques with which you can process a list of files.

http://matlab.wikia.com/wiki/FAQ

--
Steve Lord
slord@mathworks.com
To contact Technical Support use the Contact Us link on
http://www.mathworks.com

Hi Steve,

I have tried both the methods on the link you provided, but I find that all of the data is not stored, only the first number in the text file is stored. If I use the importdata function then I can load the entire text file e.g.

textfiles=dir('*.txt');

for k = 1: length(textfiles)
            textfilename = [num2str (k), '.txt];
            a = importdata (textfilename);
end

However this over writes the data with each loop. I have googled around to look if there is anyway to create sequential variables e.g. a1,a2,a3...etc. However there are lots of pages indicating this is not the way to go.

The ultimate goal is to get all the text files loaded and combined in to one dataset, however I would like to be able to get MATLAB to read direct from the text files rather then load them each into excel organise them and then load using the dataset function. Obviously this would be very time consuming...

Any help will be much appreciated.
Thanks
Dan

Daniel, I assume you're talking about the dataset array class in the
Statistics Toolbox. If you're looking for one line of code to do what
you ask, I think you're out of luck. But here's a suggestion.

If you read each file into a dataset, and they all have the same time
vector (named the same, same exact values), you can horzcat them all
together and get a single array with a single time variable:

 >> d1 = dataset((1:5)',rand(5,1),'VarNames',{'time' 'velocity1'});
 >> d2 = dataset((1:5)',rand(5,1),'VarNames',{'time' 'velocity2'});
 >> d3 = dataset((1:5)',rand(5,1),'VarNames',{'time' 'velocity3'});

 >> [d1 d2 d3]
ans =
     time velocity1 velocity2 velocity3
     1 0.81472 0.09754 0.15761
     2 0.90579 0.2785 0.97059
     3 0.12699 0.54688 0.95717
     4 0.91338 0.95751 0.48538
     5 0.63236 0.96489 0.80028

You'll have to give each velocity variable a unique name, but if you're
reading files in a loop, that's a one-liner:

 >> d.Properties.VarNames{2} = num2str(i,'velocity%d')

Of course, if these are really long files, you'll have spent time
checking to make sure that all the time variables are the same thing,
and you already know that. It may be simpler to just create one dataset
from the first file, then read into a temp dataset,and do something like
this:

 >> d.(num2str(i,'velocity%d')) = dTmp.velocity;

That ".()" is the same "dynamic dot indexing" that you may have seen
with struture arrays. Hope this helps.

On 6/12/2012 9:08 AM, Daniel Robbins wrote:
> Hi,
>
> I have a series of .txt files which contain data I would like to load
> into MATLAB and combine to form a dataset. Each file contains 2 columns:
> time and velocity.
>
> Ideally I would like to have a dataset which has one column for time
> then a series of columns for the velocities of the various texts files
> e.g. attempt one velocity, attempt 2 velocity, attempt 3 velocity etc.
>
> I have spent some time Googling and seen a few examples of loading
> multiple files but not into a dataset.
>
> Please can anyone offer help?
> Dan

Hi Peter,

Thanks for your help, I think what your suggesting is the answer to the second part of of my problem - which is great btw, thank you.

The main issue I have at the moment is actually loading the data into MATLAB from various textfiles without them over writing in the 'for' loop. I have managed to acheive this with the following code:

textfiles=dir('*.txt');

for k = 1:length(textfiles);
    textfilename = [num2str(k),'.txt'];
    a= importdata(textfilename);
    eval(['a', int2str(k) '= importdata(textfilename)']);
end

However, I have seen many warnings for this use of the eval function so I am assume its not a good way to go. Once I am confident that I have a good way of getting the data into the workspace I will follow your advice, perhaps combined with the earlier advice of Tom Lane.

I have saved some .txt files with no headers and two columns of data with ';' deliminators and I am currently working with these so that I can run quick tests. So far the above code is accurate - but the various warnings around make me nervous of this code.

I dont mind if the code gets long and is not the fastest (provided it doesn't take minutes to process). I just want to be sure that the data is accurate.

Any and all advice is greatly appreciated.
Many thanks
Dan

"Daniel Robbins" <d.w.e.robbins@gmail.com> wrote in message
news:jraij6$eal$1@newscl01ah.mathworks.com...
> Hi Peter,
>
> Thanks for your help, I think what your suggesting is the answer to the
> second part of of my problem - which is great btw, thank you.
>
> The main issue I have at the moment is actually loading the data into
> MATLAB from various textfiles without them over writing in the 'for' loop.
> I have managed to acheive this with the following code:
>
> textfiles=dir('*.txt');
>
> for k = 1:length(textfiles);
> textfilename = [num2str(k),'.txt'];
> a= importdata(textfilename);
> eval(['a', int2str(k) '= importdata(textfilename)']);
> end
>
> However, I have seen many warnings for this use of the eval function so I
> am assume its not a good way to go. Once I am confident that I have a good
> way of getting the data into the workspace I will follow your advice,
> perhaps combined with the earlier advice of Tom Lane.

Use a cell array (or if all your data is the same size, a 3-dimensional
array) as described in question 6 in the Programming section of the
newsgroup FAQ.

http://matlab.wikia.com/wiki/FAQ

Or read in the data file to a temporary variable, append it to the dataset
in question, then overwrite that temporary variable at the next iteration.
This technique is used in question 12 in the Programming section of the FAQ.

--
Steve Lord
slord@mathworks.com
To contact Technical Support use the Contact Us link on
http://www.mathworks.com

Hi Steven,

Thanks for the advice. I have actually looked at these sections before. I tried writing the code in as advised iin section 6 (see below)

textfiles=dir('*.txt');
a= zeros (1,10);

for i = 1:length(textfiles);
    textfilename = [num2str(i),'.txt'];
    a{i}= importdata(textfilename);
end

however I get the following error message:

??? Cell contents assignment to a non-cell array object.

Error in ==> urlgh at 8
    a{i}= importdata(textfilename);

I have also looked at section 12, however while the code runs it does not store all the data from the text files, only the final interger stored in the text file.

To date the only code I have managed to get to run and work is the one with the eval function.

Any ideas?

Thanks
Dan

"Daniel Robbins" <d.w.e.robbins@gmail.com> wrote in message
news:jrak9j$lnn$1@newscl01ah.mathworks.com...
> Hi Steven,
>
> Thanks for the advice. I have actually looked at these sections before. I
> tried writing the code in as advised iin section 6 (see below)
>
> textfiles=dir('*.txt');
> a= zeros (1,10);

Preallocate a as a cell array, not a numeric array.

a = cell(1, 10); % or even better
a = cell(1, length(textfiles));

> for i = 1:length(textfiles);
> textfilename = [num2str(i),'.txt'];
> a{i}= importdata(textfilename);
> end
>
> however I get the following error message:
>
> ??? Cell contents assignment to a non-cell array object.

You can't assign into a numeric array using curly braces; those are only for
cell arrays and objects that define what curly brace indexing means for
them.

--
Steve Lord
slord@mathworks.com
To contact Technical Support use the Contact Us link on
http://www.mathworks.com

Ok,

I changed the brackets as suggested:

textfiles=dir('*.txt');
a= zeros (1,10);
%for i = 1:10
% a(i)=i*2;
%end
for i = 1:length(textfiles);
    textfilename = [num2str(i),'.txt'];
    a(i)= importdata(textfilename);
end

I now get the follwing error message:

??? In an assignment A(I) = B, the number of elements in B and
 I must be the same.

Error in ==> urlgh at 10
    a(i)= importdata(textfilename);

I also noted that in section 12 of the FAQ page there is a suggestion to save within the loop but no example code explaining how to do this. For example if I were to save each section to the dataset as previously suggested then how would I tell matlab how name each section e.g. velocity1, velocity2, velocity3...etc

Thanks again!

Apologies Steven,

I missed the code in the middle of your last message (think I have been staring at the screen for too long...)

I have now managed to get the following code to run:

textfiles=dir('*.txt');

a = cell(1, length(textfiles));

for i = 1:length(textfiles);
    textfilename = [num2str(i),'.txt'];
    a{i}= importdata(textfilename);
end

However, while calling a provides the following display:

a =
    [6x2 double] [6x2 double] [6x2 double] [6x2 double] [6x2 double] [6x2 double]

Which presumably is the data from the text files, I have no idea how to access the data from each file or how to get them into a dataset.

Please can you help me with this last issue?

Many thanks for all your help so far!!
Dan

On 6/13/2012 3:12 PM, Daniel Robbins wrote:
....

> a{i}= importdata(textfilename);
> end
>
> However, while calling a provides the following display:
>
> a = [6x2 double] [6x2 double] [6x2 double] [6x2 double] [6x2 double]
> [6x2 double]
>
> Which presumably is the data from the text files, I have no idea how to
> access the data from each file or how to get them into a dataset.

> Please can you help me with this last issue?

Each subset...

x=a{1}; % use the {} to dereference a cell array

alldat=cat(1,a{:}); % put them all in a 2D array

Read the Getting Started section on cell arrays and the section on cells
in the data types section to get a better grasp on using cell arrays.
In short, the cell is essentially a pointer to the data within it;
referring to the cell w/ the () gets that cell but to get to the
contents you must use the {}.

--

Thanks dpb, appreciate the help!