Thread Subject:

Import Several Hundred .txt files

Hello,

I am currently trying to load 726 .txt files into Matlab, ultimately to vertically concatenate them into one array.

The files are named:

DSpec1203190348
DSpec1203190548
DSpec1203190748
etc etc

So that there is effectively a step of 200 in the file name between each file. I should also mention that the first 3 rows of each file is text- this is written by the program they were created in (Wavesmon) and cannot be changed unfortunately, although I think textscan would be useful here. Beyond this, the file is 65x1.

Any help would be massively appreciated, as this has got me well and truly stumped.

Many thanks,

Robbie

On Wednesday, June 13, 2012 6:51:07 AM UTC+12, Robbie Brady wrote:
> Hello,
>
> I am currently trying to load 726 .txt files into Matlab, ultimately to vertically concatenate them into one array.
>
> The files are named:
>
> DSpec1203190348
> DSpec1203190548
> DSpec1203190748
> etc etc
>
> So that there is effectively a step of 200 in the file name between each file. I should also mention that the first 3 rows of each file is text- this is written by the program they were created in (Wavesmon) and cannot be changed unfortunately, although I think textscan would be useful here. Beyond this, the file is 65x1.
>
> Any help would be massively appreciated, as this has got me well and truly stumped.
>
> Many thanks,
>
> Robbie

This is not quite correct, is it?
Isn't the format of the name DSpecyymmddHHMM?
And the samples are 2 hourly wave spectra, so the filename is not going to change in base 10, but base 24 for days and 31 or 30 for months. This going to get very messy................

A better way will be to read in the filenames like this:
d=dir([path_to_files 'DSpec*']);
then loop through the files reading one at a time and parsing the filename using datenum like this:
t(id)=datenum(d(id).name,'yymmddHHMM');
To read the files, do this:
fid=fopen([path_to_files d(id).name],'rt');
c=datenum(fid,'%f','headerlines',3);
fclose(fid);
a(:,id)=c{1};
where a is the amplitude spectrum mm/sqrt(Hz) - you need to square this to get PSD.

Once you've read all the spectra in, you may need to sort the columns of a to get them in increasing time. You can do this by sorting t:
[tsort,indx]=sort(t);
a=a(:,indx);
 

You're right about the format of the filename- this data has kind of been thrown at me with little explanation.

Thanks very much for your help, I will have a go now and see how I get on.

Many thanks,

Robbie

> This is not quite correct, is it?
> Isn't the format of the name DSpecyymmddHHMM?
> And the samples are 2 hourly wave spectra, so the filename is not going to change in base 10, but base 24 for days and 31 or 30 for months. This going to get very messy................
>
> A better way will be to read in the filenames like this:
> d=dir([path_to_files 'DSpec*']);
> then loop through the files reading one at a time and parsing the filename using datenum like this:
> t(id)=datenum(d(id).name,'yymmddHHMM');
> To read the files, do this:
> fid=fopen([path_to_files d(id).name],'rt');
> c=datenum(fid,'%f','headerlines',3);
> fclose(fid);
> a(:,id)=c{1};
> where a is the amplitude spectrum mm/sqrt(Hz) - you need to square this to get PSD.
>
> Once you've read all the spectra in, you may need to sort the columns of a to get them in increasing time. You can do this by sorting t:
> [tsort,indx]=sort(t);
> a=a(:,indx);
>

Hmmmm... When running the script I get the following error:

??? Undefined function or variable 'id'.

Error in ==> velocity_spectrum_script at 4
t(id)=datenum(d(id).name,'yymmddHHMM');

I have tried a few things but nothing works, as you may have guessed I am somewhat of a Matlab novice.

Many thanks,

Robbie

"Robbie Brady" <robbie.brady@postgrad.plymouth.ac.uk> wrote in message
news:jr8agv$i9n$1@newscl01ah.mathworks.com...
>> This is not quite correct, is it?
>> Isn't the format of the name DSpecyymmddHHMM?
>> And the samples are 2 hourly wave spectra, so the filename is not going
>> to change in base 10, but base 24 for days and 31 or 30 for months. This
>> going to get very messy................
>>
>> A better way will be to read in the filenames like this:
>> d=dir([path_to_files 'DSpec*']);
>> then loop through the files reading one at a time and parsing the
>> filename using datenum like this:
>> t(id)=datenum(d(id).name,'yymmddHHMM');
>> To read the files, do this:
>> fid=fopen([path_to_files d(id).name],'rt');
>> c=datenum(fid,'%f','headerlines',3);
>> fclose(fid);
>> a(:,id)=c{1};
>> where a is the amplitude spectrum mm/sqrt(Hz) - you need to square this
>> to get PSD.
>>
>> Once you've read all the spectra in, you may need to sort the columns of
>> a to get them in increasing time. You can do this by sorting t:
>> [tsort,indx]=sort(t);
>> a=a(:,indx);
>>
>
> Hmmmm... When running the script I get the following error:
>
> ??? Undefined function or variable 'id'.
>
> Error in ==> velocity_spectrum_script at 4
> t(id)=datenum(d(id).name,'yymmddHHMM');
>
> I have tried a few things but nothing works, as you may have guessed I am
> somewhat of a Matlab novice.

When TideMan said:

>> then loop through the files reading one at a time and parsing the
>> filename using datenum like this:

the loop variable he used was id, and that choice carried through the rest
of his example.

for id = 1:numel(d)
    % use TideMan's code
end

Note that you'll also need to define the variable path_to_files; the name
suggests what information should be stored in that variable.

--
Steve Lord
slord@mathworks.com
To contact Technical Support use the Contact Us link on
http://www.mathworks.com

> When TideMan said:
>
> >> then loop through the files reading one at a time and parsing the
> >> filename using datenum like this:
>
> the loop variable he used was id, and that choice carried through the rest
> of his example.
>
> for id = 1:numel(d)
> % use TideMan's code
> end
>
> Note that you'll also need to define the variable path_to_files; the name
> suggests what information should be stored in that variable.
>
> --
> Steve Lord
> slord@mathworks.com
> To contact Technical Support use the Contact Us link on
> http://www.mathworks.com

Thanks for your help.
I now have a script that looks like this:

%%%% Read in filenames %%%%
d=dir(['Users/robbrady/Documents/ADCP/adcp_data/may_2012/DCWaveHubWest/Velocity_Spectrum_Files' 'VSpec*']);

%%%%loop files and parse filename%%%
for id= 1:numel(d)
    
t(id)=datenum(d(id).name,'yymmddHHMM')

end
%%%%%Read files in %%%%%%

fid=fopen(['Users/robbrady/Documents/ADCP/adcp_data/may_2012/DCWaveHubWest/Velocity_Spectrum_Files' d(id).name],'rt');
c=datenum(fid,'%f','headerlines',3);
fclose(fid);
a(:,id)=c{1};

But receive the following error:
"Error in ==> velocity_spectrum_script at 12
c=datenum(fid,'%f','headerlines',3);

Caused by:
    Error using ==> datenum at 167
    Incorrect number of arguments "

I can't seem to find much info on this error online. If anyone has any ideas where I'm going wrong I would be eternally grateful.

On Wednesday, June 13, 2012 6:51:07 AM UTC+12, Robbie Brady wrote:
> Hello,
>
> I am currently trying to load 726 .txt files into Matlab, ultimately to vertically concatenate them into one array.
>
> The files are named:
>
> DSpec1203190348
> DSpec1203190548
> DSpec1203190748
> etc etc
>
> So that there is effectively a step of 200 in the file name between each file. I should also mention that the first 3 rows of each file is text- this is written by the program they were created in (Wavesmon) and cannot be changed unfortunately, although I think textscan would be useful here. Beyond this, the file is 65x1.
>
> Any help would be massively appreciated, as this has got me well and truly stumped.
>
> Many thanks,
>
> Robbie

Ooops, my 2nd datenum should have been textscan:
c=textscan(fid,'%f','headerlines',3);

TideMan <mulgor@gmail.com> wrote in message <7478fe55-0687-41d8-8414-8184024c28cb@googlegroups.com>...
> On Wednesday, June 13, 2012 6:51:07 AM UTC+12, Robbie Brady wrote:
> > Hello,
> >
> > I am currently trying to load 726 .txt files into Matlab, ultimately to vertically concatenate them into one array.
> >
> > The files are named:
> >
> > DSpec1203190348
> > DSpec1203190548
> > DSpec1203190748
> > etc etc
> >
> > So that there is effectively a step of 200 in the file name between each file. I should also mention that the first 3 rows of each file is text- this is written by the program they were created in (Wavesmon) and cannot be changed unfortunately, although I think textscan would be useful here. Beyond this, the file is 65x1.
> >
> > Any help would be massively appreciated, as this has got me well and truly stumped.
> >
> > Many thanks,
> >
> > Robbie
>
> Ooops, my 2nd datenum should have been textscan:
> c=textscan(fid,'%f','headerlines',3);

I am now getting the error:
??? Error using ==> textscan
Invalid file identifier. Use fopen to generate a valid file identifier.

Error in ==> velocity_spectrum_script at 13
c=textscan(fid,'%f','headerlines',3);

And the fid returned is -1, telling me that fopen has failed. I don't understand why though as I can't see anything wrong?

On Wednesday, June 13, 2012 6:51:07 AM UTC+12, Robbie Brady wrote:
> Hello,
>
> I am currently trying to load 726 .txt files into Matlab, ultimately to vertically concatenate them into one array.
>
> The files are named:
>
> DSpec1203190348
> DSpec1203190548
> DSpec1203190748
> etc etc
>
> So that there is effectively a step of 200 in the file name between each file. I should also mention that the first 3 rows of each file is text- this is written by the program they were created in (Wavesmon) and cannot be changed unfortunately, although I think textscan would be useful here. Beyond this, the file is 65x1.
>
> Any help would be massively appreciated, as this has got me well and truly stumped.
>
> Many thanks,
>
> Robbie

An fid of -1 means it cannot open the file.
So, the filename in fopen is wrong.
Print it out and see what it is.
You've probably omitted a / or the extension or something.
You should learn to do this simple debugging yourself.

On 6/13/2012 4:28 PM, Robbie Brady wrote:

> I am now getting the error:
> ??? Error using ==> textscan
> Invalid file identifier. Use fopen to generate a valid file identifier.
>
> And the fid returned is -1, telling me that fopen has failed.
>I don't understand why though as I can't see anything wrong?

Use

" [FID, MESSAGE] = fopen(FILENAME,...) returns a system dependent error
     message if the open is not successful."

always check for fid=-1, and if so, print the MESSAGE. This
tells you why Matlab could not open the file.

No need for guessing. There are 101 reasons why a file
can't be opened successfully.

--Nasser

TideMan <mulgor@gmail.com> wrote in message <9b52e1cc-9002-4d05-9967-185e1963c3f9@googlegroups.com>...
> On Wednesday, June 13, 2012 6:51:07 AM UTC+12, Robbie Brady wrote:
> > Hello,
> >
> > I am currently trying to load 726 .txt files into Matlab, ultimately to vertically concatenate them into one array.
> >
> > The files are named:
> >
> > DSpec1203190348
> > DSpec1203190548
> > DSpec1203190748
> > etc etc
> >
> > So that there is effectively a step of 200 in the file name between each file. I should also mention that the first 3 rows of each file is text- this is written by the program they were created in (Wavesmon) and cannot be changed unfortunately, although I think textscan would be useful here. Beyond this, the file is 65x1.
> >
> > Any help would be massively appreciated, as this has got me well and truly stumped.
> >
> > Many thanks,
> >
> > Robbie
>
> An fid of -1 means it cannot open the file.
> So, the filename in fopen is wrong.
> Print it out and see what it is.
> You've probably omitted a / or the extension or something.
> You should learn to do this simple debugging yourself.

You're right, I should. Code is working perfectly now, thanks very much for your help guys!

Many thanks,

Robbie

Hi Robbie,

here might be an easy solution to do it (i am not sure about your file format):

for i= 1:726
    num = 1203190348 + (i-1)*200;
    eval(['result(:,i) = dlmread(''DSpec',num,'''']); % assume 65x1, just read it in as a column with 65 values
end


-KJ

On Tuesday, June 12, 2012 2:51:07 PM UTC-4, Robbie Brady wrote:
> Hello,
>
> I am currently trying to load 726 .txt files into Matlab, ultimately to vertically concatenate them into one array.
>
> The files are named:
>
> DSpec1203190348
> DSpec1203190548
> DSpec1203190748
> etc etc
>
> So that there is effectively a step of 200 in the file name between each file. I should also mention that the first 3 rows of each file is text- this is written by the program they were created in (Wavesmon) and cannot be changed unfortunately, although I think textscan would be useful here. Beyond this, the file is 65x1.
>
> Any help would be massively appreciated, as this has got me well and truly stumped.
>
> Many thanks,
>
> Robbie