Thread Subject:

Combine variable length vectors into one vector

Hi,

I need to print the contents of n files in a single column vector, with the contents of the second file directly below the contents of the first, the contents of the third file directly below the contents of the second,etc.etc. All n files contain 1 column of numbers (no text). The column sizes (i.e. the number of rows in each file) are not the same (this is where I am having real trouble). The n filenames are stored in a cell array N, such that:

N{1,1} = a.txt, where the contents of a are:
12
323
5
234
9

N{2,1} = b.txt, where the contents of b are:
712
22
2
88

N{3,1} = c.txt, where the contents of c are:
1
36
52
26
914
89
2
33

I have just shown 3 files here, but I don't know (ahead of time) how many files there are.
--------------------X------------------
Here's what I have done (I have tried fprintf, rather than vertcat):

for ikk = 1:n

    fid_sav1 = fopen( 'myfile3.txt','wt' );
    Main = load( N{ikk,1} );
    for pqr = 1: size( Main,1 )
        fprintf( fid_sav1,'%15.10f\n',Main( pqr,1 )');
    end

    fclose( fid_sav1 );

end
type('myfile3.txt');

I am getting a subscripted assignment mismatch at line 4:
Main=load(N{ikk,1}); %cell array N contains the n filenames
--------------------X------------------
Question:
Is there a way to fix this error so that fprintf can print out the contents as required?
Or, is there a way to vertically concatenate n vectors?

On 8/13/2012 6:18 PM, Stan wrote:
...

> I need to print the contents of n files in a single column vector, with
> the contents of the second file directly below the contents of the
> first, the contents of the third file directly below the contents of the
> second,etc.etc. All n files contain 1 column of numbers (no text)....

> I have just shown 3 files here, but I don't know (ahead of time) how
> many files there are.
...

> Is there a way to fix this error so that fprintf can print out the
> contents as required?

Yeah, but for this you don't need fprintf(), just pass a copy command to
the OS...

!copy '*.txt' 'alltext.txt'

or, if need more flexibility than simple wild card gives, use the list

--

^^^
I actually need to print it to a text file because I then need to manipulate that file.

Is there a way to vertically concatenate n vectors together, each of different lengths?

I would have actually thought that fprintf would be simpler???

On 8/13/2012 8:12 PM, Stan wrote:
> ^^^
> I actually need to print it to a text file because I then need to
> manipulate that file.
>
> Is there a way to vertically concatenate n vectors together, each of
> different lengths?

Sure, but

> I would have actually thought that fprintf would be simpler???

what's simpler than

copy *.txt allfiles.txt

and then read in allfiles.txt?

And, to do the copy reading the individual files doesn't need to
concatenate, either (and not knowing the lengths a priori, it'll be
faster if the sizes are significant to not concatenate because that has
overhead of reallocation as the result vector "grows" unless you
preallocate and test and so on).

If you insist on reading/writing then open the output file at the
beginning, read each in as a stream (binary) file and then out and then
close that file and reopen it as text and snarf it back up.

There's an example of reading a text (.m) file into a character array in
the fread() documentation.

But, to concatenate a vector is simple enough;

 >> v=rand(3,1); % default 1D column vectors...
 >> v=[v;randperm(5)'];
 >> v
v =
     0.9660
     0.1946
     0.1539
     1.0000
     5.0000
     4.0000
     1.0000
     2.0000
     3.0000
 >> c(1)={rand(3,1)}; % cell array of column vectors
 >> c(2)={randperm(5)'};
c =
     [3x1 double]
     [5x1 double]
 >> vertcat(c{:})
ans =
     0.1086
     0.1159
     0.3512
     4.0000
     1.0000
     5.0000
     3.0000
     2.0000
 >>

--

Ah, your vertcat command suggestion is just what I needed.

But, I have file names. I mean, I have 'a.txt', 'b.txt' and 'c.txt'.

How would I be able to load these and then vertcat() them?

On 8/13/2012 9:46 PM, Stan wrote:
> Ah, your vertcat command suggestion is just what I needed.
>
> But, I have file names. I mean, I have 'a.txt', 'b.txt' and 'c.txt'.
>
> How would I be able to load these and then vertcat() them?

Why would you want to? It's the hard way around...

copy *.txt allfile.txt

dat=textscan('allfiles.txt');

--

On 8/13/2012 11:26 PM, dpb wrote:
> On 8/13/2012 9:46 PM, Stan wrote:
>> Ah, your vertcat command suggestion is just what I needed.
>>
>> But, I have file names. I mean, I have 'a.txt', 'b.txt' and 'c.txt'.
>>
>> How would I be able to load these and then vertcat() them?
>
> Why would you want to? It's the hard way around...
>
> copy *.txt allfile.txt
>
> dat=textscan('allfiles.txt');

Of course, inside Matlab the copy needs a bang operator to submit to the OS

!copy *.txt allfile.txt

If the wild card really, really doesn't get the right files for some
reason, then you can use your list to make a command of the form

copy a.txt+b.txt+... allfile.txt

to append. Testing in my release Matlab builtin copyfile() isn't smart
enough to handle multiple inputs. :(

If there are so many files that the command line exceeds the length
allowed by the OS, then loop and start by copying the first into the
target then add the remainder one or several at a time building the
final result.

Doing the copies and then reading the result will be faster overall than
the reallocation on the fly as the size grows.

But, if your still just adamant about it,

d=dir('*.txt'); % a list of files
x=textread(d(1).name); % read first into array
for i=2:length(d) % and then the rest...
   x=[x;textread(d(i).name);]; % concatenate--this can get _s_l__o___w_!
end

NB -- I used textread because it returns a vector rather than a cell
which makes concatenation simpler into a long vector. Since TMW in
their infinite wisdom has turned textscan into return cell, you can go
to the trouble of opening the files explicitly and using fscanf()
instead or just accept the warning for the convenience or deal w/ the
cell...

I'll still say the better route if there are many files or the fewer
files are very long is to do the copy then read the result...

--

^^^^Ah, thanks.

You were right about the speed. So, what I ended up doing was going with your original suggestion (in the previous post). I just performed the concatenation when I defined the vectors themselves.i.e. at that step the vector was available, so I just used it rather than using filenames. This was much easier than working with cells containing the filenames. Your previous post worked exactly as expected. That was really all I needed. Thanks for this.

I've got another question. This is about loading filenames into a cell array. Specifically, it is about auto pre-allocating a cell.

BACKGROUND:
I've got a cell array that loads a certain set of files (based on the filenames) the first time through a loop. The second time, it loads a different set of files.

EXAMPLE of what is happening:
A file named "List.txt" is provided with the contents being:
a1bc
d1ef
g1hi

Then the loop does the following (loads files with one of these 3 strings in their filenames):
ex. i = 1 --> dataa1bc1.txt, dataa1bc2.txt, dataa1bc3.txt, dataa1bc4.txt
 i = 2 --> datad1ef1.txt, datad1ef2.txt, datad1ef3,.txt datad1ef4.txt
 i = 3 --> datag1hi1.txt, datag1hi2.txt, datad1ef3.txt, datad1ef4.txt

DESCRIPTION OF THE PROBLEM:
Now, look at line three (i = 3). The problem is that for the third iteration, there are only two files that should be loaded into the cell, not 4. So, the cell should shrink from 4 X 1 (for i = 1 and i = 2) to 2 X 1 for i = 3. Unfortunately, this isn't happening. So for i = 3, the cell holds 2 correct filenames (the first 2) and 2 incorrect filenames (the last 2).
------------------------------------------X------------------------------------------

Here is the actual code:

fid_1=fopen('List.txt');
nnn=textscan(fid_1,'%s');
fclose(fid_1);

for abcd=1:numel(nnn{1});
    filess = dir(['*' char(nnn{1}(abcd)) '*L*.txt'])
    spec_fc_row=1;
      for kk = 1:numel(filess);
         file=filess(kk).name;
         M{spec_fc_row,1}=filess(spec_fc_row).name;
         spec_fc_row=spec_fc_row+1;
      end
end
------------------------------------------X------------------------------------------

QUESTION:
As you can imagine, the next step involves loading the contents of the cell, one row at a time. Well this works well for i = 1 and i = 2 (assuming that numel(nnn{1}) = 3...though this number is not known ahead of time so pre-allocating using cell() is a problem), but for i = 3, 2 wrong filenames are being loaded as I mentioned above. Is there a way to auto shrink the cell array size? Or is there a more effective (alternative) way to load these files?

PS: In the code above, f the line starting with filess, n contains:
'a1bc'
'd1ef'
'g1hi'

On 8/16/2012 11:06 AM, Stan wrote:
> ^^^^Ah, thanks.
>
> You were right about the speed. ...

Toldja! <vbg> :)

> ... concatenation when I defined the vectors themselves.i.e. at that step
> the vector was available, so I just used it rather than using filenames.
...

If your algorithm is such that you know or compute the length of these
vectors, you can minimize the concatenation time growth by allocating
the longer vector and then populating it appropriately. Or, at least
minimize the number of reallocations by starting w/ some large length M
and don't grow the size until you've used that amount up and need more;
then grow it by another sizable chunk. The extra overhead in the
bookkeeping is worth the time if the process is repetitive.

...

> A file named "List.txt" is provided with the contents being:
> a1bc
> d1ef
> g1hi
>
> Then the loop does the following (loads files with one of these 3
> strings in their filenames):
> ex. i = 1 --> dataa1bc1.txt, dataa1bc2.txt, dataa1bc3.txt, dataa1bc4.txt
...
>
> Here is the actual code:
>
> fid_1=fopen('List.txt');
> nnn=textscan(fid_1,'%s');
> fclose(fid_1);
>
> for abcd=1:numel(nnn{1});
> filess = dir(['*' char(nnn{1}(abcd)) '*L*.txt'])
> spec_fc_row=1;
> for kk = 1:numel(filess);
> file=filess(kk).name;
> M{spec_fc_row,1}=filess(spec_fc_row).name;
> spec_fc_row=spec_fc_row+1;
> end
> end
>
> QUESTION:
> As you can imagine, the next step involves loading the contents of the
> cell, one row at a time. Well this works well for i = 1 and i = 2
> (assuming that numel(nnn{1}) = 3...though this number is not known ahead
> of time so pre-allocating using cell() is a problem), but for i = 3, 2
> wrong filenames are being loaded as I mentioned above. Is there a way to
> auto shrink the cell array size? Or is there a more effective
> (alternative) way to load these files?
>
> PS: In the code above, f the line starting with filess, n contains:
> 'a1bc'
> 'd1ef'
> 'g1hi'

Well, you kinda' lost me but let me try and see where I go wrong...

I start w/ a cell array c w/ your file partial names to match...

 >> c
c =
     'a1bc'
     'd1ef'
     'g1hi'
 >>

Since I don't have a bunch of files matching these, dir() here will be
null but I _can_ look at the argument to the dir() function --

 >> for i=1:numel(c{1}),disp(['*' char(c{1}(i)) '*L*.txt']),end
*a*L*.txt
*1*L*.txt
*b*L*.txt
*c*L*.txt
 >>

That doesn't look at all like your earlier description of wanting

dir('dataa1bc*.txt')

etc., for the strings in the file...also note there are four elements
there not just the three strings. I think here's where most of your
problems arise.

To do what you described above you don't want the loop over numel(c{1})
w/ the curly brackets on the first element of the cell array--the
curlies dereference the _content_ of the cell which is a 4-character
character array. You want to loop instead on the number of cell strings
in c.

Try this instead--
 >> for i=1:length(c),disp(['*' char(c(i)) '*L*.txt']),end
*a1bc*L*.txt
*d1ef*L*.txt
*g1hi*L*.txt
 >>

In your code as the argument to dir() this will result in returning the
files you wish in the proper number.

Now let me see if I can decipher intent of inner loop...that confused me
on first reading--

d = dir(['*' char(c(i)) '*L*.txt']);
row=1;
for k = 1:length(d);
   file=d(k).name;
   M{row,1}=d(row).name;
   row=row+1;
end


OK, in the end you've got a cell array M that contains the names of the
directory structure. Why? Why not just use the name itself to do
whatever you want with opening the file(s)?

I don't know where you're headed so I don't really see the next step
clearly, sorry.

If you really do just want a cell array of the names for some reason,
you can accomplish what the above loop does simply by (I demonstrate
here on a local directory result)

 >> d=dir('*.dat')
d =
16x1 struct array with fields:
     name
     date
     bytes
     isdir
 >> M=cellstr({d.name}')
M =
     '30test.dat'
     'AIRPORT.dat'
     'alldat.dat'
...[elided for brevity]...
     'TESTIT.DAT'
     'VEHIC.DAT'
     'WORK.DAT'
 >>

See if that helps and if still can't get where going, try again.
Probably ought to start a new thread for that question; this is pretty
far afield from where this one started...

--

On 8/16/2012 1:43 PM, dpb wrote:
,,,

> Try this instead--
> >> for i=1:length(c),disp(['*' char(c(i)) '*L*.txt']),end
> *a1bc*L*.txt
> *d1ef*L*.txt
> *g1hi*L*.txt
> >>
>
> In your code as the argument to dir() this will result in returning the
> files you wish in the proper number.
...

I intended to add a stylistic alternative to the above concatenation of
strings in the []...use sprintf() to build the string instead...

 > for i=1:length(c),f=sprintf('*%s*L*.txt', char(c(i))),end
f =
*a1bc*L*.txt
f =
*d1ef*L*.txt
f =
*g1hi*L*.txt
 >>

And, of course, you can build more complicated format strings as
variables which can often simplify what otherwise appears to be a
mishmash of special characters '*%\n, etc., etc., etc., ...

This one is pretty easy but the idea is still worth pointing out...

fmt='*%s*L*.txt';
f=sprintf(fmt, char(c(i)));

--

With that code, you would still have the same problem that I had. Suppose you had 2 dir() commands - d = () and c = (). The first dir() returned 4 files and the second dir() returned 2 files. Well, if you used the same cell array, M, to store the file names, then M would first contain 4 file names, as it should. But if you then try to overwrite the contents of M will the file names from the second dir() command, only the first two file names will be overwritten:

After first dir():
M =
> '30test.dat'
> 'AIRPORT.dat'
> 'alldat.dat'
> ...[elided for brevity]...
> 'TESTIT.DAT'
> 'VEHIC.DAT'
> 'WORK.DAT'

After second dir(), suppose you wanted 2 files from this 2nd dir() - abcdefghi.dat and jklmnopqr.dat, you would get:
M =
> 'abcdefghi.dat'
> 'jklmnopqr.dat'
> 'alldat.dat'
> ...[elided for brevity]...
> 'TESTIT.DAT'
> 'VEHIC.DAT'
> 'WORK.DAT'

All the remaining row entries, in from row 3 down, are wrong.

"Stan" wrote in message <k0lq7e$3do$1@newscl01ah.mathworks.com>...

>
> All the remaining row entries, in from row 3 down, are wrong.

I'm not sure about I fully get what you want to achieve (too much to read through this long developing thread which seems only you and dpb are still keeping track), but what wrong with deleting the tail of the cell:

c(3:end) = [];

% or fill it with empty array.

c(3:end) = {''};

What is the final purpose already?

Bruno

On 8/17/2012 11:07 AM, Stan wrote:
> With that code, you would still have the same problem that I had.
> Suppose you had 2 dir() commands - d = () and c = (). The first dir()
> returned 4 files and the second dir() returned 2 files. Well, if you
> used the same cell array, M, to store the file names, then M would first
> contain 4 file names, as it should. But if you then try to overwrite the
> contents of M will the file names from the second dir() command, only
> the first two file names will be overwritten:
...

That's just not correct. Back to my working directory for example--

 >> d=dir('*.dat')
d =
16x1 struct array with fields:
     name
     date
     bytes
     isdir
 >> c=dir('*.m')
c =
20x1 struct array with fields:
     name
     date
     bytes
     isdir
 >> M=cellstr({c.name}') % the longer of the two...
M =
     'all.m'
     'blokdiag.m'
     'cellstrmatch.m'
     'CHIP.M'
     'CSSM.M'
     'DiasRead.m'
     'mycombs.m'
     'myfind.m'
     'STEAMUS.M'
     'subtestfunction.m'
     'test.m'
     'testit.m'
     'TOKENS.M'
     'TRUE.M'
     'try.it.m'
     'TRYIT.M'
     'varfreq.m'
     'WCHILL.M'
     'winddata.m'
     'XSTEAMUS.M'
 >> M=cellstr({d.name}') % now the shorter...
M =
     '30test.dat'
     'AIRPORT.dat'
     'alldat.dat'
     'cplex.dat'
     'DATA.DAT'
     'DATE.DAT'
     'FARM.dat'
     'FMT.DAT'
     'junk.dat'
     'MON.DAT'
     'output.dat'
     'PORT.DAT'
     'save.dat'
     'TESTIT.DAT'
     'VEHIC.DAT'
     'WORK.DAT'
 >>

M is completely reassigned automagically...there's nothing left over.
The reason you had stuff left over was that you went through a loop and
if the second pass thru the loop is shorter than the first then yes,
Virginia, the presents from last Christmas are still under the tree...

But _WHY_ are you doing this, anyway???? The filenames are in the data
structure returned from dir(); why aren't you using that? What's the
point of M at all? As Bruno notes we don't know where you're headed in
the end so it's not possible to know what the best direction is, but I
think you're heading N when want to go E or W...

--

What I was doing:
Use a for loop to store results of dir() in a cell array, N. Then, I would use a second for for loop to access the files in the cell array N.

I've decided to just use the files directly from the results of dir() as you both said. It is much easier and I don't have to build up a cell array.

"Stan" wrote in message <k0m9qa$53f$1@newscl01ah.mathworks.com>...
> What I was doing:
> Use a for loop to store results of dir() in a cell array, N. Then, I would use a second for for loop to access the files in the cell array N.

Suppose you want to do that, it is just about using few simple commands:

>> a=dir('c*.m') % first dir

a =

3x1 struct array with fields:
    name
    date
    bytes
    isdir
    datenum

>> b=dir('f*.m') % second dir

b =

2x1 struct array with fields:
    name
    date
    bytes
    isdir
    datenum

>> c = {a.name b.name} % now you have your cell

c =

    [1x25 char] [1x21 char] 'comparearrays.m' 'findminloc.m' 'finish.m'

>> c{:} % check

ans =

ComputeStatisticsOnData.m


ans =

circumscribedcircle.m


ans =

comparearrays.m


ans =

findminloc.m


ans =

finish.m

% You can even loop many times dir(), like this
c = cell(1,26);
for k=1:26
    alphabet='a'+(k-1);
    s = dir([alphabet '*.m']);
    c{k} = {s.name};
end
c = flattencell(c) % use the bellow mfile

% Save this in an mfile
function c=flattencell(c)
% c=flattencell(c)
% Set the hierarchy cell flat
if iscell(c)
    c = cellfun(@flattencell, c, 'UniformOutput', 0);
    c = cat(2,c{:});
else
    c = {c};
end
end % flattencell

% Bruno

On 8/17/2012 3:33 PM, Stan wrote:
> What I was doing:
> Use a for loop to store results of dir() in a cell array, N. Then, I
> would use a second for for loop to access the files in the cell array N.

Two bad ideas... :)

First, as my response before shows, you don't need the first loop at all
to create the cell array and the direct assignment doesn't suffer from
the problem of not defining all the previous elements in a subsequent
shorter-than-the-longest previous pass. (The solution there, btw, if
there were a real reason otherwise for a loop would be to simply 'clear
M;' or 'M=cell(length(d));' before starting the loop.)

> I've decided to just use the files directly from the results of dir() as
> you both said. It is much easier and I don't have to build up a cell array.

That is indeed, a far better direction...

You might try

d=dir('yourmask*.dat');
for f={d.name}, disp(f),end

just for fun... :)

doc for % for the mystery unraveled (assuming it is one, anyway...)

--

On 8/17/2012 4:39 PM, dpb wrote:
...

>> I've decided to just use the files directly from the results of dir() as
>> you both said. It is much easier and I don't have to build up a cell
>> array.
>
> That is indeed, a far better direction...
>
> You might try
>
> d=dir('yourmask*.dat');
> for f={d.name}, disp(f),end
>
> just for fun... :)
>
> doc for % for the mystery unraveled (assuming it is one, anyway...)

And, another way to do the loop thru the files looking at your list in
what appears to be a single pass...

Assuming c is the cell of file patterns to match of the earlier posting..

for i=1:length(c)
   d=dir(sprintf('*s*L*.txt', char(c(i));
   for fn={d.name}
     x=textread(fn);
     % process the file data here...
   end
end

This does use two loops but nests them. In the end it processes the
files in the same order as the two separate steps of getting all the
files into another list and the processing the list.

What is "most bestest" for your needs depends on what you've not told
us--what you're actually doing w/ the file data.

--

dpb <none@non.net> wrote in message <k0mneb$gtg$1@speranza.aioe.org>...

>
> for i=1:length(c)
> d=dir(sprintf('*s*L*.txt', char(c(i));
> for fn={d.name}
> x=textread(fn);
> % process the file data here...
> end
> end
>

% Alternatively, one can also loop on the struct array
for i=1:length(c)
   d=dir(yourfolder);
   for f = d' % transpose to row
     x=textread(f.name);
     % process the file data here...
   end
end

% Bruno

"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message
> % You can even loop many times dir(), like this
> c = cell(1,26);
> for k=1:26
> alphabet='a'+(k-1);
> s = dir([alphabet '*.m']);
> c{k} = {s.name};
> end
> c = flattencell(c) % use the bellow mfile
>

or growing the cell

filenames = {};
for k=1:26
    alphabet='a'+(k-1);
    s = dir([alphabet '*.m']);
    filenames = [filenames {s.name}];
end
filenames

% Bruno

To complement my code in post #15, there is no need to call for specific function,
> c = flattencell(c)

The above can be replaced more friendly by a stock function

> c = cat(2,c{:})

% Bruno

On 8/18/2012 1:26 AM, Bruno Luong wrote:
...

> % Alternatively, one can also loop on the struct array
> for i=1:length(c)
> d=dir(yourfolder);
> for f = d' % transpose to row
> x=textread(f.name);
...

Many ways and cats, indeed, Bruno... :)

BTW, as a nit the ' isn't needed here on the iterator. It's little more
concise than the {} in the iterator; not sure why I go in the habit of
the other way 'round...maybe I'll try to break an old habit! :)

Slow morning at cs-sm, isn't it??? <vbg>

--