Thread Subject:

load part of variable in mat file without variable name

Hello,

I am trying to load part of a variable stored in a mat file, but will not always know the name of the variable (This is the only variable in the file, and is an array ). Consider a file array1.mat, in which there is a variable called "varName" that is a big matrix. I know you can use the following code

array1_name = sprintf('array1')
matObj = matfile(array1_name);
part = matObj.varName(1:2,1:2);

but this requires knowing and hard coding in "varName" (which I will not always know). I cannot load the whole variable into memory as they are too large. I have considered finding the variable name from matObj and building a string "matObj.varName" and then tried to make a function out of it using str2func, but couldn't get it to work. Otherwise I have tried defining another temporary variable that stores the string "varName", but it tries to find that variable in the mat file (which isn't there).

Any help would be appreciated.
Thanks!

On Thursday, July 5, 2012 2:31:07 PM UTC+12, Iain wrote:
> Hello,
>
> I am trying to load part of a variable stored in a mat file, but will not always know the name of the variable (This is the only variable in the file, and is an array ). Consider a file array1.mat, in which there is a variable called "varName" that is a big matrix. I know you can use the following code
>
> array1_name = sprintf('array1')
> matObj = matfile(array1_name);
> part = matObj.varName(1:2,1:2);
>
> but this requires knowing and hard coding in "varName" (which I will not always know). I cannot load the whole variable into memory as they are too large. I have considered finding the variable name from matObj and building a string "matObj.varName" and then tried to make a function out of it using str2func, but couldn't get it to work. Otherwise I have tried defining another temporary variable that stores the string "varName", but it tries to find that variable in the mat file (which isn't there).
>
> Any help would be appreciated.
> Thanks!

I'm not familiar with the function matfile and I don't know what it does, but here's how I'd do it:
s=load('array1');
fname=fieldnames(s); % Retrieve all the field names in the structure
data=s.(fname{1})(1:2,1:2); % Extract some of the data from the 1st field name

On 7/4/2012 9:31 PM, Iain wrote:
...

> I am trying to load part of a variable stored in a mat file, but will
> not always know the name of the variable (This is the only variable in
> the file, and is an array ). Consider a file array1.mat, in which there
> is a variable called "varName" that is a big matrix. I know you can use
> the following code
> array1_name = sprintf('array1')
> matObj = matfile(array1_name);
> part = matObj.varName(1:2,1:2);
>
> but this requires knowing and hard coding in "varName" (which I will not
> always know)....

Use the .who method of the object...

<http://www.mathworks.com/help/techdoc/ref/matlab.io.matfileclass.html>

--

TideMan <mulgor@gmail.com> wrote in message <e84cafe5-69ec-4e7e-b704-8aa03e5711fa@googlegroups.com>...
> On Thursday, July 5, 2012 2:31:07 PM UTC+12, Iain wrote:
> > Hello,
> >
> > I am trying to load part of a variable stored in a mat file, but will not always know the name of the variable (This is the only variable in the file, and is an array ). Consider a file array1.mat, in which there is a variable called "varName" that is a big matrix. I know you can use the following code
> >
> > array1_name = sprintf('array1')
> > matObj = matfile(array1_name);
> > part = matObj.varName(1:2,1:2);
> >
> > but this requires knowing and hard coding in "varName" (which I will not always know). I cannot load the whole variable into memory as they are too large. I have considered finding the variable name from matObj and building a string "matObj.varName" and then tried to make a function out of it using str2func, but couldn't get it to work. Otherwise I have tried defining another temporary variable that stores the string "varName", but it tries to find that variable in the mat file (which isn't there).
> >
> > Any help would be appreciated.
> > Thanks!
>
> I'm not familiar with the function matfile and I don't know what it does, but here's how I'd do it:
> s=load('array1');
> fname=fieldnames(s); % Retrieve all the field names in the structure
> data=s.(fname{1})(1:2,1:2); % Extract some of the data from the 1st field name


This worked perfectly! Thanks!
Do you know of a similar way to add to a variable in a mat file?

On 7/5/2012 8:46 AM, Iain wrote:
> TideMan <mulgor@gmail.com> wrote in message
> <e84cafe5-69ec-4e7e-b704-8aa03e5711fa@googlegroups.com>...
>> On Thursday, July 5, 2012 2:31:07 PM UTC+12, Iain wrote:
...

>> ... I cannot load the whole variable into memory as
>> they are too large. ...
>>
>> I'm not familiar with the function matfile and I don't know what it
>> does, but here's how I'd do it:
>> s=load('array1');
>> fname=fieldnames(s); % Retrieve all the field names in the structure
>> data=s.(fname{1})(1:2,1:2); % Extract some of the data from the 1st
>> field name
>
>
> This worked perfectly! Thanks!
...

OK, I'm curious--my release has none of this expanded stuff about
multilevel indexing nor the matfile() function so can't test it but
according to the doc I read and the way load() works in this release
(which I did try) the form

s=load('array1');

loads the entire mat-file into the structure s and the

data=s.(fname{1})(1:2,1:2);

simply makes a copy of a subset of that. It would seem that would be
what had to have happened in your case, too, or there wouldn't have been
anything in the resulting data array if all load() now does is return
the variable information from the mat-file to a structure and not the
data (which would seem a major behavioral change in Matlab).

So, the end question is == If the array in question were really so
large, wouldn't that cause the problem you were trying to avoid? Or
was/is it not actually a problem after all?

--

dpb <none@non.net> wrote in message <jt46gg$h9a$1@speranza.aioe.org>...
> On 7/5/2012 8:46 AM, Iain wrote:
> > TideMan <mulgor@gmail.com> wrote in message
> > <e84cafe5-69ec-4e7e-b704-8aa03e5711fa@googlegroups.com>...
> >> On Thursday, July 5, 2012 2:31:07 PM UTC+12, Iain wrote:
> ...
>
> >> ... I cannot load the whole variable into memory as
> >> they are too large. ...
> >>
> >> I'm not familiar with the function matfile and I don't know what it
> >> does, but here's how I'd do it:
> >> s=load('array1');
> >> fname=fieldnames(s); % Retrieve all the field names in the structure
> >> data=s.(fname{1})(1:2,1:2); % Extract some of the data from the 1st
> >> field name
> >
> >
> > This worked perfectly! Thanks!
> ...
>
> OK, I'm curious--my release has none of this expanded stuff about
> multilevel indexing nor the matfile() function so can't test it but
> according to the doc I read and the way load() works in this release
> (which I did try) the form
>
> s=load('array1');
>
> loads the entire mat-file into the structure s and the
>
> data=s.(fname{1})(1:2,1:2);
>
> simply makes a copy of a subset of that. It would seem that would be
> what had to have happened in your case, too, or there wouldn't have been
> anything in the resulting data array if all load() now does is return
> the variable information from the mat-file to a structure and not the
> data (which would seem a major behavioral change in Matlab).
>
> So, the end question is == If the array in question were really so
> large, wouldn't that cause the problem you were trying to avoid? Or
> was/is it not actually a problem after all?
>
> --

I think matfile() is something that is in R2011 and later. I have R2011b, and I managed to load from parts of the variable from the mat file using:

array1_name = sprintf('file_A');
matObj = matfile(array1_name);
fname=fieldnames(matObj);
arraypart=matObj.(fname{2})(1:2,1:2);

This avoids loading the whole variable using a combination of matfile() and fieldnames(). I deal with variables in my work that are upwards on 30+GB, so I needed an ability to load only parts of them if i wanted to say multiply 2 matrices on a machine with only 8GB of ram.

Now I have to figure out how to do the opposite, by adding to a variable in a mat file with an arbitrary variable name.

"dpb" <none@non.net> wrote in message news:jt46gg$h9a$1@speranza.aioe.org...
> On 7/5/2012 8:46 AM, Iain wrote:
>> TideMan <mulgor@gmail.com> wrote in message
>> <e84cafe5-69ec-4e7e-b704-8aa03e5711fa@googlegroups.com>...
>>> On Thursday, July 5, 2012 2:31:07 PM UTC+12, Iain wrote:
> ...
>
>>> ... I cannot load the whole variable into memory as
>>> they are too large. ...
>>>
>>> I'm not familiar with the function matfile and I don't know what it
>>> does, but here's how I'd do it:
>>> s=load('array1');
>>> fname=fieldnames(s); % Retrieve all the field names in the structure
>>> data=s.(fname{1})(1:2,1:2); % Extract some of the data from the 1st
>>> field name
>>
>>
>> This worked perfectly! Thanks!
> ...
>
> OK, I'm curious--my release has none of this expanded stuff about
> multilevel indexing nor the matfile() function so can't test it but
> according to the doc I read and the way load() works in this release
> (which I did try) the form
>
> s=load('array1');
>
> loads the entire mat-file into the structure s and the
>
> data=s.(fname{1})(1:2,1:2);
>
> simply makes a copy of a subset of that.

That's correct.

> It would seem that would be what had to have happened in your case, too,
> or there wouldn't have been anything in the resulting data array if all
> load() now does is return the variable information from the mat-file to a
> structure and not the data (which would seem a major behavioral change in
> Matlab).

LOAD and MATFILE are not the same. LOAD loads entire variables from the
file; under certain circumstances (very roughly, version 7.3 MAT-files with
subscripted indexing into certain types of arrays) MATFILE can read/write
_pieces_ of variables in a MAT-file without reading/writing the whole
variable.

http://www.mathworks.com/help/techdoc/ref/matfile.html

--
Steve Lord
slord@mathworks.com
To contact Technical Support use the Contact Us link on
http://www.mathworks.com

On 7/5/2012 9:20 AM, Iain wrote:
...

> I think matfile() is something that is in R2011 and later. I have
> R2011b, and I managed to load from parts of the variable from the mat
> file using:
>
> array1_name = sprintf('file_A');
> matObj = matfile(array1_name);
> fname=fieldnames(matObj); arraypart=matObj.(fname{2})(1:2,1:2);

Yes, I'm aware matfile postdates my release... :)

Did you look at the link I posted on the .who method for the matfile io
object? Looked to me like it was more straightforward route based on
the doc's. From the online doc's...

> Methods
> size Array dimensions
> who Names of variables in MAT-file
> whos Names, sizes, and types of variables in MAT-file
>
> You cannot access help for these methods using the help command. Find
> help on the methods from the command line using the doc command, such
> asdoc matlab.io.MatFile/size.

...

> This avoids loading the whole variable using a combination of matfile()
> and fieldnames(). I deal with variables in my work that are upwards on
> 30+GB, so I needed an ability to load only parts of them if i wanted to
> say multiply 2 matrices on a machine with only 8GB of ram.

OK, but the s-load() form that you said above worked doesn't avoid that
does it--even in a new release? That's the part I didn't/don't grok.

> Now I have to figure out how to do the opposite, by adding to a variable
> in a mat file with an arbitrary variable name.

I think you're talking the API here and rewriting a file--a matrix in a
.mat file is a contiguous stream w/ some header data thrown in.

That link to the matlab.io.matfile class describes the writing semantics
the class supports. I didn't study it in depth but I think you've your
work cut out for you if I understand what you're after... :)

--

On 7/5/2012 9:53 AM, Steven_Lord wrote:
> "dpb" <none@non.net> wrote in message
> news:jt46gg$h9a$1@speranza.aioe.org...
>> On 7/5/2012 8:46 AM, Iain wrote:
>>> TideMan <mulgor@gmail.com> wrote in message
>>> <e84cafe5-69ec-4e7e-b704-8aa03e5711fa@googlegroups.com>...
...
>>>> ... here's how I'd do it:
>>>> s=load('array1');
>>>> fname=fieldnames(s); % Retrieve all the field names in the structure
>>>> data=s.(fname{1})(1:2,1:2); % Extract some of the data from the 1st
>>>> field name
>>>
>>>
>>> This worked perfectly! Thanks!
>> ...
>>
>> OK, I'm curious--my release has none of this expanded stuff about
>> multilevel indexing nor the matfile() function so can't test it but
>> according to the doc I read and the way load() works in this release
>> (which I did try) the form
>>
>> s=load('array1');
>>
>> loads the entire mat-file into the structure s and the
>>
>> data=s.(fname{1})(1:2,1:2);
>>
>> simply makes a copy of a subset of that.
>
> That's correct.
>
...

>
> LOAD and MATFILE are not the same. LOAD loads entire variables from the
> file; under certain circumstances (very roughly, version 7.3 MAT-files
> with subscripted indexing into certain types of arrays) MATFILE can
> read/write _pieces_ of variables in a MAT-file without reading/writing
> the whole variable.
>
> http://www.mathworks.com/help/techdoc/ref/matfile.html
>

Yes, I grok that, Steven. For once I actually did go and read the
current release doc's on MATFILE as well as checking that LOAD syntax,
anyway, hadn't been grossly modified in a later release. :)

It was Iain's apparent agreement w/ Tideman that the s=load() operation
and the subsequent extraction of a subset of data from the returned
structure seemed to have satisfied his requirement given the previously
stated very large matrices that gave him memory problems that I couldn't
follow as satisfying that concern at all.

You've confirmed what I thought--it won't "work" to solve his problem if
he does run into that case.

--

dpb <none@non.net> wrote in message <jt4c0q$vsc$1@speranza.aioe.org>...
> On 7/5/2012 9:53 AM, Steven_Lord wrote:
> > "dpb" <none@non.net> wrote in message
> > news:jt46gg$h9a$1@speranza.aioe.org...
> >> On 7/5/2012 8:46 AM, Iain wrote:
> >>> TideMan <mulgor@gmail.com> wrote in message
> >>> <e84cafe5-69ec-4e7e-b704-8aa03e5711fa@googlegroups.com>...
> ...
> >>>> ... here's how I'd do it:
> >>>> s=load('array1');
> >>>> fname=fieldnames(s); % Retrieve all the field names in the structure
> >>>> data=s.(fname{1})(1:2,1:2); % Extract some of the data from the 1st
> >>>> field name
> >>>
> >>>
> >>> This worked perfectly! Thanks!
> >> ...
> >>
> >> OK, I'm curious--my release has none of this expanded stuff about
> >> multilevel indexing nor the matfile() function so can't test it but
> >> according to the doc I read and the way load() works in this release
> >> (which I did try) the form
> >>
> >> s=load('array1');
> >>
> >> loads the entire mat-file into the structure s and the
> >>
> >> data=s.(fname{1})(1:2,1:2);
> >>
> >> simply makes a copy of a subset of that.
> >
> > That's correct.
> >
> ...
>
> >
> > LOAD and MATFILE are not the same. LOAD loads entire variables from the
> > file; under certain circumstances (very roughly, version 7.3 MAT-files
> > with subscripted indexing into certain types of arrays) MATFILE can
> > read/write _pieces_ of variables in a MAT-file without reading/writing
> > the whole variable.
> >
> > http://www.mathworks.com/help/techdoc/ref/matfile.html
> >
>
> Yes, I grok that, Steven. For once I actually did go and read the
> current release doc's on MATFILE as well as checking that LOAD syntax,
> anyway, hadn't been grossly modified in a later release. :)
>
> It was Iain's apparent agreement w/ Tideman that the s=load() operation
> and the subsequent extraction of a subset of data from the returned
> structure seemed to have satisfied his requirement given the previously
> stated very large matrices that gave him memory problems that I couldn't
> follow as satisfying that concern at all.
>
> You've confirmed what I thought--it won't "work" to solve his problem if
> he does run into that case.
>
> --

Yea, it was the use of:

fname=fieldnames(s); % Retrieve all the field names in the structure
data=s.(fname{1})(1:2,1:2); % Extract some of the data from the 1st field name

that Tideman suggested that got it to work for me. I avoided using the load() to prevent loading a whole variable into memory, and used matfile() instead

s = matfile(array1_name);
fname=fieldnames(s);
arraypart=s.(fname{2})(1:2,1:2);

By combining these two, and using a similar reversed command

s.(fnameB{2})(1:2,1:2) = arraypartB;

I was able to save to part of a variable already in a mat file.

On 7/5/2012 10:46 AM, Iain wrote:
...

> Yea, it was the use of:
>
> fname=fieldnames(s); % Retrieve all the field names in the structure
> data=s.(fname{1})(1:2,1:2); % Extract some of the data from the 1st
> field name
>
> that Tideman suggested that got it to work for me. I avoided using the
> load() to prevent loading a whole variable into memory, and used
> matfile() instead
>
> s = matfile(array1_name);
> fname=fieldnames(s); arraypart=s.(fname{2})(1:2,1:2);
>
> By combining these two, and using a similar reversed command
> s.(fnameB{2})(1:2,1:2) = arraypartB;
>
> I was able to save to part of a variable already in a mat file.

OK, I gotcha'...it was enough of a hint on a direction to try not the
actual form itself that helped.

Interesting it does the inplace rewrite. Did that replace the 2x2 array
only, I guess? It's certainly possible to count bytes and replace in a
memory buffer then flush.

Have you tried that for performance on a "really, really, big shew" file?

--

dpb <none@non.net> wrote in message <jt4f4t$7sg$1@speranza.aioe.org>...
> On 7/5/2012 10:46 AM, Iain wrote:
> ...
>
> > Yea, it was the use of:
> >
> > fname=fieldnames(s); % Retrieve all the field names in the structure
> > data=s.(fname{1})(1:2,1:2); % Extract some of the data from the 1st
> > field name
> >
> > that Tideman suggested that got it to work for me. I avoided using the
> > load() to prevent loading a whole variable into memory, and used
> > matfile() instead
> >
> > s = matfile(array1_name);
> > fname=fieldnames(s); arraypart=s.(fname{2})(1:2,1:2);
> >
> > By combining these two, and using a similar reversed command
> > s.(fnameB{2})(1:2,1:2) = arraypartB;
> >
> > I was able to save to part of a variable already in a mat file.
>
> OK, I gotcha'...it was enough of a hint on a direction to try not the
> actual form itself that helped.
>
> Interesting it does the inplace rewrite. Did that replace the 2x2 array
> only, I guess? It's certainly possible to count bytes and replace in a
> memory buffer then flush.
>
> Have you tried that for performance on a "really, really, big shew" file?
>
> --

The example I showed filled the top left 2x2 part of a much larger array. I load the partitions of arrays based on the available system memory. As far as performance goes, since I use it for matrix multiplication, its a tradeoff between loading fewer partitions of a larger size for performing the multiplication versus loading more smaller partitions and doing more quick multiplications. In the end it just makes the process of multiplying two incredibly large matrices on a machine that could not otherwise do the calculation.

On 7/5/2012 7:38 PM, Iain wrote:
...

> The example I showed filled the top left 2x2 part of a much larger
> array. I load the partitions of arrays based on the available system
> memory. As far as performance goes, since I use it for matrix
> multiplication, its a tradeoff between loading fewer partitions of a
> larger size for performing the multiplication versus loading more
> smaller partitions and doing more quick multiplications. In the end it
> just makes the process of multiplying two incredibly large matrices on a
> machine that could not otherwise do the calculation.

I got that... :)

I was just wondering whether it became a serious bottleneck when had to
deal w/ the i/o of small(er) pieces in a large block on disk--I suppose
it's swapping some memory for disk cache/buffer w/ that for use in
computations. When machines now have several Gig memory it makes us old
fogies gasp when remembering the mainframe upgrade to the before
unfathomable 128k words not to mention the 2M!B extended core. And it
was only a mere $8-10M back when a fully-loaded automobile might be a
whopping $3k.

--