Job management, control flow

6 Ansichten (letzte 30 Tage)
Benjamin
Benjamin am 14 Okt. 2014
Kommentiert: Benjamin am 26 Nov. 2014
I work with large (0.2 - 1 Tb) sets of 3D image data. I would like to set up a queue within a function that reads in an 'on-deck' image into RAM while data is being processed on the GPU. I can sort out the details, but would be grateful if someone could help point me in the right direction to do the following:
Read in 2 files run loop1 on file1 ; end loop ; clear file 1
run loop1 on file 2; while running, load file3 into memory.
...and so on.
In other words, i need to essentially be able to send a child process to the gpu, and wait on its return value

Akzeptierte Antwort

Jon Boerner
Jon Boerner am 21 Okt. 2014
Bearbeitet: Jon Boerner am 21 Okt. 2014
Hi Benjamin,
The easiest way to do this is probably using either the batch function, or the parfeval function in newer versions. Basically, you would use one of those functions to start a worker which would handle interfacing with the GPU, while the main thread moves on to loading the next file. The code might look something like:
loadedData = loaddata(...);
c = parcluster('local');
c.NumWorkers = 2; % You may want more workers depending on how many GPU's you have, etc.
j = batch(c,@myGPUFcn,1,{loadedData});
while(...)
loadedData = loadnextdata(...);
j.wait(); %Wait for GPU to finish in case file loaded faster
outputs = j.fetchOutputs;
end
function y = myGPUFcn(data)
% make gpuArray and perform operations on it
end
There are some details missing, but that would be the general approach. Using parfeval would be a very similar workflow. loaddata and loadnextdata are just place holders for however you load your files.
Let me know if you have any questions!
  1 Kommentar
Benjamin
Benjamin am 26 Nov. 2014
Jon,
After working through a small test case, the idea you suggest seems to work fine, however, on testing with some more realistic data, I ran into some trouble.
It seems as though when passing the {loadeddata} to the worker, the file is written to disk as a -mat file. Not only is this terribly slow for very large images, it also entirely defeats the purpose, which was to deal with the relatively slow read rate of the data in the first place. (effectively 3x the time.)
Am I missing something obvious in how to pass a workspace variable to a worker, that exists in memory?
(This test data is ~ 6Gb / file, which is still not so large.)
Thanks,
Ben

Melden Sie sich an, um zu kommentieren.

Weitere Antworten (0)

Kategorien

Mehr zu Get Started with MATLAB finden Sie in Help Center und File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by