How do I process big datasets ??

1 Ansicht (letzte 30 Tage)
Add
Add am 5 Apr. 2016
Kommentiert: Walter Roberson am 5 Apr. 2016
I have data of on file per day , that goes for each day of the month and then each month of the year. I now process daily files separately for daily results. How can I do this for complete analysis of full one year file. My file is like this YYYYMMDD.loc ; this means YYYY- Years, MM- months, DD-day. What if I want to take data of particular month and analyse ??

Akzeptierte Antwort

Ced
Ced am 5 Apr. 2016
Bearbeitet: Ced am 5 Apr. 2016
Hi
Do you now have each day in a separate file, or everything in one file?
Personally, if you want to evaluate different sections of time, I would keep each day in a separate file. Then, you can always just read the files you need.
E.g. Let's say you want to read the months of June and July of years 2013-2015:
(I am assuming that you want to read the same days and months for each year here)
years = 2013:2015;
months = 6:7;
days = [ 1 1 ; 30 31 ]; % first and last day to read, one month per column
N_years = length(years);
N_months = length(months);
N_days = sum(diff(days)+1); % Note: 1 is added to EACH month
N_files = N_years*N_months*N_days;
filenames = cell(N_files,1);
current_index = 1; % makes indexing easier
for i = 1:N_years
for j = 1:N_months
next_index = current_index + days(2,j)-days(1,j);
filenames(current_index:next_index,1) = ...
arrayfun(@(x)sprintf('%i%02i%02i.loc',years(i),months(j),x), days(1,j):days(2,j), 'UniformOutput',0);
current_index = next_index+1;
end
end
Then, you can load all the data you need
for i = 1:N_files
% load data, process, combine.... whatever you need
end
Cheers
  2 Kommentare
Add
Add am 5 Apr. 2016
Where does it load the data ?
Walter Roberson
Walter Roberson am 5 Apr. 2016
As Ced says,
Then, you can load all the data you need
for i = 1:N_files
% load data, process, combine.... whatever you need
end

Melden Sie sich an, um zu kommentieren.

Weitere Antworten (0)

Kategorien

Mehr zu Data Type Identification finden Sie in Help Center und File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by