Code covered by the BSD License  

Highlights from
MATLAB xUnit Test Framework

5.0

5.0 | 40 ratings Rate this file 74 Downloads (last 30 days) File Size: 410 KB File ID: #22846
image thumbnail

MATLAB xUnit Test Framework

by

 

31 Jan 2009 (Updated )

MATLAB xUnit is a unit test framework for MATLAB code.

Editor's Notes:

This file was selected as MATLAB Central Pick of the Week

| Watch this File

File Information
Description

NOTE: As of R2013a (March 2013), MATLAB includes a unit test framework. There are no plans to continue further development of MATLAB xUnit. For information about the new unit test framework in MATLAB, see the documentation (http://www.mathworks.com/help/matlab/matlab-unit-test-framework.html).

MATLAB xUnit Test Framework is a unit test framework for MATLAB code.

MATLAB xUnit is designed to be easy to use for MATLAB users with a wide range of experience. Users can write tests using ordinary M-files that are very simple in structure.

MATLAB xUnit comes with extensive documentation that ranges in scope from a "Getting Started" section to advanced techniques and architectural notes. You can view this documentation online without downloading the package. For example, scroll down to the "Published M Files" section on this page and click on "MATLAB xUnit Quick Start - How to write and run tests." To see all the MATLAB xUnit documentation online, scroll down to the "HTML Files" section on this page and click on "Readme.html."

Only the "xunit" directory is needed to use the framework. The "tests" directory contains the framework's own test suite. The "architecture" directory contains architectural notes on the framework's design and how it might be extended.

MATLAB xUnit can be used with MATLAB releases R2008a and later. MATLAB xUnit relies heavily on object-oriented language features introduced in R2008a and will not work with earlier releases.

Acknowledgements

This file inspired Oblique Shock Calculator, Jsonrpc2: A Set Of Classes To Encode/Decode Json Rpc Messages, Doctest Embed Testable Examples In Your Function's Help Comments, Testit, Xml Test Run Display, and Xunit4.

MATLAB release MATLAB 7.6 (R2008a)
Tags for This File   Please login to tag files.
Please login to add a comment or rating.
Comments and Ratings (144)
07 May 2014 Andy Campbell

Asan, this is a unit testing framework to help ensure your MATLAB programs have been written correctly and to prevent changes from introducing bugs. There is now a unit test framework included in MATLAB, and you may be interested in the following blog post describing how someone can use such a framework to ensure the quality of science & engineering:

http://blogs.mathworks.com/loren/2013/10/15/function-is-as-functiontests/

07 May 2014 asan masraf

I tried to understand 'MATLAB xUnit Test Framework' from description but unfortunately the most part I couldn't understand, can tell me is useful for what in more simple way, sorry I'm new user of MATLAB

24 Feb 2014 Andrew Newell

In R2013a, a curious glitch has emerged in the behavior of the documentation. When I type 'doc', go to the home page, and click on 'Supplemental Software', I get the documentation for this package (and some others). However, the display flickers. MATLAB Support uncovered the source of the problem: the line
<meta http-equiv="REFRESH" content="0;url=xunit_product_page.html">
in the file index.html.

The fix was to delete index.html and replace it with a copy of xunit_product_page.html, renamed to index.html.

27 Oct 2013 Steve Eddins

Keith - On my Mac, MATLAB xUnit passes 150 of the 151 test cases in its own test suite. (The one failure seems to be a bad test case.) Use the contact form on my author page (http://www.mathworks.com/matlabcentral/fileexchange/authors/22204) to send me more details about the problem you are experiencing.

27 Oct 2013 Keith Rogers

Hey, I understand that xUnit is superceded by the official unit test framework in 2013b, but I have a lot of test functions set up for xUnit. When I try to run them in 2013b, even with xUnit put at the top of the path, it doesn't work. Is there a way to either (a) get my old functions to work, or (b) automatically convert my xUnit tests to the new system?

17 Oct 2013 Petr Pošík

Hi all.

I am sorry it took me so long to post here the promised link to the mocking framework created by Vladimir Peric, a student of mine.

You can download mmockito from GitHub:
https://github.com/vperic/mmockito

It is a lightweight package, that definitely cannot do all the mocking things out there, but for the basic usage it shall be quite handy.

It is created using the new MATLAB unit test framework and should be compliant with it. It should also work with Steve Edins' MATLAB xUnit Framework, although maybe with a bit limited capabilities.

Feel free to report any issues directly on the GitHub page.

We hope you will find it useful.

11 Mar 2013 Steve Eddins

David,

You say, "I believe that xUnit creates a new object for each method in that class for testing rather than iterating through the methods of a single (subclassed) TestCase object." That's correct, and that's a classic xUnit design pattern to ensure order independence of individual test methods. See http://martinfowler.com/bliki/JunitNewInstance.html.

Although it's possible to consider adding setup and teardown methods on a TestSuite level, I haven't done so for MATLAB xUnit.

Because of the release of a new unit testing framework in MATLAB itself (in R2013a), I probably will not put further effort into developing my MATLAB xUnit package. If you can upgrade to R2013a, I encourage you to give the new testing framework a try.

11 Mar 2013 David

This maybe a my misunderstanding of the testing architecture. However, I believe that it would be useful if there were an init method in TestCase. This would be run once to setup particular parameters of dependencies. To illustrate my request further, I am using xUnit to test the submodules of a Simulink model which I am building up with model referenced components. I would like the ability to compile the model once in some init method(to take account of any new changes) and then to turn off the automatic rebuild/checking for efficiency of running the subsequent tests. Is this possible? I believe that xUnit creates a new object for each method in that class for testing rather than iterating through the methods of a single (subclassed) TestCase object.

08 Mar 2013 Tim  
26 Feb 2013 Steve Eddins

Daniel, you are correct.

26 Feb 2013 Daniel

In the document 'MATLAB xUnit Test Framework: Architectural Notes'
A function: assertExceptionRaised is mentioned, I think it is supposed to read assertExceptionThrown, which actually exists in matlab x-unit unit tests correct me if I am wrong.

18 Feb 2013 Michael

Hi Petr Posik. If this (http://vperic.blogspot.com/2012/11/announcing-my-bachelors-thesis-mock.html) is your student, I already mentioned my support. Seems like a great project. Looking forward to hearing about the progress.

18 Feb 2013 Pete

A phenomenal piece of work - many thanks.

One minor observation. In terms of learning how to use it: for me personally it would have been nice if the OOP example (TestUsingTestCase.m) was much more prominent (since it seems to give best-practice), and if all the other examples were tucked away or removed altogether. I think if I'd seen TestUsingTestCase.m straight-away, it would have taken me a couple of minutes to get started. As it was it took more like 20 mins (which still isn't too bad). Maybe that's just me though.

18 Feb 2013 Steve Eddins

Michael - no, I don't know anything about that.

15 Feb 2013 Petr Pošík

@Michael: A student of mine works on a mocking framework for MATLAB right now. Give us a few weeks. We will definitely post an announcement here.

15 Feb 2013 Michael

This framework is excellent. Thank you. One question: do you know of any way to create mock objects (http://en.wikipedia.org/wiki/Mock_object) and functions?

15 Feb 2013 Michael  
31 Jan 2013 IainIsm

@ Carl

"...Content submitted to File Exchange may only be used with MathWorks products."

From 2.a.iii at http://www.mathworks.co.uk/matlabcentral/termsofuse.html#access

13 Nov 2012 Daniel Armyr

Two years old or not, this package still rocks.

24 Aug 2012 Stefan

Steve,

is there a way to let 'runtests' iterate over test subdirectories automatically ?
Thanks,

21 Jul 2012 Carl

Hi,

nice to see that a unit test framework has been written for Matlab.

Is this project still maintained? The last version is almost two years old.

I also wonder if anyone knows if the framework can be run with Octave, or if it can be easily ported?

06 Jul 2012 Stefan

Steve,

oh, I see ! I thought these functions were actually nested, and my adding the "end" markers would only affect syntac, not semantics.
Now I understand that I actually misunderstood how things work, so I acceidentally changed the semantics.

All is working fine now. Thanks for your prompt help !

06 Jul 2012 Steve Eddins

Stefan, are you trying to nest your test functions inside the top-level functions? That is not supported.

There should be no problem terminating your functions with "end" as long as you are not trying to nest them.

06 Jul 2012 Stefan

Steve,

I have just discovered this unit testing framework. Thanks for sharing this !

I'm trying to write a simple test suite, with multiple nested test functions in a single .m file. The MATLAB docs suggest that all nested functions need to terminate with "end".

Modifying your "testFliplr.m" file to insert these "end" keywords makes runtests no longer recognize and run the tests from this file.
Am I missing something, or is this a bug in the harness' test parser ?
I'm using MATLAB R2012a.

Thanks,
Stefan

05 Jul 2012 Steve Eddins

Chris, the setup and teardown methods are intended to be called for each test case method that is executed. That's the normal xUnit style. It makes the test cases independently executable.

05 Jul 2012 chris

I have two test cases as subfunctions and a setup methods. For some reason the setup method is called twice - any reason why it would be called more than once?
(2011a and latest xunit download)

08 Jun 2012 John Lee (Complete)  
18 Apr 2012 Patrick Cloke  
15 Apr 2012 Erik

I just converted some tests over from being a mixture of scripts and functions to class-based (inherit from TestCase) xUnit tests. It works fine, but they take about 3x as long to complete. I ran the profiler and it appears that there are a few very "hot" methods that consume 20-30 times more time when run inside xUnit for the same number of invocations.

Anyone have an idea why?

09 Mar 2012 Steve Eddins

Patrice, it is likely that you have made a coding error of some sort in constructing your test files. That would explain why the test suite appears to be empty. Try placing one of your test files outside of a package and running it directly to see if it does what you expect.

09 Mar 2012 Steve Eddins

Federico, if you want to use a simple script as a test case you can. Follow the example in exQuickStart.html.

If you want to use subfunctions as multiple test cases in a single file, then you must use the function line at the top of the file because scripts in MATLAB can't contain functions.

01 Mar 2012 Steve Eddins

Emmet, assertAlmostEqual has been obsolete since version 2.0 of MATLAB xUnit, which was released 2.5 years ago. It has been superseded by assertElementsAlmostEqual and assertVectorsAlmostEqual. The assertAlmostEqual function is only mentioned in the doc in two places: release-history.html, which tells you when it became obsolete, and the main doc page, xunit_product_page.html, which tells you where you can find it if you still need it (in the obsolete folder).

MATLAB xUnit works just fine in R2011a. Try running its own test suite by adding the xunit folder to the path, changing to the tests directory, and typing runtests. 151 test cases execute successfully in R2011a, including a large number of cases testing the subfunction style.

Automatic test case discovery in ordinary functions is difficult to implement because the MATLAB language has barely enough functionality to do it. As you noted, it requires the use of a script and an undocumented variant of which. When a test writer makes a coding error writing test case files, it isn't easy to determine what is wrong with the file other than it errors out when run, or that it doesn't appear to contain any test cases.

But it does work. My own test suite for the MATLAB files in the book Digital Image Processing Using MATLAB is written entirely using subfunction-style tests and MATLAB xUnit.

I suggest that you start with the working example shown in the doc file exSubfunctionTests.html and go from there.

28 Feb 2012 Emmet

FYI, assertAlmostEqual.m (I erroneously added an 's' to the end in my earlier comment) is not in the 'xunit' directory in the zipfile. You can fetch it manually from the menu (right), but it uses 'mtest.utils.isAlmostEqual()'. If you manually change this to 'xunit.utils.isAlmostEqual()', it then works OK.

I think I've figured out why subfunction tests don't work (at least, not in R2011a). There are a number of problems. When you put in the call to the 'initTestSuite' script, it seems that the (undocumented?) '-subfun' option to 'which', which it uses to discover the subfunction names, just doesn't work, and always returns an empty array. Then, even if you can hack it to get the names of the subfunctions reliably, which I did by hacking Robert Bemis's 'SubFuns', it seems that you're in a "Catch 22": the initTestSuite script has its own variables, of course, but in the context of a function with subfunctions, adding new variables is not allowed, apparently. OTOH if you convert initTestSuite to a function, it is not then running in the context of the function with subfunctions, and the subfunctions aren't then visible, and the 'str2func' on their names (even though you now know them) fails.

Hope this helps someone save the day I just wasted.

28 Feb 2012 Emmet

OK, so my original comment appears to have disappeared for some reason.

In short, assertAlmostEquals() doesn't work, maybe because there's no assertAlmostEquals.m in the download.

Second, using subfunction tests, as described in "How to Put Multiple Test Cases in One M-file" doesn't appear to work at all, returning the error:

--------------------
??? Error using ==> runtests at 100
No test cases found.
--------------------

The reason for this appears to be that line 230 in TestSuite.m ( 'suite=feval(name);') gets executed and throws an exception, resulting in an empty TestSuite.

If you put this line on the Matlab command-line, it says:

--------------------
??? Attempt to add "ST" to a static workspace.
See MATLAB Programming, Restrictions on Assigning to Variables for
details.

Error in ==> initTestSuite at 18
[ST,I] = dbstack('-completenames');

Error in ==> testFluid at 2
initTestSuite;
--------------------

28 Feb 2012 Emmet

Sorry, I mean "empty TestSuite being *returned*" not "being *caught*", above.

31 Jan 2012 Federico

Is there a reason why 'runtests' ignores scripts (rather than functions)? If I forget the useless line 'function testBLAH' at the beginning of a test case, then that test is silently ignored. This does not seem a sound practice, since it can lead to unnoticed failure. I strongly suggest you to fix this behaviour, unless it is intended for some reason.

21 Dec 2011 Patrice

I am discovering this usefull framework and testing with examples.
It seems that package content isn't taken in account running

runtests test_packageName
or
runtests 'xunit.mocktests'

The suite.TestComponents is always empty.

Thanks for your help.

20 Dec 2011 Steve Eddins

Maddy, I'm sorry, but I am not familiar with the details of deploying applications using the MATLAB Compiler.

20 Dec 2011 Maddy

I am creating the following function:

function mytests
runtests('/opt/tests');
end

So now I build this function by running deploytool and I give the path for the additional libraries as /opt/tests and the path for the source files of functions I am testing in my tests. Then I run the generated executable from the unix command line which is outside the MATLAB environment. It gives me the same error.
If I call runtests('/opt/tests') on the MATLAB command line it works fine.

20 Dec 2011 Maddy

I am creating the following function:

function mytests
runtests('/opt/tests');
end

So now I build this function by running deploytool and I give the path for the additional libraries as /opt/tests and the path for the source files of functions I am testing in my tests. Then I run the generated executable from the unix command line which is outside the MATLAB environment. It gives me the same error.
If I call runtests('/opt/tests') on the MATLAB command line it works fine.

20 Dec 2011 Steve Eddins

Maddy, the directory containing your tests needs to be on the MATLAB path in order for MATLAB to run the test code found in that directory.

20 Dec 2011 Maddy

Hi Steve! I am trying to create a function that calls runtests.m with the specified directory name that contains my tests and then compile it. My problem is that when I try to run the executable of my function from unix command line it errors out saying that, "error in file: no test cases found."
How can I make it see my tests?
Thanks in advance!

15 Dec 2011 Steve Eddins

Jan, if all the tests are in the same directory, and if you only want to run a subset of the tests, but there's no naming pattern to easily select the desired subset, then ... the only suggestion I can think of is to write a script to automate the tediousness of typing in all the test file names manually.

15 Dec 2011 Jan Stolarek

Steve >> There is no name pattern. I just throw my tests into one directory (which perhaps is not the best idea...) and I know that some of theme belong to one program, the others belong to another.

05 Dec 2011 John Colby  
05 Dec 2011 Jonathan Karr

I've posted a JUnit-style XML logger here:
http://www.mathworks.com/matlabcentral/fileexchange/33971-xmltestrundisplay

The code was built with xunit version 3.0.1.

05 Dec 2011 Steve Eddins

Jan, write a function to do it. How do you know which test cases should be selected? By some name pattern, or some other criteria?

04 Dec 2011 Jan Stolarek

Hm... but how could I add many classes to a TestSuite using this?

02 Dec 2011 Steve Eddins

Jan, try using the static method TestSuite.fromTestCaseClassName().

02 Dec 2011 Jan Stolarek

I have a usage problem. I have some test classes that extend TestCase class. I would like to group some of these classes into a suite that runs them. How do I do that? I don't know how to use add method from TestSuite class. I came up with something like this:

suite = TestSuite( 'NameOfTheSuite' );
suite.add( testClassNo1Test( 'testMethodNo1' ) );
suite.add( testClassNo1Test( 'testMethodNo2' ) );
suite.add( testClassNo2Test( 'testMethodNo1' ) );
suite.add( testClassNo2Test( 'testMethodNo2' ) );

So basically I have to manually specify the name of every test method which is very tedious task. I don't want to use TestCase.fromPwd() - I want to create test suite containing only selected test cases.

29 Nov 2011 Rodrigo Starr  
25 Oct 2011 Ryan Ollos

Here is the workaround I came up with for now. Define `packageName`, e.g. `mypackage.test`, then build up the test suite from the files in the package. There is probably a more straightforward way to implement something like this directly xunit, which I'll investigate when I have more time.

ts = TestSuite();
for i = 1:numel( fileName )
fqClassName = [ packageName '.' fileName(i).name(1:end-2) ];
if xunit.utils.isTestCaseSubclass( fqClassName )
ts.add( TestSuite( fqClassName ) );
end
end

ts.run();

25 Oct 2011 Ryan Ollos

Dennis, thanks for the proposed fix. This fix wouldn't directly apply to my situation because I'm running tests cases in a package, by passing the package name to `runtests`. If I try specifying the relative or full path to a package, I get "No test cases found", which seems to be because fully qualified function names are not being used to build up the test suite. So your fix might work for the case of specifying a package name if some additional modifications were made to the `fromName` function.

When a package name is specified when calling runtests, the issue seems to be that matlab's meta.package.fromName (line 283 in TestSuite.m) will not return in the list of classes any class that has a syntax error. I'm not sure how to work around that.

25 Oct 2011 Steve Eddins

Dennis, thanks for the suggested fix.

25 Oct 2011 Dennis Wouters

To Ryan (and Steve of course),

I had a similar issue. I group multiple test cases in a file and place all of the test files in one directory. Then I call runtests with the directoryname as the argument.
I also noticed that if one of the test cases in one file has a syntax error, the whole file will be skipped silently.

To mend this, I made a small change in 'fromName' method in 'TestSuite.m'.

Original:
>>>
else

try
if nargout(name) == 0
<<<

modified:
>>>
else
try
nout = nargout(name);
catch
warning('''%s'' does not appear to be a valid function.',...
name)
end
try
if nout == 0
<<<

If the file has a syntax error, the nargout test will fail and a warning is printed before the actual testing starts.

Hopefully this will help you too.

21 Oct 2011 Steve Eddins

James, thanks for the suggestion.

20 Oct 2011 James

I've a bug report: xunit.utils.isTestCaseSubclass should return false for Abstract classes.

This allows you to specify some tests in an abstract class derived from TestCase, then implement concrete classes that use different implementations so that the tests are shared.

For example, look at the jUnit tests in: http://timefinder.svn.sourceforge.net/viewvc/timefinder/trunk/timefinder-algo/src/test/java/de/timefinder/algo/roomassignment/

07 Oct 2011 Joseph Burgel

Nice Tool. One question an maybe my philosophy is just wrong (new to TDD) but is there any way to have the code break on error if the code in a given test errors? (i.e. dbstop if error) Thanks,

29 Sep 2011 Ryan Ollos

Bug report: I have a package of tests, let's call it mypkg.tests. I run all of the tests in the package by typing `runtests mypkg.tests`. I have noticed that if there is a syntax error in one of the test classes, it will be silently skipped. However, if I run the test class by specifying the class explicitly, e.g. `runtests mypkg.tests.MyTestClass`, I see that the tests fail to run because of the syntax error. The syntax error can be an incorrect import statement, or something else equally likely to arise in practice. We have hundreds of tests and dozens of test classes, so this can cause problems for us that are not obvious.

29 Sep 2011 Jan Stolarek

This is a great peace of code, definitely should be part of standard Matlab distribution! Since I moved from Java to Matlab I've really missed TDD - now I can use it to full extent. There's however one thing that keeps annoying me. After running "runtests" I would really appreciate some sort of summary at the bottom that would say "X out of Y test failed" or something like that. Now when I run 16 tests I have to count how many have failed and recount it every time I change something in the code, to check if more tests are passing. I know I can use logger to get such information, but it's a bit awkward.

BTW. The name should be mUnit, not xUnit, but I see the name's already taken...

02 Sep 2011 Federico

This should work as a first approximation:

function assertWarningThrown(f,id)
% warning test to match assertExceptionThrown in XUnit
%
% assertWarningThrown(f,id)
%
% throws an error unless f throws a warning with id "id".
% well, in fact, unless it throws *some* warnings, whose last one has id
% "id". I don't see how to make it work in the other way.

s=warning('query',id);
warning('off',id);
lastwarn('dummy warning test','dummy:warning:id');

f();

[~,id_obtained]=lastwarn;
if strcmp(id_obtained,'dummy:warning:id')
error('assertWarningThrown:noWarning','expected warning with message_id %s, but none thrown',id);
elseif not(strcmp(id_obtained,id))
error('assertWarningThrown','expected warning %s, but got warning %s instead',id,id_obtained);
end

warning(s);

end

30 Aug 2011 Steve Eddins

Federico - No.

30 Aug 2011 Federico

Is there a assertWarningThrown?

09 Jun 2011 Matthew M.

That works! Thanks.

08 Jun 2011 Steve Eddins

Matthew, good question. I suggest that you create a "helper" subfunction, something that doesn't begin or end with "test", that creates an object and then does:

myObj.my3Vector = [1 1];

Then you can create a test case that does this:

assertExceptionThrown(@helperFun, 'myClass:3Vec');

08 Jun 2011 Matthew M.

Hi Steve,
This is a superb tool, and it's helping me write better code already. I have a question.

I'm writing a value class that doesn't use the handle graphics set/get functions, and I want to use the xUnit test framework to verify that the proper errors pop up if I set properties to incorrect values, like:

myObj.my3Vector = [1 1];

In the set.my3Vector method I've defined an error message:

function set.my3Vector(vec)
if numel(vec)~=3
error('myClass:3Vec','Needs 3 elements')
end
end

The assert* functions work perfectly in diagnosing problems when I set the values as properties in the class constructor, because the class constructor expression doesn't need an '=' sign. The problem is that in diagnosing the first expression, I can't use an anonymous function handle because of the '=' sign. I guess if I were writing a handle class, the call would be set(myObj,'my3Vector',[1 1]) and there would be no problem. But I don't think I want a handle class for this application (though I could be wrong, being relatively new to OOP).

Is there a workaround I'm missing? I tried putting try-catch blocks in my test file, but it just errored out in the try block (I suppose the point of the test framework is to not use complicated try-catch expressions).

Any advice would be appreciated. Don't know if it matters, but I'm using R2010b.
Thanks!

08 Jun 2011 Matthew M.

Oh - don't know if it matters but I'm using R2010b.

27 May 2011 HaveF

It seems I lost the comment I write just now.

Hi, Steve, I notice you build your project at "31 Jan 2009", maybe you survey the related code of unit test.
I find a similar project(release at 2008-04-08) here:
http://www.mathworks.com/matlabcentral/fileexchange/21888-mlunit2008a
http://sourceforge.net/projects/mlunit/

maybe you can tell me your original consideration , or the differences between them.
or think of merging the mlunit's GUI or other useful thing to your project.

Thanks!

27 May 2011 HaveF

or think of merging the mlunit's GUI or other useful thing to your project.

12 May 2011 Steve  
29 Mar 2011 Steve Eddins

Luke, files that begin or end with "[tT]est" are treated as test files. See this doc page: http://www.mathworks.com/matlabcentral/fx_files/22846/11/content/matlab_xunit/doc/html/exTestCaseSearching.html

28 Mar 2011 Luke

I think there is a bug when you have a fucntion name with 'test' in it,(ie. not at the start of hte name). It processes it as if it were a test.

21 Feb 2011 Andrew McLean

I think the test framework is great.

One thing I find myself wanting is to have test scripts which provide informative output when called standalone, but are "quiet" when called from "runtests".

The following code helps this along:

[st, ~] = dbstack();
isBatch = strcmpi(st(end).name, 'runtests');

But this feels a bit kludgy. I could put it in a function, but maybe this, or something equivalent, could be provided with the framework.

09 Feb 2011 Steve Eddins

Daniel, usually this means that one or more test files are incorrectly written, possibly by omitting the output argument "test_suite" from the function definition line. Take a careful look at this doc page and make sure your test files are written as described: http://www.mathworks.com/matlabcentral/fx_files/22846/11/content/matlab_xunit/doc/html/exSubfunctionTests.html

I will see if I can make the harness smarter about recognizing this kind of malformed test file and producing helpful messages.

09 Feb 2011 Daniel Armyr

Hi.
Wonderful tool. I have been using it now for a few months, and it really does what it says on the box.

One little nitpick though:
If I have a folder full of test cases and run the 'runtests' command, it will run all the tests. If one of the tests fail, that test will be marked as a fail, but the bottom line that summarizes the result of all tests will say 'Pass'. I find it a bit disturbing that there are failed tests, but the last line printed out says 'Pass'. Now, I went through the code to see if there was a easy fix I could do myself, but it seems that the test results reach the ' initTestSuite' function but stop there. I tried to have initTestSuite pass its own result out, but then something in your code protested, so I had to give up for now.

Anyways, if there is a way to get the entire test run to know if every test passed or not and then print that result out would be great.

28 Jan 2011 Steve Eddins

Andrew, there's no special trick I know about for unit testing private methods.

28 Jan 2011 Andrew Newell

I meant class methods with access=private.

27 Jan 2011 Steve Eddins

Andrew, MATLAB xUnit framework code runs in MATLAB with no special scope privileges, so it cannot run private-directory code directly. One possibility is to reorganize your code to put private-directory functions into a package instead. Another possibility is to put a "gateway" function in the directory above the private directory that returns a structure of function handles to the private functions you want to test.

27 Jan 2011 Andrew Newell

Steve, do you have some neat way of testing private methods using your test framework?

27 Jan 2011 Steve Eddins

I am collaborating with Prof. Diane Kelly (Royal Military College, kelly-d@rmc.ca) and Prof. Greg Wilson (University of Toronto, gvwilson@cs.utoronto.ca) on a study to explore how engineers and scientists test their software, and I'd like to invite you to help.

If you use MATLAB xUnit or a similar unit testing framework to test your MATLAB programs, and if you are willing to let us have a look at the tests you have written and the code that they exercise, please get in touch with us by sending e-mail to research@software-carpentry.org.

Ethical guidelines for the study are posted here: http://software-carpentry.org/research/

Thanks for your help!

23 Jan 2011 Steve Eddins

Andrew, thanks for your comments. Only class methods go into a class folders, so you should definitely move the test files elsewhere.

22 Jan 2011 Andrew Newell

A follow-up to the file name problem: it goes away if I move the test folder out of the class folder.

21 Jan 2011 Andrew Newell

I have used other unit testing packages for Matlab and this is the best. It is so easy to use - I am especially pleased with the method for testing error messages. And it is nice to have tests for real numbers built in.

21 Jan 2011 Andrew Newell

Steve, I am getting some strange reactions to file names. I created the following test suite:

-------
function test_suite = test_polystr
initTestSuite;

function testNoInput
out = [];
assert(isempty(out.Value));

function testCharInput
out.Value = 's';
assertEqual(out.Value,'s');
-----

Then I tried the following commands:
>> runtests
??? Error using ==> runtests at 100
No test cases found.

>> runtests('test_polystr.m')
??? Error using ==> runtests at 100
No test cases found.

I get the same problem if I change the name to testConstructor, but if I change it to testPolystr:

>>runtests
Test suite: /Users/andrew/matlab/my_classes/@polystr/polystrTests
21-Jan-2011 10:54:59

Starting test run with 2 test cases.
..
PASSED in 0.003 seconds.

13 Dec 2010 Jonathan Karr

Great job! It would be great if there was a way to mark tests as disabled, and then notify the user in the test report which tests are disabled. This would be great so that you don't forget about disabled tests. Thanks again.

09 Dec 2010 Jonathan Karr

Great job. Thanks for putting this together.

14 Nov 2010 Steve Eddins

Thanks, Jeff. I've made a note to update the documentation regarding this point.

12 Nov 2010 Jeff Beck

Understood, Steve. Then you may just want to throw an error if nargout(self.SetupFcn()) > 1 or add a line to the setup() documentation that says there can only be ONE setup function and it can return at most ONE argument. It might save someone from beating their head against a wall trying to figure out why their additional output from setup ends up undefined. Thanks, again!

11 Nov 2010 Steve Eddins

Hi Jeff. I would prefer to keep the functional definition for setup and teardown as simple as possible. When I need to return multiple pieces of data from setup, I just bundle all the pieces into a single struct.

11 Nov 2010 Nick

Excellent testing framework without a lot of overhead. Good job!

10 Nov 2010 Jeff Beck

Steve,
First let me add my thanks for developing this test framework. I'd like to suggest a fix to a small issue I ran into. When I wanted my setup() function to return more than 1 output argument, I found that only the first output argument was recognized when being passed into the tests. To fix this, I forced FunctionHandleTestCase.TestData to be a cell array. Here's a diff showing my changes to FunctionHandleTestCase.m:
--- FunctionHandleTestCase.m (revision 220)
+++ FunctionHandleTestCase.m (working copy)
@@ -53,7 +53,7 @@
TeardownFcn;

%TestData - Data needed by test function or teardown function.
- TestData;
+ TestData = {};
end

methods
@@ -107,7 +107,7 @@
% pass self.TestData to the test function. Otherwise, call the
% test function with no input arguments.
if ~isempty(self.SetupFcn) && nargout(self.SetupFcn) > 0
- self.TestFcn(self.TestData);
+ self.TestFcn(self.TestData{:});
else
self.TestFcn();
end
@@ -120,7 +120,7 @@
% argument in instance data (TestData).
if ~isempty(self.SetupFcn)
if nargout(self.SetupFcn) > 0
- self.TestData = self.SetupFcn();
+ [self.TestData{1:nargout(self.SetupFcn)}] = self.SetupFcn()
;
else
self.SetupFcn();
end
@@ -135,7 +135,7 @@
% arguments.
if ~isempty(self.TeardownFcn)
if ~isempty(self.SetupFcn) && (nargout(self.SetupFcn) > 0)
- self.TeardownFcn(self.TestData);
+ self.TeardownFcn(self.TestData{:});
else
self.TeardownFcn();
end

09 Nov 2010 Joel  
02 Nov 2010 Steve Eddins

Cengiz - No, I haven't done anything with Andrei's suggestion. Your own assert case is already supported by assertElementsAlmostEqual by specifying 'absolute' for the tolerance type.

02 Nov 2010 Cengiz Gunay

Is the idea of 'precision' implemented in your assert functions? Andrei Rotenstein above suggested it.

I also came up with my own assert case for checking the 'maximal' error during a vector comparison; e.g.,

assert(max(abs(m_int - m_ideal)) < 1e-1);

It would be nice to have it implemented inside assertElementsAlmost Equal. A shorter name for the function would also make test functions clearer ;)

Otherwise this is a simple and great framework. Thanks!

28 Oct 2010 Michael Teo

Thanks Steve and Robert.

Changing the current dir / setting the MATLAB path sound like a good workaround. I will try it out, cheers!

28 Oct 2010 Robert Kagy

In the past I've written test scripts outside the MATLAB xUnit Test Framework that modify the current directory or the MATLAB path so the stub functions appear first. Taking advantage of the MATLAB function name resolution rules to call something else instead.

Unfortunately, I'm still stuck developing in R2007b, haven't used the MATLAB xUnit Test Framework, and so don't have any idea if this is something the Framework or the individual test case is in a better position to handle.

28 Oct 2010 Steve Eddins

Michael, the framework doesn't call the stub, your function computeProjectTime does. The framework doesn't have any way to change MATLAB function name resolution rules in order for your function to call something else instead.

28 Oct 2010 Michael Teo

Definitely Steven, i agreed with you totally - test stub is coded by the test writer. (-;

My question is does this framework support test stub? i.e. how do i 'configure' the framework to call the stub instead of the original function?

To illustrate further, for example to compute project timeline:

<code>
function totalTime = computeProjectTime
x = computeAnalysisTime();
y = computeDesignTime();
z = computeTestTime();
totalTime = x + y + z;
</code>

The unit tests for functions computeAnalysisTime(), computeDesignTime(), computeTestTime() are done and tested. In order to test function computeProjectTime(), the above 3 functions should be "replaced" by test stubs (using the framework) so as to ensure consistent inputs.

Without consistent inputs, we cannot test for expected outputs.

27 Oct 2010 Steve Eddins

Michael, it sounds like providing a test stub for a portion of the system under test is the responsibility of a test writer, not the test harness.

24 Oct 2010 Michael Teo

hi Steven,

"test stubs are programs which simulate the behaviors of software components (or modules) that are the dependent modules of the module being tested." - http://en.wikipedia.org/wiki/Test_stubs

22 Oct 2010 Steve Eddins

Hi Michael ... I don't know what that means.

22 Oct 2010 Michael Teo

Does this framework support stub?

21 Oct 2010 Steve Hoelzer  
01 Oct 2010 Michael Teo

Thanks for sharing, its a very handy framework.
I will try enhance the framework with addition TestCase "Description" properties so that the verbose output will printout the description.

30 Sep 2010 Steve Eddins

Michael - test_suite is a TestSuite object. This is how the framework gathers up all the test cases into one big hierarchical test suite and then runs them all together. Call one of your test files with an output argument to see what this thing looks like.

There's no HTML reporting tool.

30 Sep 2010 Michael Teo

Is there a reporting tool to generate a html report on the list of test cases, status(passed/failed)?

30 Sep 2010 Michael Teo

may i know what is the output variable 'test_suite' for?

17 Sep 2010 Andrei Rotenstein

Steve,

Although 'absolute' and 'relative' tolerances are sufficient certainly for most unit testing situations in signal processing, I feel that what's lacking is equality testing to a definable level of precision (i.e. in the mantissa).

To this end, I've added support 'precision' as a new ToleranceType. I've added support for the string ''precision'' in the obvious way to ParseFloatAssertInputs, and to CompareFloats.m, I've added to the switch block:

switch params.ToleranceType

the following new case:

case 'precision'

coreCompareFcn = @(A, B) magFcn(chop(A,params.Tolerance) - chop(B,params.Tolerance)) <= eps(max(magFcn(A), magFcn(B)));

I hope that others may find this useful.

15 Jul 2010 David

This has literally changed how I think about and write software! TYVM!

23 Jun 2010 Steve Eddins

I have submitted an update that corrects the problem with test cases in packages reported Jared Jacobs on June 16, 2010.

17 Jun 2010 Jared Jacobs

Thanks for the prompt fix Steve!

16 Jun 2010 Steve Eddins

I've submitted new version 3.0.1 to fix the problem with handling TestCase subclasses inside packages. It should show up on the File Exchange in a day or two.

Here's my fix to the methodIsConstructor function inside TestSuite.m:

function result = methodIsConstructor(method)
method_name = method.Name;
if ~isempty(method.DefiningClass.ContainingPackage)
method_name = [method.DefiningClass.ContainingPackage.Name, '.', ...
method_name];
end
result = strcmp(method_name, method.DefiningClass.Name);
end

16 Jun 2010 Steve Eddins

Jared - sorry about that. I'll work on a fix right away.

16 Jun 2010 Jared Jacobs

I get an error trying to run a test defined in a package too. xUnit tries to run the constructor as a test method. Suppose I put the following code in a +foo directory:

classdef FooTest < TestCase
methods
function object = FooTest(name)
object = object@TestCase(name);
end
function test_sanity(object)
assertEqual(0, 0)
end
end
end

The error message I get is:

>> runtests 'foo'
Starting test run with 2 test cases.
.F
FAILED in 0.060 seconds.

===== Test Case Failure =====
Location: ...\src_test\+foo\FooTest.m
Name: FooTest

...\matlab_xunit_3.0\xunit\TestCase.m at line 74
...\matlab_xunit_3.0\xunit\TestSuite.m at line 85
...\matlab_xunit_3.0\xunit\TestSuite.m at line 85
...\matlab_xunit_3.0\xunit\runtests.m at line 96

Undefined function or method 'FooTest' for input arguments of type 'foo.FooTest'.

I think the problem is in TestSuite.methodIsConstructor, which is comparing 'FooTest' with 'foo.FooTest' and expecting an exact match. One idea for a fix would be to have the method do something like this instead:

result = regexp(method.DefiningClass.Name, ['(.*[.])?' method.Name '$']);

15 Jun 2010 Jared Jacobs  
09 Jun 2010 Steve Eddins

William - Thanks. I'll make the fix.

01 Jun 2010 William Payne

Steve,

I find xUnit to be an essential tool; and I wish to thank you for and congratulate you on its creation.

Further to Arthur's comment of 02 Feb 2010, I too have experienced similar issues; specifically, when attempting to compare strings that contain escape characters. For example:

function test_suite = temp
initTestSuite;
end
function testCase
assertEqual( 'C:\', 'D:\', 'message.' );
end

Rather than trying to escape the escape-characters, I would suggest modifying the calls to the "MException" constructor within assert*.m, so that the messages get passed as a string parameter rather than as the format string itself. For example, the following:

MException('assertEqual:nonEqual', message)

would become:

MException('assertEqual:nonEqual', '%s', message)

I believe that these modifications do the trick.

27 Apr 2010 Steve Eddins

David, In principle one could do this by subclassing TestRunMonitor (http://www.mathworks.com/matlabcentral/fx_files/22846/7/content/matlab_xunit/architecture/html/matlab_xunit_architecture.html#6) with something that spits out XML. I skimmed the information available at the link you provided, and it looks to me like the XML output described there is quite extensive and includes a number of elements that don't apply to MATLAB xUnit. Do you have more specific needs or requirements in mind? Drop me a note if so. My contact information is on my MATLAB Central author page.

27 Apr 2010 David

Would it be possible to generate JUnit-style XML Output?

see: http://www.junit.org/node/399

25 Apr 2010 Steve Eddins

Nick, thanks for the suggestion. I'll investigate.

24 Apr 2010 Nick Henderson

Steve, I would like to suggest a change in the default floor_tol for relative tolerances. The issue is when testing values that are near zero:

>> assertElementsAlmostEqual(1e-10,1e-9)
??? Input elements are not all equal within relative tolerance: 1.49012e-08
First input:
1.0000e-10
Second input:
1.0000e-09

At first glance the error message is confusing. Of course this works if the 'absolute' option is used or floor_tol is set to sqrt(eps).

I recommend setting floor_tol to sqrt(eps). This would give more consistent default behavior.

Cheers,
Nick

23 Apr 2010 Steve Eddins

Mike - no, that doesn't work right now. I've been asked about it a couple of times, and I intend to do it, but I just haven't had a chance yet.

23 Apr 2010 Mike

Can tests work inside packages? I could not get this to work. In particular, any time I try to put a test in a package directory, runtests.m cannot detect it even when I first import the package.

I would like to organize a test directory that mirrors my actual directory structure but only containing tests. So if I have written a package in directory +somePackage, then inside my test directory, I would like to have a directory +somePackageTest, which contains my tests. However, I cannot get runtests to detect my tests. This issue isn't major because I can change +somePackageTest to somePackageTest (i.e., no longer a package of tests) and have everything work, but I would prefer to organize with packages.

02 Feb 2010 Steve Eddins

Arthur,

Thanks for the report. I'll investigate and prepare an update.

02 Feb 2010 Arthur

I love this library; this makes testing in MATLAB nearly painless.

I've found one small bug in how error messages are handled in version 2.0.1 -- if your data contains characters which are meaningful to sprintf (backslashes in a Windows path, for example), the message is truncated and you get warnings about invalid escape sequences. Adding the following line to the end of comparisonMessage.m will insert the necessary escapes in the error message:

msg = strrep(strrep(msg, '%', '%%'), '\', '\\');

22 Sep 2009 Jader

I found the source of my problem: My test files names did not begun with "Test"

21 Sep 2009 Jader

I couldn't make it work on Matlab R2008b 7.7 and Windows 7
It doesn't finds the m-files with test functions in the current pwd

>> runtests
Starting test run with 0 test cases.

PASSED in 0.000 seconds.

04 Aug 2009 Steve Eddins

Nick, thanks for the error report about assertElementsAlmostEqual and assertVectorsAlmostEqual. I uploaded a new version (2.0.1) today that fixes the problem.

04 Aug 2009 Steve Eddins

Nick, the behavior with your setupdata function is a feature. See the doc page called "How to Write Tests That Share Common Set-Up Code."

http://www.mathworks.com/matlabcentral/fx_files/22846/6/content/matlab_xunit/doc/html/exTestFixtures.html

03 Aug 2009 Nick

If you are putting multiple tests in a single file, don't start any function names with "setup". The following code fails. Remove setup and it works.

function test_suite = testFliplr
initTestSuite;
end

function testFliplrMatrix
in = magic(3);
assertEqual(fliplr(in), in(:, [3 2 1]));
end

function testFliplrVector
assertEqual(fliplr([1 4 10]), [10 4 1]);
end

function test3
[a b] = setupdata;
assertEqual(a,1);
assertEqual(b,2);
end

function [a b] = setupdata
a = 1;
b = 2;
end

22 Jul 2009 Nick

I think there is a divide by inf in computing the relative tolerance. Using the 'absolute' flag provides the correct result.

22 Jul 2009 Nick

Great software! Thanks. I noticed a bug:

function testStuff
assertElementsAlmostEqual(0,Inf)
assertVectorsAlmostEqual(0,Inf)
end

Will not flag a failure.

--N

08 Jun 2009 Frank

Steve, thanks for making this available. I had used xtargets/munit in the past, and while quite powerful, the syntax was different from other *Unit frameworks, which I felt took away from it a little. I've used *Unit in Python, Perl, Java, and C - I'm used to the way they work, I didn't really want to learn a "better" way to do it, much less teach my colleagues to use it. I'm glad to have the familiar framework available now in MATLAB.

As you mentioned in your 2006 talk (I just read the slides), most of us are not primarily programmers - we're scientists, engineers, etc. However, we use languages like Python and MATLAB to express our scientific ideas. I've long been a believer in using software engineering tools to make my *science* better, easier, more reliable and more reproduceable, and I agree that unit-testing is one of the keys to that. Still (as you also point out), it's not widely appreciated in the scientific community, and no scientist ever formally learns about these tools.

Having a simple interface to unit-testing in MATLAB is a big win, in my book. It should make convincing my colleagues to use it a whole easier. Thanks

08 Jun 2009 Steve Eddins

Ryan, I decided not to do the sprintf-style conversion specifiers in the assert functions, as least for now. My experience designing syntaxes for MATLAB and toolbox functions makes me shy away from a syntax design with an indefinite set of trailing arguments, as in sprintf. Such a syntax design makes it difficult to expand the syntax to support other options in the future. For now, you can just inline a call to sprintf directly in the call to the assertion functions.

12 May 2009 Ming Li

Thank you for putting this together and provide support for the framework. Great work.

06 May 2009 Steve Eddins

Ryan and Guido, the Help Browser team has identified the problem and is trying to fix the problem for the next MATLAB release. In the meantime, you can try this workaround: Put the MTEST directory after the MATLAB directories in your path, instead of before them.

Ryan, thanks for the enhancement suggestion. I have received several good suggestions, and I plan to work on an MTEST update soon.

05 May 2009 Ryan Ollos

I'm also having the issue with the doc browser freezing, but I find it doesn't occur if I type 'doc mtest' from the command line to access the help, rather than trying to expand the entry in the help browser.

>> version
ans =
7.8.0.347 (R2009a)

One small feature request would be to allow messages with sprintf style conversion specifiers.

28 Apr 2009 Ryan Ollos

This is very nicely written, and useful. For those MATLAB coders that desire to learn better practices.

23 Apr 2009 Steve Eddins

Hi again Guido,

The Help Browser team was able to reproduce the problem you reported, and they are investigating it. I don't know yet if there's a work-around I could apply to the MTEST doc files.

Steve

20 Apr 2009 Steve Eddins

Hi Guido,

I have not been able to reproduce the problem using R2009a on my 32-bit WinXP SP3 machine.

Would you please double-check your MATLAB version? I believe the released version was 7.8.0.347, not 7.8.0.324 as you reported.

Thanks,

Steve

20 Apr 2009 gg

Steve, thank you for MTEST available I find it very useful.

Nevertheless I encountered a problem involving the mtest help-file. Choosing "MTEST Unit

Test Framework" in the help window causes Matlab to freeze. No input (mouse and keyboard) is

accepted and the application needs to be terminated by the task-manager. This problem can be

replicated by different users on different machines.

Details to replicate the problem:

- Copy Mtest to <mtest-dir>
- "Add Path with Subfolders" <mtest-dir>
- Open help in Matlab
- click on "MTEST Unit Test Framework"
=> Freeze

All other mtest help entries work as expected.

Matlab version: R2009a, 7.8.0.324 (win32)
OS: Windows XP Professional, 5.1.2600 SP3

It is really more a nuisance than a no-go but maybe you can track down the problem.

Thank you & kind regards
Guido

16 Apr 2009 Shi vera

That's exactly what i need! Thank you very much!

12 Mar 2009 Rick Bosi

What can I say... Thank you for building and providing this. This should be part of the base MATLAB package!

11 Mar 2009 Ryan Ollos  
Updates
11 Mar 2009

Version 1.1

This release adds new assertion functions (assertElementsAlmostEqual and assertVectorsAlmostEqual) for performing floating-point comparisons.

16 Mar 2009

Corrected a problem with the no-input-argument call to mtest. It was incorrectly picking up M-files not matching the test naming convention as test files. Also integrated the documentation with the Help Browser.

02 Apr 2009

Version 1.1.2. This version corrects a problem with assertVectorsAlmostEqual when the caller provides a custom message.

20 May 2009

Version 1.1.3 - Remove LICENSE.txt file because the open source BSD license is now supplied automatically for new submissions to the MATLAB Central File Exchange. This version has no functional changes.

05 Jun 2009

Version 2.0
* Value info in failure messages
* assertFilesEqual
* Run tests in multiple folders
* Renamed test runner
* assertAlmostEqual deprecated
To use the old names or assertAlmostEqual, put the “obsolete” folder on the path.

04 Aug 2009

Version 2.0.1
Corrected errors in assertElementsAlmostEqual and assertVectorsAlmostEqual related to NaN and Inf inputs.

13 Jun 2010

Version 3.0
Run tests in packages; -verbose option for runtests; option to name test functions with "test" at end; loosen default floor tolerance; fix handling of message strings in assert functions

16 Jun 2010

Version 3.0.1
Fixed handling of TestCase subclasses inside test packages.

29 Jul 2010

Version 3.0.2
Fixed bug that caused TestCase subclass tests in packages to get run twice in some versions of MATLAB. Minor doc updates.

19 Nov 2010

Version 3.1
* runtests: Add -logfile option; accept cell array of test names
* Fix bug regarding subfunction tests in packages
* Allow test package to contain tests in subpackages

12 Jul 2013

Version 3.1.1
* Add links to the documentation for the new unit test framework in MATLAB
* Subfunction test files no longer execute the tests automatically when called with no output argument.

12 Jul 2013

Version 3.1.1
* Add links to the documentation for the new unit test framework in MATLAB
* Subfunction test files no longer execute the tests automatically when called with no output argument.

Contact us