[{"content":" Originally published on get-testy.com in 2018.\nOne of the bigger discussions I see occur frequently is the discussion about how to deal with and address Flaky Tests.\nJust go back to Google and search for \u0026ldquo;How to deal with flaky tests\u0026rdquo;. You\u0026rsquo;ll get over 10 million hits with a front page littered with blog posts about how to handle flaky tests. There will even be a hit from the Google Testing Blog, where they offer up a definition of a \u0026ldquo;flaky\u0026rdquo; test.\nWe define a \u0026ldquo;flaky\u0026rdquo; test result as a test that exhibits both a passing and a failing result with the same code.\nThe problem that I have with this blog post, and the numerous others that cover this topic, is that they classify an Error and Failure as the same test result.\nIn a previous post of mine, Top 5 pitfalls writing integration tests, I went over the difference between an Error and a Failure.\nError: An unexpected error has occurred during the execution of the test. Failure: An assertion in a test has failed.\nIf there is a Failure during a test execution, it should mean that there is a defect in the production code of your unit under test. If the test failed but there isn\u0026rsquo;t a defect, then you need to rethink your test design.\nIf there is an Error during a test execution, it should mean that there is a defect in the APIs that you are using to setup your preconditions.\nIf you are running into the same issues in which I am, the reason that you are seeing flakiness in your test results is because your tests are not flaky, it\u0026rsquo;s your libraries that your tests are using that cause intermittent errors.\nI want to amend the flaky test definition:\nA flaky test result exhibits both a passing and an error result with the same code.\nI think that we can all be more diligent in the language we use when we describe flaky failures and errors in test code.\nI\u0026rsquo;d love to hear your thoughts on Flaky tests. Should we treat Errors and Failures the same?\n","permalink":"https://jamespenning.dev/flaky-test-definition/","summary":"Why an error and a failure are different test results, and why that matters when you call a test flaky.","title":"What actually makes a test flaky?"},{"content":" Originally published on get-testy.com in 2018.\nUpdated Oct 2026: added a note on Testcontainers and CI.\nThe Pragmatic Programmer states Integration Testing shows the major subsystems that make up the project work and play well with each other.\nUnit tests isolate individual components and mock out the behavior of the external system dependencies while Integration tests flow through your code with the actual systems in place and with real data.\nIntegration Testing is often one of the toughest parts of the software development life cycle. Depending on the complexity of your application, your integration tests can be quite a burden to get up and running. Many software engineers write Integration tests, but they tend to struggle along the way, myself included.\nHere is my list of the top things that I have struggled with when writing my Integration tests. It certainly isn\u0026rsquo;t a complete list, but it\u0026rsquo;s a place to start.\n1. Writing too many tests The number one thing that I see happen is that people are writing too many Integration tests. When you find yourself with hundreds or even thousands of tests, you are likely writing tests that could be pulled down to the unit level.\nWhen you are developing software, faster feedback on the quality of changes is a big deal. If you are stuck waiting for hundreds or thousands of integration tests to run, you are going to kill developer productivity.\nOne of the benefits of moving them to Unit tests is the faster execution time. Comparatively, a thousand Unit tests are going to run significantly faster than a thousand Integration tests.\nAnother thing to consider is the cost to maintain Integration tests. Not only are they costly to create, they are arguably more costly to maintain them. Making sure they are up-to-date and kept in a runnable state takes a lot of time, and that is time that you aren\u0026rsquo;t devoted to writing new features.\nThis is also a good time to mention the testing pyramid. If you aren\u0026rsquo;t familiar with the concept, I came across an article that defines it fairly well. Basically, you want to have a strong foundation of Unit tests, with fewer Integration tests, and even fewer Acceptance tests.\n2. Lack of error and exception handling It took me far to long to figure out what the difference between and Error and a Failure was in a test execution.\nError: An unexpected error has occurred during the execution of the test. Failure: An assertion in a test has failed.\nPerfect tests will never error and will only ever fail when there is a defect in the software. Since you never write perfect tests, you need to make sure that you have proper error handling. Your future self will thank you when trying to figure out why your tests have errored on the CI server!\nErrors in Integration tests typically happen when configuration values haven\u0026rsquo;t been passed in properly. If you don\u0026rsquo;t have any error handling, at least put some around any place you make connections to external services.\nAn exception that says that a SqlTimeout has occurred is going to be harder to track down than an exception saying that the connection string was not passed into the test.\n3. Not automating the setup and preconditions Have you ever gotten all your tests passing locally only just to have your teammates attempt to run them where they fail immediately?\n\u0026ldquo;But they worked on my machine!\u0026rdquo;\nThis is something I hear all too often (and sometimes catch myself saying).\nMake sure that you document all the preconditions! If you downloaded any packages, altered your environment variables, performed any manual steps, or anything else to get preconditions in place.\nOnce it is documented, go all out and automate those steps! This way you can run your tests on any machine, especially if you want to get these running on a CI server.\n4. Leaving Artifacts laying around Have you heard of the Boy Scout rule, \u0026ldquo;Always leave the campground cleaner than you found it.\u0026rdquo;? This is kind of like that, except it\u0026rsquo;s an application instead of a campground! So instead: \u0026ldquo;Leave the application cleaner than when you started testing it.\u0026rdquo;\nLet’s look at Facebook for this pitfall. (In this example, I am classifying their Groups, Pages, Posts, Users, etc as physical artifacts.)\nSay you are writing tests to verify that you can create a Page artifact. If this runs against a dev server, and each Page creation creates a new database under the covers (I know it probably doesn\u0026rsquo;t, but let\u0026rsquo;s pretend!). You could take down the dev server just by filling up the HDD if you don\u0026rsquo;t delete those Pages after they\u0026rsquo;ve been created.\nMost frameworks have Teardown events that you can hook into. I suggest that in your test fixtures, you write a service that tracks all of the artifacts that you create during the test execution. That service then can handle the deletion of those artifacts in the Teardown.\n5. Using physical artifacts inside of a TestCaseSource If you read my other post about the TestCaseSource attribute, you know that I use it as often as I can. This has gotten me in trouble when using real data though.\nIf the TestCaseData that you are building up is performing these Read and Write operations on your artifacts, you are going to be affecting every suite that is in your test project. This is due to how testing frameworks execute tests. They build up a map of all the tests that are in project before they get executed, even for those that may not be selected.\nLet’s stick to the Facebook example. Pretend that you have a fixture that is testing Pages. One of the inputs you are testing is the different types of Posts that you can make on a Page. If you are building up those Posts inside of the TestCaseSource by calling out to the Posts service, the testing framework is going to actually call out to that service and create them while it\u0026rsquo;s building up that map.\nSo if it takes 1 minute to create those posts, you are adding 1 minute of overhead for every other fixture in the project. Even if you have a test in a separate fixture that has no preconditions and runs in under a second, it’s still going to take that test over a minute to execute.\nThis issue was plaguing me for the longest time. When I finally found why all my tests were loading so slow, I thought it was a really interesting problem. I will likely create another post devoted entirely to this pitfall because I want to go more in-depth on the technical aspects of how tests are loaded into memory and executed.\nA note from 2026 Two of these pitfalls, not automating the setup (3) and leaving artifacts around (4), are a lot easier to avoid now than they were in 2018. Testcontainers for .NET starts the real dependency your test needs, like a PostgreSQL database, in a throwaway Docker container from your test code:\n// dotnet add package Testcontainers.PostgreSql var postgres = new PostgreSqlBuilder(\u0026#34;postgres:15.1\u0026#34;).Build(); await postgres.StartAsync(); string connectionString = postgres.GetConnectionString(); The setup lives in the test project, so it runs the same way on a teammate\u0026rsquo;s machine and on the CI server. When the run ends, pass or fail, its Resource Reaper removes the containers it started. That\u0026rsquo;s the cleanup I used to write by hand in every TearDown.\nIn CI, the only requirement is Docker. GitHub-hosted runners already have it. One catch: Windows runners use the Windows Docker engine and can\u0026rsquo;t run Linux containers, so run these tests on a Linux runner.\nThe other three pitfalls are still on you. Containers make integration tests easier to write, which makes it easier to write too many of them.\nAt the end of the day, Integration tests should be there to help you. For a long time, my Integration tests were hurting more than helping. Hopefully now when you are writing your Integration tests, you can avoid these pitfalls.\nThis was just my list of top 5 pitfalls that I have run into. I am curious to hear what pitfalls you have run into when writing Integration tests. Leave a note so we can discuss!\nIf you want to hear more about this topic, take a listen to the TestTalks podcast episode 205 where I talk to Joe Colantonio about this topic.\nhttps://joecolantonio.com/testtalks/205-net-integration-testing-james-penning/\n","permalink":"https://jamespenning.dev/top-5-pitfalls-writing-integration-tests/","summary":"Five mistakes I kept making with integration tests, from writing too many to leaving artifacts behind.","title":"Top 5 pitfalls writing integration tests"},{"content":"Right in the middle of relaunching this blog, every pull request started failing in about two seconds. No logs. No error message worth reading. The jobs never got a machine.\nThe reason was boring: I\u0026rsquo;d hit my GitHub Actions usage limit. I was working through lots of small issues across my personal projects, each one its own pull request, and every pull request ran the full CI pipeline. I wasn\u0026rsquo;t thinking about CI minutes at all, because I\u0026rsquo;ve never had to do that in the past.\nThat chart is the whole story. Flat for two weeks, then a cliff once the small-issue workflow kicked in.\nI build CI/CD services for a living, so the fix was obvious. I already had a Windows PC that runs 24/7 for my AI model experiments, and most of the time its compute was sitting there gathering dust. So I pointed the pipeline at it. Every build, link check, Lighthouse run and deploy for this site now happens on that machine.\nMoving a pipeline from Linux runners to a Windows box is the fastest way to find every assumption your workflows made without telling you. Here\u0026rsquo;s what that looked like.\nThe setup A self-hosted runner is a small agent you install on your own machine. It connects out to GitHub, waits for jobs, and runs them locally. Nothing has to be opened up on your router, because the runner calls GitHub, not the other way around.\nThe install is short:\nIn the repo, go to Settings → Actions → Runners → New self-hosted runner and pick Windows. Download and unzip the runner into its own folder, then run config.cmd with the URL and the one-time token GitHub shows you. Say yes to running it as a Windows service, so it starts with the PC and doesn\u0026rsquo;t need a terminal open. Add a custom label. Mine is blog. Then every workflow job asks for that machine by label:\nruns-on: [self-hosted, Windows, blog] Honestly, this part was easy. No snags at all.\nThe only thing to know: on a personal account, a runner belongs to one repository. Organizations can share a runner across repos, but personal accounts can\u0026rsquo;t. This PC now runs separate runners for three personal projects, this blog included. Each one is its own install, in its own folder, as its own service, and labels keep each repo\u0026rsquo;s jobs on the right runner.\nWhat broke 1. Bash wasn\u0026rsquo;t there My workflow steps use shell: bash. On GitHub\u0026rsquo;s hosted Windows runners that just works, because Git Bash is already on the PATH. My runner runs as a Windows service, and the service\u0026rsquo;s PATH only had Git\u0026rsquo;s cmd folder, not the folder where bash.exe lives. Every bash step failed.\nFix: a small composite action, setup-git-bash, that runs first in every job. It finds git.exe, looks for bash.exe next to it, and adds that folder to GITHUB_PATH for the rest of the job.\n2. Then bash broke in a weirder way That fix had a bug of its own (I\u0026rsquo;ve definitely never fixed a bug that introduced a new one\u0026hellip; not me!). To log which version it found, the action ran bash --version | Select-Object -First 1. PowerShell\u0026rsquo;s Select-Object -First 1 stops reading after the first line, which closes the pipe while bash is still writing. Bash notices and exits with an error, and the step failed. Sometimes. It depended on timing.\nAn intermittent failure in a step whose only job was to print a version number. If you\u0026rsquo;ve read my post on what actually makes a test flaky, you know how I feel about calling that \u0026ldquo;flaky.\u0026rdquo; It wasn\u0026rsquo;t random. It was a real bug that only showed up some of the time.\nFix: capture all of bash\u0026rsquo;s output first, then print the first line.\n3. An action that couldn\u0026rsquo;t even be downloaded I\u0026rsquo;d been using the popular Lighthouse CI action to check accessibility and SEO scores. On Windows it fails before it starts: its bundled node_modules has file names that differ only by upper and lower case. Linux treats those as two files. Windows treats them as one, so the checkout breaks.\nFix: skip the action and call the Lighthouse CI command-line tool with npx @lhci/cli, pinned to a version. Fewer moving parts anyway.\n4. A Linux-only link checker The link checker\u0026rsquo;s GitHub Action only ships Linux binaries. Fix: another small composite action that downloads the Windows build of lychee once, caches it in the runner\u0026rsquo;s tool cache, and puts it on the PATH. Hugo gets the same treatment, so after the first run neither one is downloaded again.\n5. Strangers\u0026rsquo; code on my machine This one made me stop and think. The repo is public. With a self-hosted runner, anyone who opens a pull request from a fork could, in principle, get their code run on my PC. GitHub\u0026rsquo;s own docs warn against self-hosted runners on public repositories for exactly this reason.\nFix: every CI job checks that the pull request comes from this repo, not a fork, and skips otherwise:\nif: github.event.pull_request.head.repo.full_name == github.repository For a one-person blog with that guard in place, I\u0026rsquo;m comfortable. For a project with outside contributors, I wouldn\u0026rsquo;t be. I also decided to make this repo private, as it was only needing to be public to use the GitHub Pages feature on a Free plan. That gets fixed if you bump up to the $4 pro plan.\nWhat it looks like now Every pull request runs markdown lint, a Hugo build, a check that every old get-testy URL still resolves, an internal link check, Lighthouse on three pages, and a check that no post goes live with an unfinished gap. Then it builds a preview with drafts that I can download and open. Merging to master builds again, deploys to GitHub Pages, and checks the old URLs one more time against the live site.\nAll of it runs on my own PC, as often as I want, without watching a usage meter.\nWas it worth it? For me, 100%. I get to run CI as much as I like, on compute I already own that was otherwise idle. The machine is on around the clock anyway, because I also run CLEF on it, Cloudflare\u0026rsquo;s open-weight decision models, to try out multimodal classification. A few CI runners on top cost me nothing extra.\nBut this is a personal-project answer. If I were working on something with other people, I wouldn\u0026rsquo;t put their CI on my PC. I\u0026rsquo;d go the other way: tune the pipeline to run on smaller hosted runners, and trim down what actually needs to run on every pull request. Not every typo fix needs three Lighthouse runs.\nThe bigger lesson is one I keep relearning in DevOps: a pipeline that works is not the same as a pipeline you understand. Moving where it runs is the fastest way to find out which one you have.\n","permalink":"https://jamespenning.dev/my-blogs-ci-runs-on-my-home-pc/","summary":"\u003cp\u003eRight in the middle of relaunching this blog, every pull request started failing in about two seconds. No logs. No error message worth reading. The jobs never got a machine.\u003c/p\u003e\n\u003cp\u003eThe reason was boring: I\u0026rsquo;d hit my GitHub Actions usage limit. I was working through lots of small issues across my personal projects, each one its own pull request, and every pull request ran the full CI pipeline. I wasn\u0026rsquo;t thinking about CI minutes at all, because I\u0026rsquo;ve never had to do that in the past.\u003c/p\u003e","title":"My blog's CI runs on my home PC now"},{"content":" Originally published on get-testy.com in 2018. Since NUnit 4 the classic asserts live in ClassicAssert, which makes the case for constraints even stronger.\nUp until recently, I have been stuck in the past with using version 2.6.4 of NUnit. I had never upgraded because I never knew of all the benefits that I could receive by using Version 3.\nOne of the biggest changes in going from version 2 to 3 is that Rob, Charlie, and the other guys developing NUnit are pushing people to use the Constraint Assertion Model.\nUp until now, I had only known about the Classic Assertion Model and looking back, I don\u0026rsquo;t know how I lasted as long as I did without switching! In this post, I want to show you why using constraints is so much better than using classic and how it will truly change your life.\nIf you are like me and you write your tests, add them to your CI pipeline, passing on green, think everything is good and move on to the next project. Well, what happens when they fail?\nSuddenly, a random error appears!\n\u0026ldquo;Expected True but was False.\u0026rdquo;\nHow many times have you seen this and got upset by the lack of messaging around your test failures? You wouldn\u0026rsquo;t let this in your production code. Why then let it happen in your tests?\nUsing the constraint model, you can write your assertions, using fluent language, and have better messaging around assertions that fail so you can know exactly what is failing and why.\nClassic Assertion Model You can read about the classic model in the NUnit docs, but these are probably the assertions that you are most familiar with and using in your tests right now.\nThe classic model uses individual methods for expressions on the assertion. Most commonly used methods would be Assert.AreEqual, Assert.True, and Assert.Throws.\nHere is an example of a test using the classic model. In this test, I have a list of integers and I want to verify that there is exactly one int in the list that is equal to 4.\n[Test] public void ClassicAssertionModel() { List\u0026lt;int\u0026gt; myList = new List\u0026lt;int\u0026gt; { 1, 2, 3 }; Assert.AreEqual(1, myList.Where(x =\u0026gt; x == 4).Count()); } This particular test would fail due to the absence of 4. The output of the test would give us no information about the failure unless we opened up the code and looked at the assertion that is failing.\n1) Failed : BlogSamples.Tests.ConstraintExample.ClassicAssertionModel Expected: 1 But was: 0 We can do better than this, so let\u0026rsquo;s check out constraints!\nConstraint Assertion Model With the constraint model, you move the expressions from the method to something you pass into the assertion. This makes it so that you aren\u0026rsquo;t just stuck with the builtin assertion methods and is what makes the constraint model become powerful. (In a later post, I will go into building out custom constraints, but for now I will keep it simple.)\nWith constraint assertions, you use one assertion method Assert.That.\nSo if we take our test from before and refactor it to use a constraint, it would look like this.\n[Test] public void ConstraintAssertionModel() { List\u0026lt;int\u0026gt; myList = new List\u0026lt;int\u0026gt; { 1, 2, 3 }; Assert.That(myList, Has.Exactly(1).EqualTo(4)); } The assertion now reads fluently and doesn\u0026rsquo;t require a description of what it is checking. \u0026ldquo;Verify that my list has exactly one int equal to 4.\u0026rdquo;\nAlready, our test reads much better with constraints than it did with using classic, but what does our output look like?\n2) Failed : BlogSamples.Tests.ConstraintExample.ConstraintAssertionModel Expected: exactly one item equal to 4 But was: \u0026lt; 1, 2, 3 \u0026gt; Boom! It tells you exactly what the assertion is looking for and what the value is that you are checking.\nAlready by using the constraint model, our tests are much better off. If you got this error on a build server, you would know exactly what is happening. You haven\u0026rsquo;t even started using this model and you can already feel your life changing, right??\nMoving forward in all my tests, I will be only using the constraint model and not even think twice about going back. What is your excuse for still using the classic model?\nLet me know your opinions on what model you think is better!\n","permalink":"https://jamespenning.dev/nunit-classic-vs-constraint-assertion-models/","summary":"Why NUnit\u0026rsquo;s constraint assertion model gives you better failure messages than the classic model.","title":"NUnit's classic vs. constraint assertion models"},{"content":" Originally published on get-testy.com in 2018. Written against NUnit 3.\nUpdated Oct 2026: fixed a code sample.\nI have recently developed this pattern when I am writing tests that may not be new to some of you, but I wanted to share. It is using the TestCaseData class in combination with the TestCaseSource attribute. I have found that, in using the TestCaseSource for my tests, my code stays much cleaner, becomes easier to read, and allows it to be easily refactored in the future.\nThe TestCaseData class gives you a way to create a test case and set the information for that test case outside the test method. On TestCaseData, you can set the Test Name, Description, Category, and other information you want for the given test case. This allows you to document your test cases, and the code that is testing, so well that any developer can look at them and know what to expect from your code. You will clearly see with a given input to a method, what the expected output will be.\nTestCaseSource is an attribute that you put on your test method to make it parameterized. If you are familiar with the Arrange, Act, and Assert pattern (Read More Here), this is a way that you can move part of your Arrange step out of your test. This allows you to reduce a lot of test code and put the focus of your testing on the behavior of your methods under test instead of all the boiler plate setup required. In using the TestCaseSource, you can collapse multiple test methods that are testing the same method under test into one test method.\nLet\u0026rsquo;s see how by digging into an example!\nI am going to go back to our HelloWorld class, from previous posts, and make a new method. Let\u0026rsquo;s create a method, GetGreetingByDay, that will take in a string and DayOfWeek and modify it based on those input values.\npublic static string GetGreetingByDay(DayOfWeek day, string name) { string retVal = string.Empty; switch (day) { case DayOfWeek.Monday: retVal = $\u0026#34;{day.ToString()} is hard, make it easier with another cup of coffee, {name}!\u0026#34;; break; case DayOfWeek.Friday: retVal = $\u0026#34;It\u0026#39;s {day.ToString()}, {day.ToString()}, gotta get down on {day.ToString()}, {name}!\u0026#34;; break; default: retVal = $\u0026#34;Keep on truckin\u0026#39;, {name}!\u0026#34;; break; } return retVal; } Now with this as our Method Under Test, we can construct the TestCaseData for our test cases.\nWe will just cover the most basic scenarios:\nDayOfWeek.Monday and valid string DayOfWeek.Friday and valid string Valid DayOfWeek and valid string Invalid DayOfWeek and valid string null and null With our scenarios covered, we can start to construct our TestCaseData.\nDayOfWeek.Monday and valid string scenario would translate to this TestCaseData:\nnew TestCaseData(DayOfWeek.Monday, \u0026#34;James\u0026#34;) .Returns(\u0026#34;Monday is hard, make it easier with another cup of coffee, James!\u0026#34;) When you construct the TestCaseData, you give the values that you will be passing into the test method. In this scenario, that would be DayOfWeek.Monday and James. With that given input, we can specify what the Return value for these given inputs would be. Based off the GetGreetingByDay method, we would expect the string returned to be Monday is hard, make it easier with another cup of coffee, James!.\nNow let\u0026rsquo;s construct the rest of our TestCaseData objects and add it to a static list which will be given to the TestCaseSource attribute on our test method.\nThis could look like:\nprivate static List\u0026lt;TestCaseData\u0026gt; _getGreetingByDayData = new List\u0026lt;TestCaseData\u0026gt;() { new TestCaseData(DayOfWeek.Monday, \u0026#34;James\u0026#34;).Returns(\u0026#34;Monday is hard, make it easier with another cup of coffee, James!\u0026#34;), new TestCaseData(DayOfWeek.Friday, \u0026#34;James\u0026#34;).Returns(\u0026#34;It\u0026#39;s Friday, Friday, gotta get down on Friday, James!\u0026#34;), new TestCaseData(DayOfWeek.Sunday, \u0026#34;James\u0026#34;).Returns(\u0026#34;Keep on truckin\u0026#39;, James!\u0026#34;), new TestCaseData((DayOfWeek)8, \u0026#34;James\u0026#34;).Returns(\u0026#34;Keep on truckin\u0026#39;, James!\u0026#34;), // 8 is not a defined DayOfWeek, on purpose new TestCaseData(null, null).Returns(\u0026#34;Keep on truckin\u0026#39;, !\u0026#34;) }; Now that we have our TestCaseData set up, we need to set it as the TestCaseSource of a test method. By sending this into a test method as the TestCaseSource, we will be creating five test cases.\nThis is what our test method could look like:\n[TestCaseSource(nameof(_getGreetingByDayData))] public string GetGreetingByDay(DayOfWeek day, string name) { string greeting = BlogSamples.HelloWorld.GetGreetingByDay(day, name); return greeting; } Now we can run our tests and verify our GetGreetingByDay method was implemented properly based off the criteria we set.\nThe output from the VS Adapter looks like this:\nIf we wanted more descript names, we could call the SetName method on each instance of TestCaseData we created to set the Name for the test. By default, this is what the names will look like if you don\u0026rsquo;t explicitly set the name.\nnew TestCaseData(DayOfWeek.Monday, \u0026#34;James\u0026#34;) .Returns(\u0026#34;Monday is hard, make it easier with another cup of coffee, James!\u0026#34;) .SetName(\u0026#34;Monday with valid name\u0026#34;) If we wanted to add a description, we could call the SetDescription method the same way we did with the SetName. This is where you can start to document each test case to pin down the expected behavior of your methods. Your future self, or future developers needed to work with your code will thank you!\nnew TestCaseData(DayOfWeek.Monday, \u0026#34;James\u0026#34;) .Returns(\u0026#34;Monday is hard, make it easier with another cup of coffee, James!\u0026#34;) .SetName(\u0026#34;Monday with valid name\u0026#34;) .SetDescription(\u0026#34;When you send in Monday along with any string, you will receive a string formatted with Monday and the given string.\u0026#34;) The best part about using this pattern is how it can help you easily fix defects. Say you have a defect where performing some action errors out and in the stack trace you see your method throwing an unhandled exception. In a mature application that has logging, you can get the values that were passed in that caused it to blew up and create a new TestCaseData with those given inputs.\nWhen you run that test, you can expect it to break. From there, you can make your changes and run your tests to verify your fix. Having the existing suite to help you handle any regressions, it makes fixing your code much easier while giving you that piece of mind that you aren\u0026rsquo;t introducing any other defects.\nIt\u0026rsquo;s worth noting that this isn\u0026rsquo;t something that you always want to do. In the cases when you can, I suggest trying this pattern out. See if it organizes your test code better and gives you the ability to document your code without having to use a BDD framework like Specflow.\nHow do you guys organize your tests and what are you preferred patterns when writing them? Let me know!\n","permalink":"https://jamespenning.dev/testcasedata-with-testcasesource-to-better-organize-your-tests/","summary":"Pair NUnit\u0026rsquo;s TestCaseData with TestCaseSource to move setup out of your tests and document each case.","title":"TestCaseData with TestCaseSource to better organize your tests"},{"content":" Originally published on get-testy.com in 2017. Written against NUnit 3 and Visual Studio 2017.\nUpdated Oct 2026: fixed a code sample.\nAt my current job, one big topic of conversation lately has been how we are passing in our configuration values to our Integration tests. I want to show you how you can use the *.runsettings file to inject in configuration values to our NUnit tests, make your tests more configurable, and make handling configuration values in your tests easier to manage.\nOur current solution is not easy to manage and not scalable with the numerous new developers coming onboard and increasing number of tests that we are writing. We have a static configuration class that uses the System.Configuration library to access app.settings section inside the app.config for our test assembly. The idea of this is great, you have central authority for getting configuration values that your tests need. The implementation and execution of this idea is what is holding us back.\nIn our implementation, we use a python script on our build server in our CI pipeline, that takes in the parameters, such as: The name of dynamically spun up Server Under Test, the WebAPIUrl of our application, the name of the Database, etc. The script then dynamically sets these values by building up the *.dll.config file for the test assembly on the fly based off a template. In order to add a new value to our tests, that are a lot of pieces and components that have to be updated. This makes this solution not scalable and causes numerous headaches when things like binding redirects or other settings in your app.configs for specific Integration tests come up.\nHow can we change our implementation to make it more configurable and easier to manage? That is where the *.runsettings comes into play. In my previous post about the TestContext, I had briefly mentioned the Parameters member. In this post, I want to dig in a little deeper into this class and how you can best use it in your tests.\nFirst thing we want to do is create our local.runsettings for our local development. All we need to do is add a new file to our project, call it local.runsettings, and add our first parameter. It will end up looking like this:\n\u0026lt;RunSettings\u0026gt; \u0026lt;TestRunParameters\u0026gt; \u0026lt;Parameter name=\u0026#34;WebAppUrl\u0026#34; value=\u0026#34;http://localhost\u0026#34; /\u0026gt; \u0026lt;/TestRunParameters\u0026gt; \u0026lt;/RunSettings\u0026gt; Now we can add a test that uses this variable. We can make a simple test that just asserts that the value we pull from the Parameters is equal to the value we expect it to be.\n[TestFixture] public class Runsettings { [Test] public void VerifyWebAppUrlParameter() { string webAppUrl = TestContext.Parameters[\u0026#34;WebAppUrl\u0026#34;]; Assert.That(webAppUrl.Equals(\u0026#34;http://localhost\u0026#34;)); } } Now we can run this test to verify that we are pulling back the value from our local.runsettings in our test. Some of you might be ahead of me and saying that this won\u0026rsquo;t pass, and you are right! We still need to do one of two things, depending on where we are running our test, before we can get the value from the local.runsettings and get this test to pass.\nIf we are running our test in Visual Studio, we need to specify a settings file that the NUnit3 adapter will use when running tests.\nOn your toolbar, go to Test -\u0026gt; Test Settings -\u0026gt; Select Test Settings File. It will open a file browser, where you can then choose the local.runsettings file that we created earlier.\nIf we are running our test using the NUnit3-Console, we need to pass in the parameter and value in as an argument in the command line.\nWithout sending in the WebAppUrl value into our command\nCommand: D:\\SourceCode\\nunit-console\\bin\\Release\\nunit3-console.exe .\\bin\\Debug\\BlogSamples.NUnit.dll --test=BlogSamples.NUnit.Runsettings\nOutput:\nD:\\SourceCode\\BlogSamples\\BlogSamples.NUnit\u0026gt; D:\\SourceCode\\nunit-console\\bin\\Release\\nunit3-console.exe .\\bin\\Debug\\BlogSamples.NUnit.dll --test=BlogSamples.NUnit.Runsettings NUnit Console Runner 3.7.0 Copyright (C) 2017 Charlie Poole Runtime Environment OS Version: Microsoft Windows NT 10.0.14393.0 CLR Version: 4.0.30319.42000 Test Files .\\bin\\Debug\\BlogSamples.NUnit.dll Test Filters Test: BlogSamples.NUnit.Runsettings Errors, Failures and Warnings 1) Error : BlogSamples.NUnit.Runsettings.VerifyWebAppUrlParameter System.NullReferenceException : Object reference not set to an instance of an object. at BlogSamples.NUnit.Runsettings.VerifyWebAppUrlParameter() in D:\\SourceCode\\BlogSamples\\BlogSamples.NUnit\\Runsettings.cs:line 29 Run Settings DisposeRunners: True WorkDirectory: D:\\SourceCode\\BlogSamples\\BlogSamples.NUnit ImageRuntimeVersion: 4.0.30319 ImageTargetFrameworkName: .NETFramework,Version=v4.6.2 ImageRequiresX86: False ImageRequiresDefaultAppDomainAssemblyResolver: False NumberOfTestWorkers: 12 Test Run Summary Overall result: Failed Test Count: 1, Passed: 0, Failed: 1, Warnings: 0, Inconclusive: 0, Skipped: 0 Failed Tests - Failures: 0, Errors: 1, Invalid: 0 Start time: 2017-12-07 17:55:14Z End time: 2017-12-07 17:55:14Z Duration: 0.634 seconds With sending in WebAppUrl into our command\nCommand: D:\\SourceCode\\nunit-console\\bin\\Release\\nunit3-console.exe .\\bin\\Debug\\BlogSamples.NUnit.dll --test=BlogSamples.NUnit.Runsettings --params WebAppUrl=http://localhost\nOutput:\nD:\\SourceCode\\BlogSamples\\BlogSamples.NUnit\u0026gt; D:\\SourceCode\\nunit-console\\bin\\Release\\nunit3-console.exe .\\bin\\Debug\\BlogSamples.NUnit.dll --test=BlogSamples.NUnit.Runsettings --params WebAppUrl=http://localhost NUnit Console Runner 3.7.0 Copyright (C) 2017 Charlie Poole Runtime Environment OS Version: Microsoft Windows NT 10.0.14393.0 CLR Version: 4.0.30319.42000 Test Files .\\bin\\Debug\\BlogSamples.NUnit.dll Test Filters Test: BlogSamples.NUnit.Runsettings Run Settings DisposeRunners: True WorkDirectory: D:\\SourceCode\\BlogSamples\\BlogSamples.NUnit TestParametersDictionary: [WebAppUrl, http://localhost] TestParameters: WebAppUrl=http://localhost ImageRuntimeVersion: 4.0.30319 ImageTargetFrameworkName: .NETFramework,Version=v4.6.2 ImageRequiresX86: False ImageRequiresDefaultAppDomainAssemblyResolver: False NumberOfTestWorkers: 12 Test Run Summary Overall result: Passed Test Count: 1, Passed: 1, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0 Start time: 2017-12-07 17:55:18Z End time: 2017-12-07 17:55:19Z Duration: 0.622 seconds It\u0026rsquo;s important to note the behavior when you try accessing the Parameter member and asking for a key that isn\u0026rsquo;t in the *.runsettings file specified or passed into the commandline.\nIt will throw everyone\u0026rsquo;s favorite NullReferenceException: Object reference not set to an instance of an object. Which is not helpful to us when we are trying to figure out why our tests are failing.\nThis is where having that separate static configuration class comes in handy. Instead of accessing our parameters directly from the TestContext every time, we can use our configuration class which gives us the ability to check the existence of the parameters before we access them.\nThis might look something like this.\npublic class Configuration { private static string _webAppUrl; public static string WebAppUrl { get { if (_webAppUrl == null) { if (!TestContext.Parameters.Exists(\u0026#34;WebAppUrl\u0026#34;)) { throw new System.ArgumentException(\u0026#34;The parameter \u0026#39;WebAppUrl\u0026#39; was not found, please provide a value for this parameter.\u0026#34;); } _webAppUrl = TestContext.Parameters[\u0026#34;WebAppUrl\u0026#34;]; } return _webAppUrl; } } } If we move the accessor into our configuration class, we can then update our test to look like this instead.\n[Test] public void VerifyWebAppUrlParameter() { Assert.That(Configuration.WebAppUrl, Is.EqualTo(\u0026#34;http://localhost\u0026#34;)); } Now tooled with this implementation, we can:\nRemove the need for dynamically creating these *.dll.config files on the fly. Allow people to add new configuration values with more ease. Pull out our configuration values from our tests for more dynamic execution. For us, our solution becomes extremely simplified. We can still use our static configuration class, and how we access our configuration values does not change. The big change for us is completely removing the need for building up the *.dll.config and just pass those parameters into the NUnit3-Console when executing our tests making our process much simpler and easier to manage!\nI am curious to hear how you all pass in your configuration values to your tests. Maybe there is a better way yet, but for now, this is the best solution that I have found!\n","permalink":"https://jamespenning.dev/using-runsettings-to-configure-nunit-tests/","summary":"Use a .runsettings file and TestContext.Parameters to pass configuration into NUnit integration tests without config-file templating.","title":"Using .runsettings to configure your NUnit tests"},{"content":" Originally published on get-testy.com in 2017. Written against NUnit 3.\nUpdated Oct 2026: fixed a code sample.\nIn my previous post, Overview of the TestContext in NUnit. Part 1: Static properties and methods, I went over the static properties and methods that are available on the TestContext. I had briefly mentioned the CurrentContext and in this post we are going to dive deeper into this property.\nThe CurrentContext gets created separately for each test but is still accessible anywhere within the TestFixture. It holds information about the current test, fixture execution, and even provides you with a Randomizer class that you can use inside your test to build up random data.\nLet\u0026rsquo;s dig into the CurrentContext!\nCurrent Context - Properties Test - This gets you a representation of the current test. It\u0026rsquo;s properties are:\nID - The unique Id of the test Name - The name of the test, whether set by the user or generated automatically FullName - The fully qualified name of the test ClassName - The fully qualified name of the class MethodName - The name of the method representing the test, if any Properties - An IPropertyBag of the test properties If you recall the example we were using in our previous posts, we had this HelloWorld test.\nusing NUnit.Framework; namespace BlogSamples.NUnit { [TestFixture] public class HelloWorld { [Test] public void Hello() { string greeting = BlogSamples.HelloWorld.Hello(\u0026#34;get-testy\u0026#34;); Assert.That(greeting.Equals(\u0026#34;Hello, get-testy.\u0026#34;, StringComparison.Ordinal)); } } } If we were to access TestContext.CurrentContext.Test from inside the method Hello, we would see these values:\nID = \u0026quot;0-1001\u0026quot; Name = \u0026quot;Hello\u0026quot; FullName = \u0026quot;BlogSamples.NUnit.HelloWorld.Hello\u0026quot; ClassName = \u0026quot;BlogSamples.NUnit.HelloWorld\u0026quot; MethodName = \u0026quot;Hello\u0026quot; Properties = { Keys: { } } A few things to note about the values.\nThe IPropertyBag for Properties is empty. Since there are no extra attributes putting properties onto this test case, it is empty. The ID is not guaranteed to be the same each test run. It is just to uniquely identify that execution of the test. Now let\u0026rsquo;s take this same method, but change the Name of it and also add a property.\n[TestCase( TestName = \u0026#34;Hello, get-testy.\u0026#34;, Description = \u0026#34;Verify we can get a properly formed message back from HelloWorld.\u0026#34;, Author = \u0026#34;James Penning\u0026#34;)] public void Hello() { string greeting = BlogSamples.HelloWorld.Hello(\u0026#34;get-testy\u0026#34;); Assert.That(greeting.Equals(\u0026#34;Hello, get-testy.\u0026#34;, StringComparison.Ordinal)); } Now when we access TestContext.CurrentContext.Test, we would see these values:\nID = \u0026quot;0-1001\u0026quot; Name = \u0026quot;Hello, get-testy.\u0026quot; FullName = \u0026quot;BlogSamples.NUnit.HelloWorld.Hello, get-testy\u0026quot; ClassName = \u0026quot;BlogSamples.NUnit.HelloWorld\u0026quot; MethodName = \u0026quot;Hello\u0026quot; Properties = { Keys: { \u0026quot;Description\u0026quot;: \u0026quot;Verify we can get a properly formed message back from HelloWorld.\u0026quot;, \u0026quot;Author\u0026quot;:\u0026quot;James Penning\u0026quot; } } There are a lot of different attributes that you can add in order to fill in the properties. Description, Author, Category, etc.\nYou also may have noticed that I changed the attribute that declares Hello as a test case. Test and TestCase are subtly different. The basic difference is that you can have multiple TestCase attributes on a method, which will create multiple tests based off the values set in each attribute. Whereas with the Test attribute, you can only have one. Be on the look out to a post I will be making about all the different nunit attributes you can use!\nResult - Gets a representation of the test result. Outcome - A ResultState representing the outcome of the test. PassCount - The number of tests that passed. InconclusiveCount - The number of tests that were Inconclusive. SkipCount - The number of tests that were skipped. WarningCount - The number of tests that had warnings. FailCount - The number of tests that failed. Message - Any message that nunit defines based off test results. IE: \u0026ldquo;Message: One or more child tests were ignored\u0026rdquo; would be the message if there were tests that were ignored. StackTrace - Any exception was thrown inside of a test. The ResultState is accessible during the execution of a test, however, it is best to only use it in the teardown stage. It will have all the values populated at that time.\nResultState Status - A TestStatus with four possible values. Label - An optional string value, which can provide sub-categories for each Status. Site - A FailureSite value. These are the four possible values for Status. If you are familiar with the TestResult.xml output from a test execution, these are what get associated with the test-case nodes. I gave an example in my Part 1 post.\nTestStatus Inconclusive Skipped Passed Warning Failed These are the possible values for the FailureSite. This will indicate where the failure occurred in the execution of the test execution.\nFailureSite Test SetUp TearDown Parent Child One practical way you might use the ResultState would be to check if a test failed. If you have tests that you are running in serial that require the previous to pass, you would want to stop further execution to prevent cascading failures. Here\u0026rsquo;s how you can easily do this:\nprivate bool _stopFurtherExecution = false; [SetUp] public void SetUp() { if (_stopFurtherExecution) { Assert.Inconclusive(\u0026#34;Previous test failed\u0026#34;); } } [TearDown] public void TearDown() { TestStatus status = TestContext.CurrentContext.Result.Outcome.Status; // using NUnit.Framework.Interfaces; if (status == TestStatus.Failed) { _stopFurtherExecution = true; } } You could put this in a base class and then have any TestFixture that you want to follow this behavior and inherit from it.\nTestDirectory This will return the full file path of the directory in which the current executing assembly is located.\nWorkDirectory This will return the full file path of where the test is being executed.\nWe can access the TestDirectory and WorkDirectory like this.\nTestContext.WriteLine($\u0026#34;Current TestDirectory: {TestContext.CurrentContext.TestDirectory}\u0026#34;); TestContext.WriteLine($\u0026#34;Current WorkDirectory: {TestContext.CurrentContext.WorkDirectory}\u0026#34;); Which, when we execute the tests, would produce a result that looks like this.\n\u0026lt;!-- Command to execute tests --\u0026gt; D:\\SourceCode\\BlogSamples\u0026gt; D:\\SourceCode\\nunit-console\\bin\\Release\\nunit3-console.exe .\\BlogSamples.NUnit\\bin\\Debug\\BlogSamples.NUnit.dll NUnit Console Runner 3.7.0 Copyright (C) 2017 Charlie Poole Runtime Environment OS Version: Microsoft Windows NT 10.0.14393.0 CLR Version: 4.0.30319.42000 Test Files .\\BlogSamples.NUnit\\bin\\Debug\\BlogSamples.NUnit.dll \u0026lt;!-- Output from our tests --\u0026gt; =\u0026gt; BlogSamples.NUnit.HelloWorld.Hello, get-testy. Current TestDirectory: D:\\SourceCode\\BlogSamples\\BlogSamples.NUnit\\bin\\Debug Current WorkDirectory: D:\\SourceCode\\BlogSamples You can see that the TestDirectory is where the actual assembly that we are loading to execute tests and the WorkDirectory is where we are currently at in the powershell window.\nRandom This one is probably my favorite property to use. The Random property returns a Randomizer object which can aid you in generating random values to use in your tests. In using Random, if you don\u0026rsquo;t change the assembly or the seed used, you can repeat values on reruns. Which is really cool when you want to send in random data to methods but also be able to rerun tests with the same data if they fail.\nYou might be thinking, \u0026ldquo;Why would I want to use this instead of using System.Random directly?\u0026rdquo; Well, there are a couple reasons why, but the main two include:\nUniform methods. These methods extend the behavior on a wide range of types which System.Random does not include. IE. NextLong, NextLong(long max), and NextLong(long min, long max). Repeatable values. Like I said above, this is huge for being able to still use random values while also being able to reproduce that random value consistently. Let\u0026rsquo;s see how we can use Random with our HelloWorld tests. We might want to just see if our Hello method can take in a bunch of different strings, but not have to write out hundreds of tests with data.\n[Test] public void Random() { string randomGreeting = TestContext.CurrentContext.Random.GetString(); TestContext.WriteLine($\u0026#34;RandomGreeting: {randomGreeting}\u0026#34;); string greeting = BlogSamples.HelloWorld.Hello(randomGreeting); Assert.That(greeting.Equals($\u0026#34;Hello, {randomGreeting}.\u0026#34;, StringComparison.Ordinal)); } There are a lot of useful properties on the CurrentContext that you can use right away in your tests. Others might only have use when used inside of an Engine Extension, which I will cover in a future post.\nThis concludes the 2 part series on the TestContext. Be on the lookout for more posts on NUnit in the future.\nNow I want to know, how do you use the TestContext inside of your tests?\n","permalink":"https://jamespenning.dev/test-context-part-2/","summary":"A closer look at TestContext.CurrentContext in NUnit: the current test, its result, working directories and the Randomizer.","title":"Overview of the TestContext in NUnit. Part 2: CurrentContext"},{"content":" Originally published on get-testy.com in 2017. Written against NUnit 3.\nThis is the first post of a small series in which we will dig into the TestContext class. I originally had this as one post, but part way through it was quite obvious that there was too much information to cover.\nMany of you probably aren\u0026rsquo;t familiar with all the builtin properties and methods that you have access to when using NUnit. When NUnit is executing tests, it runs tests in an execution context, which has information about the environment they are running in and on the tests themselves. The TestContext gives you the ability to access that information as well as provide methods to improve your tests.\nThe TestContext is a singleton that has a lot of useful static properties and methods. You can access the TestContext in any project you are referencing the nunit framework. You can find all the basic information about the TestContext here - TestContext\nTest Context - Static Properties CurrentContext TestContext.CurrentContext - Gets the context of the currently executing test. This context is created separately for each test before it begins execution. I\u0026rsquo;ll go over the CurrentContext in my next post: Overview of the TestContext in NUnit. Part 2: Using the CurrentContext.\nOut TestContext.Out - Gets a TextWriter used for sending output to the current test result. If using the default IResultWriter, this will place the output into the XML result.\nFor instance, you might want to save the different greetings that we test.\n[Test] public void Hello() { string greeting = BlogSamples.HelloWorld.Hello(\u0026#34;get-testy\u0026#34;); TestContext.Out.WriteLine($\u0026#34;Greeting: {greeting}\u0026#34;); Assert.That(greeting.Equals(\u0026#34;Hello, get-testy.\u0026#34;, StringComparison.Ordinal)); } When this test gets executed, this would be the test case node inside the XML result.\n\u0026lt;test-case id=\u0026#34;0-1286\u0026#34; name=\u0026#34;Post\u0026#34; fullname=\u0026#34;BlogSamples.NUnit.Hello\u0026#34; methodname=\u0026#34;Hello\u0026#34; classname=\u0026#34;BlogSamples.NUnit.Hello\u0026#34; runstate=\u0026#34;Runnable\u0026#34; seed=\u0026#34;969179069\u0026#34; result=\u0026#34;Passed\u0026#34; start-time=\u0026#34;2017-11-07 19:14:25Z\u0026#34; end-time=\u0026#34;2017-11-07 19:14:25Z\u0026#34; duration=\u0026#34;0.012080\u0026#34; asserts=\u0026#34;1\u0026#34;\u0026gt; \u0026lt;output\u0026gt;\u0026lt;![CDATA[Greeting: get-testy]]\u0026gt;\u0026lt;/output\u0026gt; \u0026lt;/test-case\u0026gt; This can be useful when you want to track variables that are created inside the test case. When you are debugging failures that occur on build servers, having these types of outputs can help you diagnose failures easier.\nError TestContext.Error - Gets a TextWriter used for sending error output intended for immediate display. Use this to send messages to the Error output stream.\nFor instance, you might want to write out an Error that occurred but not stop the execution of the test while still capturing the errors.\nfor(int i = 0; i \u0026lt; 5; i++) { try { //Perform action and suppress exceptions } catch(System.Exception exception) { TestContext.Error.WriteLine($\u0026#34;Error: {exception.ToString()}\u0026#34;); } } Progress TestContext.Progress - Gets a TextWriter used for sending normal (non-error) output intended for immediate display. Use to send information out to the immediate console output. If you have a lot going on in your tests, it\u0026rsquo;s good practice to log what is occurring during your test execution.\nFor instance, you might want to track actions that are occurring in a loop.\nfor(int i = 0; i \u0026lt; 5; i++) { //Some Action TestContext.Progress.WriteLine($\u0026#34;Action number {i} has occurred.\u0026#34;); } TestParameters Test parameters are going to need their own post to show all the things that you can do. However, I will give a quick overview here just to give a taste of what you can do with the test parameters.\nThey can be supplied in two different ways: through the nunit console and through the nunit3 visual studio adapter. The static TestParameters property returns an object representing those passed-in parameters. The TestParameters class has the following methods and properties.\nproperties Count - The number of parameters. Names - A collection of the names of the supplied parameters. this[string name] - The value of a parameter. In Vb, use Item. methods Exists(string name) - Returns true if a parameter of that name exists. Get(string name) - Returns the same value as the indexer. Get\u0026lt;T\u0026gt;(string name, T defaultValue) - Returns the value of the parameter converted from a string to type T or the specified default if the parameter doesn\u0026rsquo;t exist. Throws an exception if conversion fails. To define the test parameters in visual studio, this would be what a simple *.runsettings file would look like.\n\u0026lt;RunSettings\u0026gt; \u0026lt;TestRunParameters\u0026gt; \u0026lt;Parameter name=\u0026#34;appUrl\u0026#34; value=\u0026#34;http://localhost\u0026#34; /\u0026gt; \u0026lt;/TestRunParameters\u0026gt; \u0026lt;/RunSettings\u0026gt; Then in our test, we might access the appUrl value like this.\nTestContext.Out.WriteLine($\u0026#34;{TestContext.Parameters[\u0026#34;appUrl\u0026#34;]}\u0026#34;); Test Context - Static Methods Write and WriteLine Writes text to the current test result and writes text to the current test result but followed by a newline.\nUsing the TestContext.Write and TestContext.WriteLine is preferred to using System.Console.Write and System.Console.WriteLine due to the System class redirecting the output to standard out. In basic scenarios, you often won\u0026rsquo;t run into a problem with using the System classes, but it\u0026rsquo;s best practice to use the TestContext classes wherever possible.\nAddFormatter If you have custom objects that you are comparing inside constraints, you can use a custom formatter to change the output to make it more human readable.\nAddFormatter(ValueFormatter formatter); AddFormatter(ValueFormatterFactory formatterFactory); We can use either a ValueFormatter or a ValueFormatterFactory, which are delegates, to format our objects.\nFor instance, we have a basic data transfer object (basicDTO) that we want to compare.\nclass basicDTO { public string Name { get; set; } } private string FormatBasicDTO(basicDTO val) { return $\u0026#34;Name: {val.Name}\u0026#34;; } [Test] public void Post() { TestContext.AddFormatter\u0026lt;basicDTO\u0026gt;(val =\u0026gt; FormatBasicDTO((basicDTO)val)); basicDTO firstDTO = new basicDTO() { Name = \u0026#34;First DTO\u0026#34; }; basicDTO secondDTO = new basicDTO() { Name = \u0026#34;Second DTO\u0026#34; }; Assert.That(firstDTO, Is.EqualTo(secondDTO)); } When we run this, since we did not override the ToEquals for the comparer on the basicDTO, this test will fail.\nWithout the ValueFormatter, we get:\n1) Failed : BlogSamples.NUnit.HelloWorld.Hello Expected: \u0026lt;BlogSamples.NUnit.HelloWorld.Hello+basicDTO\u0026gt; But was: \u0026lt;BlogSamples.NUnit.HelloWorld.Hello+basicDTO\u0026gt; With the ValueFormatter, we get:\n1) Failed : BlogSamples.NUnit.HelloWorld.Hello Expected: Name: Second DTO But was: Name: First DTO This is useful only for when you don\u0026rsquo;t have the ability to override the ToString on the object you are comparing. However, best practice would be to just give a useful ToString method and use the built in formatters.\nAddTestAttachment Attaches a file, with optional description, to the current test. The file will be attached to the test result in the xml report.\nAddTestAttachment(string filePath, string description = null); You might want to do this when:\nYou want to see what state a page is in when a front end test fails. Take a screenshot and add it as an attachment so you can view afterwards. You are building up an in-memory XmlDocument or DataTable and you want to see what the data looks like in a human readable format. Here would be an example of saving an in-memory XmlDocument to a file and attaching it to the Xml Test Result. This is what you might have inside your test.\n//Get the file path for where we want to save the file string xmlData = System.IO.Path.Combine(TestContext.CurrentContext.WorkDirectory, \u0026#34;xmlData.xml\u0026#34;); //Build up your in-memory XmlDocument XmlDocument doc = new XmlDocument(); XmlDeclaration xmlDeclaration = doc.CreateXmlDeclaration(\u0026#34;1.0\u0026#34;, \u0026#34;UTF-8\u0026#34;, null); XmlElement root = doc.DocumentElement; doc.InsertBefore(xmlDeclaration, root); XmlElement getTesty = doc.CreateElement(string.Empty, \u0026#34;get-testy\u0026#34;, string.Empty); doc.AppendChild(getTesty); XmlText text1 = doc.CreateTextNode(\u0026#34;text\u0026#34;); getTesty.AppendChild(text1); //Save your XmlDocument to the actual file on disc doc.Save(xmlData); //Attach the file to the Test Result TestContext.AddTestAttachment(xmlData, \u0026#34;In memory XML Data\u0026#34;); When you have that block of code execute inside your tests, you will then have an extra file generated and and extra section added to your TestResult.xml.\n\u0026lt;!-- attachment node is added to the TestResult.xml now --\u0026gt; \u0026lt;attachment\u0026gt; \u0026lt;filePath\u0026gt;E:\\SourceCode\\BlogSamples\\xmlData.xml\u0026lt;/filePath\u0026gt; \u0026lt;description\u0026gt;\u0026lt;![CDATA[In memory XML Data]]\u0026gt;\u0026lt;/description\u0026gt; \u0026lt;/attachment\u0026gt; That covers all of the static properties and methods. I would try playing around with the TestContext and see what you can do to improve your existing tests.\nMoving forward, when building new tests, you now have a few more tools in your toolbox for writing quality tests!\n","permalink":"https://jamespenning.dev/test-context-part-1/","summary":"A tour of NUnit\u0026rsquo;s TestContext static properties and methods: output, errors, progress, parameters, formatters and attachments.","title":"Overview of the TestContext in NUnit. Part 1: Static properties and methods"},{"content":" Originally published on get-testy.com in 2017. It targets NUnit 3 and .NET Framework; modern .NET uses dotnet new nunit and dotnet test.\nSince a lot of my future posts are going to be about NUnit, I want to make sure that you all have the basics of NUnit covered before we start talking in depth about the framework. In this post, I am going to go through how to build your first unit test.\nI am going to make some assumptions in this post, primarily that you have the basic understanding of .NET development and that you\u0026rsquo;ve used Visual Studio before. If that is all new to you, check out Microsoft\u0026rsquo;s site on Visual Studio for getting started - Getting Started\nPrerequisites:\n.NET Framework - Download IDE (Visual Studio and Visual Studio Code are what I use) - Download MSBuild (Only needed if you don\u0026rsquo;t have Visual Studio) - Download Before we begin, I want to point out the NUnit landing page on GitHub - NUnit. This is going to be where we get all the necessary code, executables, code samples, and documentation for using NUnit. I will be referencing their GitHub frequently for documentation, and it\u0026rsquo;s just a good place to have bookmarked for when you are developing with NUnit.\nMethod Under Test Now let\u0026rsquo;s get started! In this example, I am going to have two projects. One project that holds the code and one that holds the tests.\nBlogSamples BlogSamples.NUnit I am going to use the classic programming example of \u0026lsquo;Hello, World.\u0026rsquo; In BlogSamples, we have a class HelloWorld and a method Hello.\npublic static string Hello(string greeting) { return $\u0026#34;Hello, {greeting}.\u0026#34;; } Hello is a simple method that just takes in a string, mutates it, and returns the mutated string. So Hello(\u0026quot;Bojangles\u0026quot;) would return, Hello, Bojangles.\nNow to set up our tests for Hello.\nInstalling NUnit In order to write our NUnit tests for Hello, we need to add NUnit framework to our BlogSamples.NUnit project. The easiest way is going to be using Nuget. All the ways to install NUnit are listed here - Install\nInside Visual Studio, go to Tools -\u0026gt; NuGet Package Manager -\u0026gt; Package Manager Console. You will see the package manager console open at the bottom of the development window. In the window, type Install-Package -Project BlogSamples.NUnit NUnit and hit return. This will grab the latest stable build of NUnit and install it to our BlogSamples.NUnit project. You will see Successfully installed 'NUnit 3.8.1' to BlogSamples.NUnit in the console if it successfully installed.\nUnder references, you should see that you now have a reference to nunit.framework. This will be what we will be using to build up our tests.\nWriting the Test Now that we have our reference to NUnit, we need to add a project reference to the BlogSamples project. This will allow us to instantiate our HelloWorld class to test it. I\u0026rsquo;ll point you to this page if you don\u0026rsquo;t know how to add a reference - Add Reference. When we have all the references we need, we can start writing the tests for our Hello method.\nWhen I am writing unit tests, I like to make a class with the same name of the class that we are going to be testing. In BlogSamples, we have our Hello method in the HelloWorld class. So we will create the same Hello method in the HelloWorld class inside of the BlogSamples.NUnit project. Using this naming convention makes it very easy for discovering where the unit tests are for a particular class.\nOnce we create our Hello test method, we want to add the using statement for NUnit - using Nunit.Framework. Then we will declare our HelloWorld class as an NUnit TestFixture. We do this by adding the [TestFixture] attribute to the class. Now we have something that looks like this\u0026hellip;\nusing NUnit.Framework; namespace BlogSamples.NUnit { [TestFixture] public class HelloWorld { } } Now that we have laid out the structure of the test class, we can add our first test case. Our first iteration of our test might look like this\u0026hellip;\n[Test] public void Hello() { string greeting = BlogSamples.HelloWorld.Hello(\u0026#34;get-testy\u0026#34;); Assert.That(greeting, Is.EqualTo(\u0026#34;Hello, get-testy.\u0026#34;, StringComparison.Ordinal)); } For those that have never seen an NUnit test, it can seem like a lot is going on in this method. Let\u0026rsquo;s break down what comprises an nunit test.\nThe first thing is that we decorate the test method with the [Test] attribute. It is one of many attributes that you can put on a method to classify it as a test. In later posts, I will go over all the different ways that you can define tests. For now, the most basic way to create a test is to use the [Test] attribute. More info can be found here on Attributes. Since we are testing the Hello method in the HelloWorld class, we actually want to call that method with something that we define. This way we know exactly what the outcome with be. In this instance, we are sending in \u0026quot;get-testy\u0026quot; to Hello, so we would expect our method to return \u0026quot;Hello, get-testy.\u0026quot;. After we call the Method Under Test, we want to verify the results from method. We do this in the Assertion. In NUnit3, they have changed their Assertion model to be constraint based. This is in favor of their classic model, where in our case we might use Assert.AreEqual(\u0026quot;Hello, get-testy.\u0026quot;, greeting). Using the constraint based model gives a lot more freedom to do what we want inside the assertions. More info can be found here on Assertions. Running the Test There are a couple ways that we can run NUnit tests.\nThe Visual Studio NUnit3 Adapter - Adapter NUnit3 Console - Console NUnit3 GUI - GUI Depending on your needs, there will always be a better way to run your tests.\nIf you are going to be actively writing tests, it is probably the best to use the Visual Studio adapter. That way when you are writing your code and your tests, you don\u0026rsquo;t need to leave your IDE in order to run them. If you are going to be running your tests nightly, or during your build process, you are going to want to use the nunit console. If you are going to want to run all existing tests, not making any changes, against your code changes, you might use the GUI. **The NUnit3 GUI is still in preview, so I might wait to use this until it gets fully release. Let\u0026rsquo;s just use the Visual Studio adapter for this example. We can install the extension into our Visual Studio environment by going to Tools -\u0026gt; Extension Manager. Search for NUnit 3.0 Test Adapter and click install. I had to restart my Visual Studio in order for the installation to complete, it might be different for you. Just follow the directions that it gives you.\nOnce you have the extension installed, we want to open up the Test Explorer. If you don\u0026rsquo;t see it, go to Test -\u0026gt; Windows -\u0026gt; Test Explorer. By default, when you first open Visual Studio, it won\u0026rsquo;t have any tests loaded. Once you build your code, you should see the new test we created. Now it is as simple as right-clicking on the Hello test, and selecting Run Selected Tests.\nNow we should have everything set up in our development environment. We can start getting into more advanced topics. Let me know if this was helpful, too in-depth, not enough in-depth, etc. Please share your thoughts!\n","permalink":"https://jamespenning.dev/getting-started-with-nunit/","summary":"Write and run your first NUnit unit test in C#, from installing the packages to reading the results.","title":"Getting started with NUnit"},{"content":" Originally published on get-testy.com in 2017. The book list still holds up; the purchase links now point straight to the publisher listings on Amazon.\nOne of the great things about working at Relativity is the investment they put towards continual training. Every year we are given a budget of $3000 to use for training. People tend to use this budget to go to a conference, like Microsoft’s Build conference. This year, instead of going to a conference, I decided to start my library of software testing related literature.\nOne of the things that I have been trying to focus on is to expand my general software development knowledge. I wanted to learn as much as I could, such that when it came time at work to make a decision on how to implement something, I could make the most informed decision possible. What better way than to learn from those that have spent their entire careers building up that knowledge and are willing to share.\nSo far, I’ve read a few “staple” books that everyone recommends, The Pragmatic Programmer, Clean Code, and a few others. All material on how to become a better developer. Not just how to develop code, but how to develop good habits in order to develop good code. While I certainly learned a lot from those books, and definitely think that every engineer should read these books, I hadn’t read any material specific to testing software. Being a Software Test Engineer, I found that to be a little backwards. I then set out to the internet to do the simplest Google search that I could; “What are the best books on software testing.” Took the list and started to read reviews.\nHere is the list that I came up with:\nHow We Test Software at Microsoft View on Amazon\nHow Google Tests Software View on Amazon\nA Practitioner’s Guide to Software Test Design View on Amazon\nSpecification by Example: How Successful Teams Deliver the Right Software View on Amazon\nLessons Learned in Software Testing: A Context-Driven Approach View on Amazon\nThinking, Fast and Slow View on Amazon\nContinuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation View on Amazon\nContinuous Integration: Improving Software Quality and Reducing Risk View on Amazon\nMore Agile Testing: Learning Journeys for the Whole Team View on Amazon\nPerfect Software: And Other Illusions about Testing View on Amazon\nTesting Computer Software, 2nd Edition View on Amazon\nBeautiful Testing: Leading Professionals Reveal How They Improve Software (Theory in Practice) View on Amazon\nSoftware Testing: A Craftsman’s Approach, Fourth Edition View on Amazon\nExplore It!: Reduce Risk and Increase Confidence with Exploratory Testing View on Amazon\nI put How We Test Software at Microsoft and How Google Tests Software on the top of my list for a couple of reasons.\nFirst, in my career, I have only worked for one company. This means that I have only had insight into how one Engineering department approaches testing. I wanted to understand how the largest names in the software industry have approached testing. I will be making a post in the future to go into more specifics and details on these two books.\nSecond, I have been having a bit of an identity crisis as of late in current role. I have the official title of Software Engineer in Test, but I have been doing a lot of development work and manual QA work, but not a lot of work that I actually feel Software Test Engineers should be doing. In these two books, they talk about the different positions that they have and the different duties that each of them perform. It was really interesting to see how other companies view Software Test Engineers.\nLast, Google and Microsoft haven’t built up their companies and engineering departments by doing things wrong. They have to be doing something right. Learning from how they operate and how they approach testing.\nIf any of you readers have suggestions for books you think that I missed or that you want to get reviews on, send me a note and I can look into getting them on my list!\n","permalink":"https://jamespenning.dev/current-reading-list/","summary":"Fourteen books on software testing I picked to build my testing library, and why two of them went to the top.","title":"My software testing reading list"},{"content":"I\u0026rsquo;m James, from Iowa. I\u0026rsquo;m a software engineer, and I build the services that keep CI/CD pipelines fast, reliable and honest. These days I use AI in almost everything I do, from writing code to reviewing it to figuring out why a build turned red at 2 AM.\nI didn\u0026rsquo;t start here. I spent my early career as a test engineer, writing NUnit frameworks and integration tests in .NET. In 2017-18 I blogged about it at get-testy, and those posts still live here in the archive. Testing taught me the thing I still believe most: a red build should mean something real. Now I get to build the systems that make sure it does.\nThis blog is me learning out loud, in a few lanes:\nAI: how I actually use it at work, including where it helps and where it quietly breaks things. DevOps and CI/CD: pipelines, build systems and the services around them. Testing: where I came from, and why it matters even more now that agents write code. Sports betting with AI: I build and test my own betting model. I post the process and the misses, never picks or \u0026ldquo;locks.\u0026rdquo; 21+ only, and bet only what you\u0026rsquo;d spend on a night out. Fatherhood: I\u0026rsquo;m a dad of four. You\u0026rsquo;ll get the lessons, the chaos and the jokes, but no names or faces. Those are theirs to share. Fitness: I\u0026rsquo;m an aspiring hybrid runner trying to be faster than the lifters and stronger than the runners. Progress is slow, and I\u0026rsquo;ll show the bad days too. Why do all of this? Partly because I like it, and because AI made it too easy to only consume. I restarted this blog to be a creator again. Mostly, though, it\u0026rsquo;s because I want my kids to see what it looks like to keep learning new things, be wrong in public, and get a little better anyway.\nSay hi I\u0026rsquo;m newest on X: @jpenning0. I\u0026rsquo;m also on LinkedIn. Code lives on GitHub. You can also follow along by RSS.\n","permalink":"https://jamespenning.dev/about/","summary":"\u003cp\u003eI\u0026rsquo;m James, from Iowa. I\u0026rsquo;m a software engineer, and I build the services that keep CI/CD pipelines fast, reliable and honest. These days I use AI in almost everything I do, from writing code to reviewing it to figuring out why a build turned red at 2 AM.\u003c/p\u003e\n\u003cp\u003eI didn\u0026rsquo;t start here. I spent my early career as a test engineer, writing NUnit frameworks and integration tests in .NET. In 2017-18 I blogged about it at get-testy, and those posts still live here in the \u003ca href=\"/categories/testing/\"\u003earchive\u003c/a\u003e. Testing taught me the thing I still believe most: a red build should mean something real. Now I get to build the systems that make sure it does.\u003c/p\u003e","title":"About"},{"content":"I\u0026rsquo;m James, a software engineer. This blog is me learning out loud. Here\u0026rsquo;s the quickest way in.\nTopics Testing: Where I came from, and testing in the age of agents. DevOps \u0026amp; CI/CD: Pipelines, build systems and the services that keep them honest. From the get-testy archive: classic NUnit posts (2017-18) I wrote these as a test engineer, back when this blog was called get-testy. Some of the tooling has moved on, but the testing ideas hold up.\nMy software testing reading list (Oct 2017) Getting started with NUnit (Oct 2017) Overview of the TestContext in NUnit. Part 1: Static properties and methods (Nov 2017) Overview of the TestContext in NUnit. Part 2: CurrentContext (Nov 2017) Using .runsettings to configure your NUnit tests (Dec 2017) TestCaseData with TestCaseSource to better organize your tests (Jan 2018) Top 5 pitfalls writing integration tests (Mar 2018) NUnit\u0026#39;s classic vs. constraint assertion models (May 2018) What actually makes a test flaky? (Aug 2018) Keep up Read more about me, or follow along by RSS.\n","permalink":"https://jamespenning.dev/start-here/","summary":"\u003cp\u003eI\u0026rsquo;m James, a software engineer. This blog is me learning out loud. Here\u0026rsquo;s the quickest way in.\u003c/p\u003e\n\u003ch2 id=\"topics\"\u003eTopics\u003c/h2\u003e\n\u003cul class=\"topics-list\"\u003e\n  \u003cli\u003e\u003ca href=\"/categories/testing/\"\u003eTesting\u003c/a\u003e: Where I came from, and testing in the age of agents.\u003c/li\u003e\n  \u003cli\u003e\u003ca href=\"/categories/devops/\"\u003eDevOps \u0026amp; CI/CD\u003c/a\u003e: Pipelines, build systems and the services that keep them honest.\u003c/li\u003e\n\u003c/ul\u003e\n\n\u003ch2 id=\"from-the-get-testy-archive-classic-nunit-posts-2017-18\"\u003eFrom the get-testy archive: classic NUnit posts (2017-18)\u003c/h2\u003e\n\u003cp\u003eI wrote these as a test engineer, back when this blog was called get-testy. Some of the tooling has moved on, but the testing ideas hold up.\u003c/p\u003e","title":"Start here"}]