Open Source Hacking
Popcorn and Popcorn Maker Testing – This is How We Do It
No jokes. I literally had that song pop into my head when trying to think of a title. Enjoy.
My primary focus over this summer has been and will be the testing we have in place for Popcorn Maker. It’s been severely lacking and we need to change that. However, I wanted to take the time to explain everything we do in both projects; what we use and why. First, let’s talk about testing in general.
So, Why is it Important?
This may be a dumb thing to talk about but I’m sure someone would want to know and for completions sake I’m going to briefly talk about it. No matter what kind of software you are writing – library or full fledged application – what you are working on will always have expected functionality. You want to have robust testing because it allows you to easily check against this expected functionality and know if you have broken any of that before you bring it out of development and into production.
Popcorn Maker Testing
However in the end this doesn’t solve the problem of the UI testing we need. In the past Chris and I played with the Selenium IDE to create some tests written in python based around our testing template. These actually worked before kindofish (totally a word by the way) until we implemented our own DragNDrop which broke these tests and to add to that when we used iframes for our editor and dialogs it was difficult to interact with them using their older Selenium RC. However I took a look recently at another option they have called the Selenium WebDriver API and see if it along with all the changes we have been making would change the results we were having in the past.
Sure enough, it did! I was able to get some basic tests working for our Export Dialog boxes and some basic drag and drop action going on track events. Here’s the code I wrote for the export dialog tests:
import unittest import time from selenium import webdriver from selenium.webdriver import ActionChains from selenium.webdriver.common.keys import Keys from selenium.common.exceptions import NoSuchElementException class PM_HeaderTests(unittest.TestCase): def setUp(self): self.driver = webdriver.Firefox() self.action = ActionChains( self.driver ) def test_export_dialog(self): driver = self.driver action = self.action templateTitle = "Basic template" driver.get( "http://localhost:8888/templates/test.html" ) driver.maximize_window() self.assertEqual( templateTitle, driver.title ) # Ensure the button is present on the page, then click it sourceButton = driver.find_element_by_id( "butter-header-source" ) self.assertTrue( sourceButton ) sourceButton.click() # Ensure an element with the class_name is present # Only ever have this class on the page at one time sourceDialog = driver.find_element_by_class_name( "butter-dialog" ) self.assertTrue( sourceDialog ) # Get JSON, HTML button in export dialog and their text areas jsonButton = driver.find_element_by_class_name( "json-button" ) htmlButton = driver.find_element_by_class_name( "html-button" ) jsonTextArea = driver.find_element_by_class_name( "json-export" ) htmlTextArea = driver.find_element_by_class_name( "html-export" ) # Ensure the elements were on the page and retrieved self.assertTrue( jsonButton ) self.assertTrue( htmlButton ) self.assertTrue( jsonTextArea ) self.assertTrue( htmlTextArea ) # Check that the appropriate display properties were set when switching to the JSON jsonButton.click() self.assertEqual( jsonTextArea.value_of_css_property( "display" ), "block" ) self.assertEqual( htmlTextArea.value_of_css_property( "display" ), "none" ) # Check that the appropriate display properties were set when switching to the HTML htmlButton.click() self.assertEqual( jsonTextArea.value_of_css_property( "display" ), "none" ) self.assertEqual( htmlTextArea.value_of_css_property( "display" ), "block" ) # Check pressing the ESCAPE key closes the dialog try: sourceDialog.send_keys( Keys.ESCAPE ) sourceDialog = driver.find_element_by_class_name( "butter-dialog" ) except NoSuchElementException as e: self.assertTrue( "Dialog not found, as expected" ) sourceButton.click() dialogCloseButton = driver.find_element_by_class_name( "close-button" ) # Check pressing the close button closes the dialog try: dialogCloseButton.click() sourceDialog = driver.find_element_by_class_name( "butter-dialog" ) except NoSuchElementException as e: self.assertTrue( "Dialog not found, as expected" ) time.sleep( 2 ) def tearDown(self): self.driver.quit() if __name__ == "__main__": unittest.main()
One of my first few stabs at python so don’t judge me. To summarize, when run this will:
1. Open a Firefox Browser.
2. Navigate to our test page.
3. Maximize the window.
4. Press the View Source button in our header (Export Dialog).
5. Press the View JSON button.
6. Check appropriate display styles for both JSON and HTML text areas.
7. Press the View HTML button.
8. Recheck display styles for both text areas.
9. Check if closing with the ESCAPE key works as expected.
10. Press the View Source button again and then try closing with the explicit close button.
What I’m really hoping for is to find some better sort of callback like system so I know when things are done. Right now things seem fine but I’m half worried I will run into asynchronous issues in the future.
Continuous Integration – Test Swarm
This is actually something we have sort of had for a while now however we never used it much. I would say it boiled down to two factors:
1. We haven’t spent nearly as much time on Popcorn over the last few months and this was the only project it worked with at the time.
2. It wasn’t very convenient to use. The way we had it setup to listen for/run jobs was kind of clunky and cumbersome to use.
Luckily, this has changed thanks to more work done by Jon bringing in Botio to the project and Chris by not only updating our test swarm stuff but also adding integration into our botio system. We use a pull request based system for everything change we make to the project, meaning for every ticket where changes to the code are being made a pull request is made to help review the code. With botio listening for various commands we can comment in the pull request with a particular command and then have results sent back to the pull request. It’s helped make things very efficient for us and when we add in Chris’ work for /botio try to run a job on our test swarm instance we will be golden.
Hope you enjoyed my mini novel on how we go about testing for the projects Popcorn and Popcorn Maker. There will definitely be more to come later on our advances with Selenium. If you want to learn more out what Chris has been doing, check out some of his blog posts such as Time to Automate Butter Unit Testing and Popcorn and Butter Testing.