Difference between revisions of "AUTOMATE WHAT'S NEEDED"

From Test Automation Patterns
Jump to navigation Jump to search
 
(7 intermediate revisions by 2 users not shown)
Line 21: Line 21:
 
<span style="font-size: 16px">When people get "stuck in" to automation, they can get carried away with what can be done, and may want to automate tests that aren't really important enough to automate.</span>  
 
<span style="font-size: 16px">When people get "stuck in" to automation, they can get carried away with what can be done, and may want to automate tests that aren't really important enough to automate.</span>  
 
=<span style="font-size: 16px">'''Issues addressed by this pattern'''</span>=
 
=<span style="font-size: 16px">'''Issues addressed by this pattern'''</span>=
<span style="font-size: 16px">''[[INADEQUATE COMMUNICATION]]''</span><br /> <span style="font-size: 16px">''[[NO PREVIOUS TEST AUTOMATION]]''</span><br /> ''<span style="font-size: 16px">[[UNFOCUSED AUTOMATION]]</span>''<br /> <br /> '''<span style="font-size: 16px">Experiences: Example 1 of 2</span>'''<br /> <br /> <span style="font-size: 16px">''Jochim Van Dorpe'' writes:</span><br /> <br /> <span style="font-size: 16px">I like the idea/theory of automated tests vs. automated testing. That's why I look as much as possible for things that we could automate in our test process, things I used to do manually, but are repetitive, boring and not interesting. So we automated some stuff, some right from the beginning of the project, others I still come up with.</span><br /> <br /> <span style="font-size: 16px">Some solutions are interesting for me as a test-analyst, others are for my Project Lead (PL) or even for the client. Some are offered by the tools we use, some are custom-made by our team.</span><br /> <br /> <span style="font-size: 16px">Some examples are:</span><br />  
+
<span style="font-size: 16px">''[[INADEQUATE COMMUNICATION]]''</span><br /> <span style="font-size: 16px">''[[NO PREVIOUS TEST AUTOMATION]]''</span><br /> ''<span style="font-size: 16px">[[UNFOCUSED AUTOMATION]]</span>''<br /> <br />
 
+
=<span style="font-size: 16px">'''Experiences'''</span>=
 +
<span style="font-size: 16px">'''Example 1'''</span>
 +
<br /> <br /> <span style="font-size: 16px">''Jochim Van Dorpe'' writes:</span><br /> <br /> <span style="font-size: 16px">I like the idea/theory of automated tests vs. automated testing. That's why I look as much as possible for things that we could automate in our test process, things I used to do manually, but are repetitive, boring and not interesting. So we automated some stuff, some right from the beginning of the project, others I still come up with.</span><br /> <br /> <span style="font-size: 16px">Some solutions are interesting for me as a test-analyst, others are for my Project Lead (PL) or even for the client. Some are offered by the tools we use, some are custom-made by our team.</span><br /> <br /> <span style="font-size: 16px">Some examples are:</span><br />  
 
* <span style="font-size: 16px">Before any tests begin, we automatically drop the test Database (DB)</span>
 
* <span style="font-size: 16px">Before any tests begin, we automatically drop the test Database (DB)</span>
 
* <span style="font-size: 16px">We populate the DB automatically with 'the things we want': typecodes, dataset, ...</span>
 
* <span style="font-size: 16px">We populate the DB automatically with 'the things we want': typecodes, dataset, ...</span>
Line 31: Line 33:
 
* <span style="font-size: 16px">A readable document of all the test cases can be composed on demand (testlink)</span>
 
* <span style="font-size: 16px">A readable document of all the test cases can be composed on demand (testlink)</span>
 
* <span style="font-size: 16px">A TSH (test scenario/case hierarchy) can be composed automatically</span>
 
* <span style="font-size: 16px">A TSH (test scenario/case hierarchy) can be composed automatically</span>
<br /> <span style="font-size: 16px">The custom-made things are re-usable by other projects in our company.</span><br /> <br /> <br /> '''<span style="font-size: 16px">Experiences: Example 2 of 2</span>'''<br /> <span style="font-size: 16px">''Gaby Spengler'' describes how she helps testers find out if a test case has already been implemented:</span><br />  
+
<br /> <span style="font-size: 16px">The custom-made things are re-usable by other projects in our company.</span><br /> <br /> <br />  
 +
<span style="font-size: 16px">'''Example 2'''</span>
 +
<br /> <span style="font-size: 16px">''Gaby Spengler'' describes how she helps testers find out if a test case has already been implemented:</span><br />  
 
{| class="wiki_table"
 
{| class="wiki_table"
 
| <span style="font-size: 16px">Sometimes testers don’t know if a test case for a specific requirement has already been implemented. I have written a macro in Excel that allows testers to search the test case directory for a specified keyword and that creates a list of all the test cases that contain the keyword.</span><br /> <br /> <span style="font-size: 16px">Here is how the macro is coded:</span><br />
 
| <span style="font-size: 16px">Sometimes testers don’t know if a test case for a specific requirement has already been implemented. I have written a macro in Excel that allows testers to search the test case directory for a specified keyword and that creates a list of all the test cases that contain the keyword.</span><br /> <br /> <span style="font-size: 16px">Here is how the macro is coded:</span><br />
 
| <br />
 
| <br />
 
|-
 
|-
| <span style="font-size: 16px">'Public full_name As String</span><br /> <span style="font-size: 16px">Public TFDir As String</span><br /> <span style="font-size: 16px">Public Searchtext As String</span><br /> <span style="font-size: 16px">'Public Makro As String</span><br /> <span style="font-size: 16px">Sub TF_Search()</span><br /> <br /> <span style="font-size: 16px">'The directory to be searched and the search argument are taken from the register „Testcase search“.</span><br /> <span style="font-size: 16px"> TFDir = ""</span><br /> <span style="font-size: 16px"> Worksheets("Testcase search").Activate</span><br /> <span style="font-size: 16px"> TFDir = Cells(2, 1)</span><br /> <span style="font-size: 16px"> Searchtext = Cells(5, 1)</span><br /> <br /> <span style="font-size: 16px">'The following procedure browses thru the test case directory and looks for test cases that contain the Searchtext and lists them in the register “Testcase search”</span><br /> <span style="font-size: 16px"> TFFiles_Search TFDir</span><br /> <br /> <span style="font-size: 16px">End Sub</span><br /> <br /> <span style="font-size: 16px">Sub TFFiles_Search(TFDir)</span><br /> <span style="font-size: 16px">'</span><br /> <span style="font-size: 16px">'This procedure browses thru the test case directory and looks for test cases that contain the Searchtext and lists them in the register “Testcase search”</span><br /> <span style="font-size: 16px">'</span><br /> <span style="font-size: 16px">Dim LineSource As Long</span><br /> <span style="font-size: 16px">Dim LineDest As Long</span><br /> <br /> <span style="font-size: 16px">Dim This_Workbook As String</span><br /> <span style="font-size: 16px">Dim Searched_Workbook As String</span><br /> <br /> <span style="font-size: 16px">Dim WSHShell, fso, oArgs</span><br /> <span style="font-size: 16px">Dim oFolders, oSubFolder, oFiles, Folder</span><br /> <span style="font-size: 16px">Dim i, Text, FileX, TCFile()</span><br /> <br /> <br /> <span style="font-size: 16px">'</span><br /> <span style="font-size: 16px"> This_Workbook = "Makro Test case search V1.0.xls"</span><br /> <span style="font-size: 16px"> Searched_Workbook = "Testcases INK.xls"</span><br /> <span style="font-size: 16px"> Set WSHShell = CreateObject("WScript.Shell")</span><br /> <span style="font-size: 16px"> Set fso = CreateObject("Scripting.FileSystemObject")</span><br /> <span style="font-size: 16px">' Set oArgs = Arguments</span><br /> <br /> <span style="font-size: 16px"> i = 0</span><br /> <span style="font-size: 16px"> Set oFolders = fso.GetFolder(TFOrdner)</span><br /> <span style="font-size: 16px"> Set oFiles = oFolders.Files</span><br /> <br /> <span style="font-size: 16px"> For Each FileX In oFiles</span><br /> <span style="font-size: 16px"> ' Test case file names are stored in the array TCFile</span><br /> <span style="font-size: 16px"> Text = Mid(FileX.Name, 1, 9)</span><br /> <span style="font-size: 16px"> If Text = "Testcases" Then</span><br /> <span style="font-size: 16px"> ReDim Preserve TCFile(i)</span><br /> <span style="font-size: 16px"> TCFile(i) = TFDir + "\" + FileX.Name</span><br /> <span style="font-size: 16px"> i = i + 1</span><br /> <span style="font-size: 16px"> End If</span><br /> <span style="font-size: 16px"> Next</span><br /> <br /> <span style="font-size: 16px"> ' Test cases that contain the sought text are inserted starting with line 10 </span><br /> <span style="font-size: 16px"> LineDest = 10</span><br /> <br /> <span style="font-size: 16px"> For i = 0 To UBound(TCFile)</span><br /> <span style="font-size: 16px"> 'Open the test case file</span><br /> <span style="font-size: 16px"> Workbooks.Open TCFile(i)</span><br /> <br /> <br /> <span style="font-size: 16px"> Searched_Workbook = TCFile(i)</span><br /> <br /> <span style="font-size: 16px"> Length = Len(TCFile(i))</span><br /> <span style="font-size: 16px"> pos1 = 1</span><br /> <span style="font-size: 16px"> Do Until Length = 0</span><br /> <span style="font-size: 16px"> pos1 = InStr(1, Searched_Workbook, "\")</span><br /> <span style="font-size: 16px"> If pos1 = 0 Then</span><br /> <span style="font-size: 16px"> Exit Do</span><br /> <span style="font-size: 16px"> End If</span><br /> <span style="font-size: 16px"> Length = Length - pos1</span><br /> <span style="font-size: 16px"> Searched_Workbook = Mid(Searched_Workbook, (pos1 + 1), Length)</span><br /> <br /> <span style="font-size: 16px"> Loop</span><br /> <br /> <span style="font-size: 16px"> With Worksheets("Testcases").Range("A2:J1000")</span><br /> <span style="font-size: 16px"> Set c = .Find(Searchtext)</span><br /> <span style="font-size: 16px"> ' If the Searchtext is found then the test case is copied to the macro file</span><br /> <span style="font-size: 16px"> If Not c Is Nothing Then</span><br /> <span style="font-size: 16px"> firstAddress = c.Address</span><br /> <br /> <br /> <span style="font-size: 16px"> ' The search goes on until the first hit has been found again</span><br /> <span style="font-size: 16px"> Do</span><br /> <span style="font-size: 16px"> 'The line in which the Searchtext was found, is stored</span><br /> <span style="font-size: 16px"> LineSource = c.Row</span><br /> <span style="font-size: 16px"> Worksheets("Testcases").Activate</span><br /> <span style="font-size: 16px"> Rows(LineSource).Select</span><br /> <span style="font-size: 16px"> Selection.Copy</span><br /> <span style="font-size: 16px"> Workbooks(This_Workbook).Worksheets("Testcase_Search").Activate</span><br /> <span style="font-size: 16px"> Rows(LineDest).Select</span><br /> <span style="font-size: 16px"> 'Selection.Insert Shift:=xlDown</span><br /> <span style="font-size: 16px"> 'Selection.Insert</span><br /> <span style="font-size: 16px"> Selection.PasteSpecial Paste:=xlPasteValues, Operation:=xlNone, SkipBlanks _</span><br /> <span style="font-size: 16px"> :=False, Transpose:=False</span><br /> <span style="font-size: 16px"> ' Rows("10:10").Select</span><br /> <span style="font-size: 16px"> With Selection</span><br /> <span style="font-size: 16px"> .HorizontalAlignment = xlGeneral</span><br /> <span style="font-size: 16px"> .VerticalAlignment = xlTop</span><br /> <span style="font-size: 16px"> .WrapText = True</span><br /> <span style="font-size: 16px"> .Orientation = 0</span><br /> <span style="font-size: 16px"> .AddIndent = False</span><br /> <span style="font-size: 16px"> .IndentLevel = 0</span><br /> <span style="font-size: 16px"> .ShrinkToFit = False</span><br /> <span style="font-size: 16px"> .ReadingOrder = xlContext</span><br /> <span style="font-size: 16px"> .MergeCells = False</span><br /> <span style="font-size: 16px"> End With</span><br /> <br /> <span style="font-size: 16px"> LineDest = LineDest + 1</span><br /> <br /> <span style="font-size: 16px"> 'Workbooks(TCFile(i)).Worksheets("Testcases").Activate</span><br /> <span style="font-size: 16px"> Workbooks(Searched_Workbook).Worksheets("Testcases").Activate</span><br /> <span style="font-size: 16px"> Set c = .FindNext(c)</span><br /> <span style="font-size: 16px"> Loop While Not c Is Nothing And c.Address <> firstAddress</span><br /> <span style="font-size: 16px"> End If</span><br /> <span style="font-size: 16px"> End With</span><br /> <span style="font-size: 16px"> Workbooks(Searched_Workbook).Worksheets("Testcases").Cells(1, 1).Copy</span><br /> <span style="font-size: 16px"> 'Close the testcase file</span><br /> <span style="font-size: 16px"> Workbooks(Searched_Workbook).Close SaveChanges:=False</span><br /> <span style="font-size: 16px"> '</span><br /> <span style="font-size: 16px"> Next</span><br /> <br /> <span style="font-size: 16px"> ' Set Autofilter</span><br /> <span style="font-size: 16px"> Workbooks(This_Workbook).Worksheets("Testcase search").Activate</span><br /> <span style="font-size: 16px"> Rows("9:9").Select</span><br /> <span style="font-size: 16px"> Selection.AutoFilter</span><br /> <br /> <span style="font-size: 16px"> End Sub</span><br />
+
| <span style="font-size: 16px">'Public full_name As String</span><br /> <span style="font-size: 16px">Public TFDir As String</span><br /> <span style="font-size: 16px">Public Searchtext As String</span><br /> <span style="font-size: 16px">'Public Makro As String</span><br /> <span style="font-size: 16px">Sub TF_Search()</span><br /> <br /> <span style="font-size: 16px">'The directory to be searched and the search argument are taken from the register „Testcase search“.</span><br /> <span style="font-size: 16px"> TFDir = ""</span><br /> <span style="font-size: 16px"> Worksheets("Testcase search").Activate</span><br /> <span style="font-size: 16px"> TFDir = Cells(2, 1)</span><br /> <span style="font-size: 16px"> Searchtext = Cells(5, 1)</span><br /> <br />  
 +
<span style="font-size: 16px">'The following procedure browses thru the test case directory and looks for test cases that contain the Searchtext and lists them in the register “Testcase search”</span><br /> <span style="font-size: 16px"> TFFiles_Search TFDir</span><br /> <br /> <span style="font-size: 16px">End Sub</span><br /> <br /> <span style="font-size: 16px">Sub TFFiles_Search(TFDir)</span><br /> <span style="font-size: 16px">'</span><br /> <span style="font-size: 16px">'This procedure browses thru the test case directory and looks for test cases that contain the Searchtext and lists them in the register “Testcase search”</span><br /> <span style="font-size: 16px">'</span><br /> <span style="font-size: 16px">Dim LineSource As Long</span><br /> <span style="font-size: 16px">Dim LineDest As Long</span><br /> <br /> <span style="font-size: 16px">Dim This_Workbook As String</span><br /> <span style="font-size: 16px">Dim Searched_Workbook As String</span><br /> <br /> <span style="font-size: 16px">Dim WSHShell, fso, oArgs</span><br /> <span style="font-size: 16px">Dim oFolders, oSubFolder, oFiles, Folder</span><br /> <span style="font-size: 16px">Dim i, Text, FileX, TCFile()</span><br /> <br /> <br /> <span style="font-size: 16px">'</span><br /> <span style="font-size: 16px"> This_Workbook = "Makro Test case search V1.0.xls"</span><br /> <span style="font-size: 16px"> Searched_Workbook = "Testcases INK.xls"</span><br /> <span style="font-size: 16px"> Set WSHShell = CreateObject("WScript.Shell")</span><br /> <span style="font-size: 16px"> Set fso = CreateObject("Scripting.FileSystemObject")</span><br /> <span style="font-size: 16px">' Set oArgs = Arguments</span><br /> <br /> <span style="font-size: 16px"> i = 0</span><br /> <span style="font-size: 16px"> Set oFolders = fso.GetFolder(TFOrdner)</span><br /> <span style="font-size: 16px"> Set oFiles = oFolders.Files</span><br /> <br /> <span style="font-size: 16px"> For Each FileX In oFiles</span><br /> <span style="font-size: 16px"> ' Test case file names are stored in the array TCFile</span><br /> <span style="font-size: 16px"> Text = Mid(FileX.Name, 1, 9)</span><br /> <span style="font-size: 16px"> If Text = "Testcases" Then</span><br /> <span style="font-size: 16px"> ReDim Preserve TCFile(i)</span><br /> <span style="font-size: 16px"> TCFile(i) = TFDir + "\" + FileX.Name</span><br /> <span style="font-size: 16px"> i = i + 1</span><br /> <span style="font-size: 16px"> End If</span><br /> <span style="font-size: 16px"> Next</span><br /> <br /> <span style="font-size: 16px"> ' Test cases that contain the sought text are inserted starting with line 10 </span><br /> <span style="font-size: 16px"> LineDest = 10</span><br /> <br /> <span style="font-size: 16px"> For i = 0 To UBound(TCFile)</span><br /> <span style="font-size: 16px"> 'Open the test case file</span><br /> <span style="font-size: 16px"> Workbooks.Open TCFile(i)</span><br /> <br /> <br /> <span style="font-size: 16px"> Searched_Workbook = TCFile(i)</span><br /> <br /> <span style="font-size: 16px"> Length = Len(TCFile(i))</span><br /> <span style="font-size: 16px"> pos1 = 1</span><br /> <span style="font-size: 16px"> Do Until Length = 0</span><br /> <span style="font-size: 16px"> pos1 = InStr(1, Searched_Workbook, "\")</span><br /> <span style="font-size: 16px"> If pos1 = 0 Then</span><br /> <span style="font-size: 16px"> Exit Do</span><br /> <span style="font-size: 16px"> End If</span><br /> <span style="font-size: 16px"> Length = Length - pos1</span><br /> <span style="font-size: 16px"> Searched_Workbook = Mid(Searched_Workbook, (pos1 + 1), Length)</span><br /> <br /> <span style="font-size: 16px"> Loop</span><br /> <br /> <span style="font-size: 16px"> With Worksheets("Testcases").Range("A2:J1000")</span><br /> <span style="font-size: 16px"> Set c = .Find(Searchtext)</span><br /> <span style="font-size: 16px"> ' If the Searchtext is found then the test case is copied to the macro file</span><br /> <span style="font-size: 16px"> If Not c Is Nothing Then</span><br /> <span style="font-size: 16px"> firstAddress = c.Address</span><br /> <br /> <br /> <span style="font-size: 16px"> ' The search goes on until the first hit has been found again</span><br /> <span style="font-size: 16px"> Do</span><br /> <span style="font-size: 16px"> 'The line in which the Searchtext was found, is stored</span><br /> <span style="font-size: 16px"> LineSource = c.Row</span><br /> <span style="font-size: 16px"> Worksheets("Testcases").Activate</span><br /> <span style="font-size: 16px">  
 +
Rows(LineSource).Select</span><br /> <span style="font-size: 16px"> Selection.Copy</span><br /> <span style="font-size: 16px"> Workbooks(This_Workbook).Worksheets("Testcase_Search").Activate</span><br /> <span style="font-size: 16px"> Rows(LineDest).Select</span><br /> <span style="font-size: 16px"> 'Selection.Insert Shift:=xlDown</span><br /> <span style="font-size: 16px"> 'Selection.Insert</span><br /> <span style="font-size: 16px"> Selection.PasteSpecial Paste:=xlPasteValues, Operation:=xlNone, SkipBlanks _</span><br /> <span style="font-size: 16px"> :=False, Transpose:=False</span><br /> <span style="font-size: 16px"> ' Rows("10:10").Select</span><br /> <span style="font-size: 16px"> With Selection</span><br /> <span style="font-size: 16px"> .HorizontalAlignment = xlGeneral</span><br /> <span style="font-size: 16px"> .VerticalAlignment = xlTop</span><br /> <span style="font-size: 16px"> .WrapText = True</span><br /> <span style="font-size: 16px"> .Orientation = 0</span><br /> <span style="font-size: 16px"> .AddIndent = False</span><br /> <span style="font-size: 16px"> .IndentLevel = 0</span><br /> <span style="font-size: 16px"> .ShrinkToFit = False</span><br /> <span style="font-size: 16px"> .ReadingOrder = xlContext</span><br /> <span style="font-size: 16px"> .MergeCells = False</span><br /> <span style="font-size: 16px"> End With</span><br /> <br /> <span style="font-size: 16px"> LineDest = LineDest + 1</span><br /> <br /> <span style="font-size: 16px">  
 +
'Workbooks(TCFile(i)).Worksheets("Testcases").Activate</span><br /> <span style="font-size: 16px"> Workbooks(Searched_Workbook).Worksheets("Testcases").Activate</span><br /> <span style="font-size: 16px"> Set c = .FindNext(c)</span><br /> <span style="font-size: 16px"> Loop While Not c Is Nothing And c.Address <> firstAddress</span><br /> <span style="font-size: 16px"> End If</span><br /> <span style="font-size: 16px"> End With</span><br /> <span style="font-size: 16px"> Workbooks(Searched_Workbook).Worksheets("Testcases").Cells(1, 1).Copy</span><br /> <span style="font-size: 16px"> 'Close the testcase file</span><br /> <span style="font-size: 16px"> Workbooks(Searched_Workbook).Close SaveChanges:=False</span><br /> <span style="font-size: 16px"> '</span><br /> <span style="font-size: 16px"> Next</span><br /> <br /> <span style="font-size: 16px"> ' Set Autofilter</span><br /> <span style="font-size: 16px"> Workbooks(This_Workbook).Worksheets("Testcase search").Activate</span><br /> <span style="font-size: 16px"> Rows("9:9").Select</span><br /> <span style="font-size: 16px"> Selection.AutoFilter</span><br /> <br /> <span style="font-size: 16px"> End Sub</span><br />
 
| <br />
 
| <br />
 
|}
 
|}
<br /> <span style="font-size: 16px">If you have also used this pattern, please add your name and a brief story of how you used this pattern: your context, what you did, and how well it worked - or how it didn't work!</span><br /> <br /> <span style="font-size: 14px">.................................................................................................................[[Main Page]] / Back to [[Process Patterns]] / Back to [[Test Automation Patterns]]</span></div>
+
<span style="font-size: 16px">'''Example 3'''</span>
 +
<br /><span style="font-size: 16px">''Sergiusz Golec'' writes:</span><br /> <br /> <span style="font-size: 16px">One day a set of sample clients/partners observed a not desired end state for their transactions. "Hard to find & reproduce". Critical details & triggers - missing. Except the known day. Hard to reproduce & analyze.</span><br />
 +
<span style="font-size: 16px">Drop from webapp debug logs contained 9 files around 90 MB. Jms message response content with an issue not to be covered with known log analyzers. Every file around 30'000 lines long, 10'486'203 characters long. Interesting details - spreaded among different hours (timestamps), contexts and transactions. As if extremely thin layer of a butter on a 90 MB slice of bread.</span><br />
 +
<span style="font-size: 16px">Automatically it was half a day task. To write and check the needed outcome.</span><br />
 +
<span style="font-size: 16px">Script read the input file(s), line by line, and with a function (fWorthOrNotWriting) based upon keywords (like interesting XML tags, Partner ID(s) (shown in a different formats), environment, transaction  type, state or known errors) started writing the desired output (into another file). Script collected the entire XML messages into a separate file. 808 lines from around 9*30'000 lines ( = 270 K lines).</span><br />
 +
<span style="font-size: 16px">808 lines long content manually was trimmed and boiled eventually to the single perfect example: 36-lines-long XML. Alienated & identified issue. Easy to work with and fix by developers.</span><br />
 +
<span style="font-size: 16px">Manually - it would make my soul cry for a week. To look for that specific issue.</span><br /><br />
 +
<br /> <span style="font-size: 16px">If you have also used this pattern and would like to contribute your experience to the wiki, please go to [[Feedback]] to submit your experience or comment.</span><br /> <br />  
 +
 
 +
<span style="font-size: 14px">.................................................................................................................[[Main Page]] / Back to [[Process Patterns]] / Back to [[Test Automation Patterns]]</span></div>

Latest revision as of 10:19, 24 February 2019

.................................................................................................................Main Page / Back to Process Patterns / Back to Test Automation Patterns

Pattern summary

Automate what the developers or the testers need, even if it isn’t tests!

Category

Process

Context

This pattern is appropriate when your automated tests will be around for a long time, but also when you just want to write one-off or disposable scripts.
Automating what isn't needed is never a good idea!

Description

Automate the tests that will give the most value. "Smoke tests" that are run every time there is any change, for example, would be good candidates for automation, as they would be run frequently and give confidence that the latest change has not destroyed everything.
When you think about test automation, the first tests that usually come to mind are regression tests that execute themselves at night or on week-ends. Actually with automation one can support testers in many other ways, because even when testing manually (even with exploratory tests) there are lots of repetitive tasks that could be easily automated. You may need only a little script in Python or SQL to make a tester’s life that much easier.

James Tony: Keep a balance between trying to test all the code paths (which is generally good) and trying to test every possible combination of inputs (which is generally bad, because it means the same lines of code get executed thousands of times, and the test suite takes so long that it doesn’t get used) – the aim should be to maximize the (customer-relevant) “bugs for your buck”, i.e. the maximum number of customer-relevant issues highlighted for the smallest expenditure of time and money

Implementation

Some suggestions:

  • AUTOMATE GOOD TESTS: Automate only the tests that bring the most Return on Investment (ROI)
  • KNOW WHEN TO STOP: Not all test cases can or should be automated
  • SHARE INFORMATION: what do testers need? What could you deliver them? Start supporting them there.
  • Try to get at least some developers "into the boat".


Good candidates for automation in addition to tests are:

  • Complex set-ups: they can be easily automated and are great time savers for testers.
  • DB-Data: Database-data can be automatically extracted or loaded for use in creating initial conditions or checking results. Such support is valued by developers and testers alike.

Potential problems

When people get "stuck in" to automation, they can get carried away with what can be done, and may want to automate tests that aren't really important enough to automate.

Issues addressed by this pattern

INADEQUATE COMMUNICATION
NO PREVIOUS TEST AUTOMATION
UNFOCUSED AUTOMATION

Experiences

Example 1

Jochim Van Dorpe writes:

I like the idea/theory of automated tests vs. automated testing. That's why I look as much as possible for things that we could automate in our test process, things I used to do manually, but are repetitive, boring and not interesting. So we automated some stuff, some right from the beginning of the project, others I still come up with.

Some solutions are interesting for me as a test-analyst, others are for my Project Lead (PL) or even for the client. Some are offered by the tools we use, some are custom-made by our team.

Some examples are:

  • Before any tests begin, we automatically drop the test Database (DB)
  • We populate the DB automatically with 'the things we want': typecodes, dataset, ...
  • Of course we have the scripts that run the tests ...
  • Tests (unitils, selenium, soapUi) are automatically run after a new build
  • Test results are automatically transferred from the continuous integration server that executes them (jenkins) to the tool that contains the test suites with high level test cases (testlink)
  • HTML- test reports are made automatically on a scheduled basis, or on demand
  • A readable document of all the test cases can be composed on demand (testlink)
  • A TSH (test scenario/case hierarchy) can be composed automatically


The custom-made things are re-usable by other projects in our company.


Example 2
Gaby Spengler describes how she helps testers find out if a test case has already been implemented:

Sometimes testers don’t know if a test case for a specific requirement has already been implemented. I have written a macro in Excel that allows testers to search the test case directory for a specified keyword and that creates a list of all the test cases that contain the keyword.

Here is how the macro is coded:

'Public full_name As String
Public TFDir As String
Public Searchtext As String
'Public Makro As String
Sub TF_Search()

'The directory to be searched and the search argument are taken from the register „Testcase search“.
TFDir = ""
Worksheets("Testcase search").Activate
TFDir = Cells(2, 1)
Searchtext = Cells(5, 1)

'The following procedure browses thru the test case directory and looks for test cases that contain the Searchtext and lists them in the register “Testcase search”
TFFiles_Search TFDir

End Sub

Sub TFFiles_Search(TFDir)
'
'This procedure browses thru the test case directory and looks for test cases that contain the Searchtext and lists them in the register “Testcase search”
'
Dim LineSource As Long
Dim LineDest As Long

Dim This_Workbook As String
Dim Searched_Workbook As String

Dim WSHShell, fso, oArgs
Dim oFolders, oSubFolder, oFiles, Folder
Dim i, Text, FileX, TCFile()


'
This_Workbook = "Makro Test case search V1.0.xls"
Searched_Workbook = "Testcases INK.xls"
Set WSHShell = CreateObject("WScript.Shell")
Set fso = CreateObject("Scripting.FileSystemObject")
' Set oArgs = Arguments

i = 0
Set oFolders = fso.GetFolder(TFOrdner)
Set oFiles = oFolders.Files

For Each FileX In oFiles
' Test case file names are stored in the array TCFile
Text = Mid(FileX.Name, 1, 9)
If Text = "Testcases" Then
ReDim Preserve TCFile(i)
TCFile(i) = TFDir + "\" + FileX.Name
i = i + 1
End If
Next

' Test cases that contain the sought text are inserted starting with line 10
LineDest = 10

For i = 0 To UBound(TCFile)
'Open the test case file
Workbooks.Open TCFile(i)


Searched_Workbook = TCFile(i)

Length = Len(TCFile(i))
pos1 = 1
Do Until Length = 0
pos1 = InStr(1, Searched_Workbook, "\")
If pos1 = 0 Then
Exit Do
End If
Length = Length - pos1
Searched_Workbook = Mid(Searched_Workbook, (pos1 + 1), Length)

Loop

With Worksheets("Testcases").Range("A2:J1000")
Set c = .Find(Searchtext)
' If the Searchtext is found then the test case is copied to the macro file
If Not c Is Nothing Then
firstAddress = c.Address


' The search goes on until the first hit has been found again
Do
'The line in which the Searchtext was found, is stored
LineSource = c.Row
Worksheets("Testcases").Activate
Rows(LineSource).Select
Selection.Copy
Workbooks(This_Workbook).Worksheets("Testcase_Search").Activate
Rows(LineDest).Select
'Selection.Insert Shift:=xlDown
'Selection.Insert
Selection.PasteSpecial Paste:=xlPasteValues, Operation:=xlNone, SkipBlanks _
 :=False, Transpose:=False
' Rows("10:10").Select
With Selection
.HorizontalAlignment = xlGeneral
.VerticalAlignment = xlTop
.WrapText = True
.Orientation = 0
.AddIndent = False
.IndentLevel = 0
.ShrinkToFit = False
.ReadingOrder = xlContext
.MergeCells = False
End With

LineDest = LineDest + 1

'Workbooks(TCFile(i)).Worksheets("Testcases").Activate
Workbooks(Searched_Workbook).Worksheets("Testcases").Activate
Set c = .FindNext(c)
Loop While Not c Is Nothing And c.Address <> firstAddress
End If
End With
Workbooks(Searched_Workbook).Worksheets("Testcases").Cells(1, 1).Copy
'Close the testcase file
Workbooks(Searched_Workbook).Close SaveChanges:=False
'
Next

' Set Autofilter
Workbooks(This_Workbook).Worksheets("Testcase search").Activate
Rows("9:9").Select
Selection.AutoFilter

End Sub


Example 3
Sergiusz Golec writes:

One day a set of sample clients/partners observed a not desired end state for their transactions. "Hard to find & reproduce". Critical details & triggers - missing. Except the known day. Hard to reproduce & analyze.
Drop from webapp debug logs contained 9 files around 90 MB. Jms message response content with an issue not to be covered with known log analyzers. Every file around 30'000 lines long, 10'486'203 characters long. Interesting details - spreaded among different hours (timestamps), contexts and transactions. As if extremely thin layer of a butter on a 90 MB slice of bread.
Automatically it was half a day task. To write and check the needed outcome.
Script read the input file(s), line by line, and with a function (fWorthOrNotWriting) based upon keywords (like interesting XML tags, Partner ID(s) (shown in a different formats), environment, transaction type, state or known errors) started writing the desired output (into another file). Script collected the entire XML messages into a separate file. 808 lines from around 9*30'000 lines ( = 270 K lines).
808 lines long content manually was trimmed and boiled eventually to the single perfect example: 36-lines-long XML. Alienated & identified issue. Easy to work with and fix by developers.
Manually - it would make my soul cry for a week. To look for that specific issue.


If you have also used this pattern and would like to contribute your experience to the wiki, please go to Feedback to submit your experience or comment.

.................................................................................................................Main Page / Back to Process Patterns / Back to Test Automation Patterns