Debugging, Refactoring and Unit Test

by Kardi Teknomo

In this tutorial, you will learn about debugging in Spyder. You will also about refactoring your Python code and creating your unit test.

Catching Errors

Error in programming code is very common. There are many type of common errors.

• The simplest would be syntax error that can be easily catched by the compiler or the interpreter.
• If you have a division and the value of the denomination is zero, you will get error for division by zero.
• If some of the variable or name are undefied, then you will get name error.
• If you combined different data type such as integer to string you will get type error.

One simple way to catch the error and do something if such error happen is to use

try:

run these statements until it catch an error

except:

run these statements when there is exception error

else:

run these statements when there is no exception error in try clause

finally:

always run these statements regardless you have error or not

When we use try-catch statement, the error can still be printed but the program will not be halted. However, we will not know which line number on the try statement cause the error. During debugging, you can add raise statement in the exception clause to show the error and halt teh program.

# if we run the code below, we will encounter Value error and the program is halted
d=['1','abc']
a, b, c = d

# now we use try and except to remove the error
try:
   a, b, c = d
except Exception as e:
   print(e)
#    raise   # if you uncomment this line, it will show where the program is halted

name 'd' is not defined

def divide(x,y):
    try:
        z=x/y
    except ZeroDivisionError as err:
        z=0
        print("you just divided by zero")
    else:
        print("this line is printed only if no error")
    finally:
        print("result is", z)
          
divide(1,2) 
divide(1,0)

this line is printed only if no error
result is 0.5
you just divided by zero
result is 0

Debugging

What is debugging?

The process of finding where and why error happens in your code is called debugging.

How do we debug?

printing the value

The simplest way to debug without any IDE is to temporarily print the value of variables or return value of function. This simplest technique require you to run the code until it found error. Then you would rely on the compiler message and the printed value to guess what is the error. Debugging can be very frustrating experience for new coders. You can code in a few minutes but debugging might takes many hours and even days. In fact, we can check if a programmer is experience or not by checking their time in debugging, not in coding. The more experience programmer would take less time in debugging.

When the compiler still give you the error message, you should be happy because computer can help you to find the error. Sometimes, your program run smoothly without error but it produces wrong results than what you expect. This kind of logical error is much harder to debug.

using debugger in IDE (Spyder)

Using IDE like Spyder, debugging can be much easier. Simply by setting breakpoint before the error happen, you would still run the code step by step while viewing the values of all variables such that you would know where and why the error happens.

When you see some error in your code, it usually show the module name and the line number.

1. Set breakpoint in the same module, at line number right before the error happen. In Spyder, you can do this through menu Debug > Set/Clear Breakpoint (F12). You will see red code beside the line number if you have set the breakpoint
2. Run the debugger. In Spyder, you go to menu Debug > Debug (CTRL+F5)
3. You can use either one of the following:
   • Step (menu Debug > Step (CTRL+F10)) to run one step at a time of the line code
   • Step In (menu Debug > Step Into (CTRL+F11)) to inside another method or another module
   • Step Out (menu Debug > Step Return (CTRL+SHIFT+F11)) to go out from this method and return the caller
   • Continue (menu Debug > Continue (CTRL+F12)) to go to the next breakpoint
   • Stop (menu Debug > Stop (CTRL+SHIFT+F12)) to stop debugging You can also see the value of variables in Variable Explorer tab while you debug.

Refactoring

What is refactoring?

Refactoring is modifying your code without changing its input and output. The observable external behavior of your code after refactoring would be exactly the same but the internal behavior is improved.

Why do we refactor?

• make your code easier to understand
• change the data structure or the algorithm such that it works faster, consume less memory
• make the simpler design of the software
• make easier to find bugs
• improve the tollerance of teh code to change by abstracting it or make more reusable components
• provide documentations

What do we change to refactor?

• rename the variable, method, module, class based on your coding standard and system metaphor
• change the name that is not so useful into something more meaningful
• change long parameter list into simpler list
• change long if-elif (switch) statements into simpler one
• change loop into list comprehension such that easier to read
• breaking long method into reuseable methods
• breaking large class into more integrated and reusable classes
• abstracting the methods and classes
• removing the global variables by passing the variables through parameters of the methods
• removing the global constants into parameters
• removing duplicate code
• adding comments such that easier to maintain the code

What is the requirement for refactoring?

You can only refactor if you have the unit test ready. The existence of unit test is the guarantee that the code would behave the same way externally (input and output).

How do we refactor?

1. First, you need to run all of the unit tests and pass all tests without error. You can add unit tests if necessary.
2. Then, you refactor to increase understanding of your code
3. After that, you run again the unit test.
4. Next, you add more unit test for the changes and modify the main code if necessary until it passes all tests.
5. Then, you refactor the code until it would be simplest state
6. Run the unit tests

Test Driven Development

The idea of Test-Driven Development is to create the test before you develop your code. After you finish creating the test code, then you start to create the function or the class just to pass the test. Using this simple Agile technique, you keep improving your test cases and refactor your code to pass the test.

The protocol is to do unit test before you code and after you code, the code must pass all the test. Using that protocol, you can modify or refactor your code mercilessly. Then you do the automatic testing. If the refactoring break the code, you know at least it was working before you refactor.

Test-Driven Development is very useful technique especially if you work in a team of developers. Each developer must do the protocol above. Whoever breaks the code must be responsible to fix it. Or else, his/her code would be rejected to be included in the committed code.

Unit Test

Unit test is useful to test your function or your class.

Why do we need unit test?

1. To do refactoring (modifying the code without changing its external behavior)
2. To do debugging
3. To add feature
4. To review the code

Unit test is the requirement for refactoring, debugging and adding features in your code. By spending a little more time to define your unit tests, you would save a lot of time of debugging unnecessary error caused by modifying your code that already work earlier.

How do we make unit test?

To create a unit test in Python is easy.

1. import unittest
2. Create a class of test cases inherited from unittest.TestCase
3. Build test cases of the correct values inherited from unittest
4. Build test cases of the incorrect values inherited from unittest
5. Build the actual code (the function or the class that you want to test)
6. Run the unit test by calling the test class using unittest.main()

import unittest

class MyTestCases(unittest.TestCase):
    def setUp(self):
        self.abc = "abcd"

    def test1(self):
        lst=[]
        for i in self.abc:
            lst.append(i)
        answer=['a','b','c','d']
        self.assertEqual(lst,answer)
        
    def test2(self):
        solution=False
        self.assertFalse(solution)
    
    def test3(self):
        solution=True
        self.assertTrue(solution)
    
if __name__ == '__main__':
    unittest.main(argv=['first-arg-is-ignored'], exit=False)

...
----------------------------------------------------------------------
Ran 3 tests in 0.002s

OK

To give you more concrete example, let us say we want to create a function to compute a factorial of an integer number n, that is the product of 1 2 .... * n.

• We know that if
• n=1, then factorial(1) = 1
• n=2, then factorial(2) = 1 * 2 = 2
• n=3, then factorial(3) = 1 2 3 = 2 * 3 = 6
• n=4, then factorial(4) = 1 2 3 4 = 6 4 = 24
• n=5, then factorial(5) = 1 2 3 4 5 = 24 * 5 = 120
• n=6, then factorial(6) = 1 2 3 4 5 6 = 120 6 = 720

import unittest

class TestFactorial(unittest.TestCase):
    correctCases=(
             (1,1),
             (2,2),
             (3,6),
             (4,24),
             (5,120),
             (6,720)
            )
    
    def testEasyCorrectFactorial(self):
        ''' test for easy correct cases
        '''
        for inp,out in self.correctCases:
            result=factorial(inp)
            self.assertEqual(out,result)
    
    def testFloatValues(self):
        '''
        float values must not be allowed 
        '''
        self.assertRaises(NotIntegerError, factorial, 1.0)
    
    def testZero(self):
        '''
        factorial(0) is defined as 1
        '''
        self.assertEqual(1,factorial(0))
    
    
    def testNegative(self):
        '''
        factorial is not defined in negative input
        '''
        self.assertRaises(OutofRangeError, factorial, -1)
    
    def testOutOfRangeBigInput(self):
        '''
        factorial can handle big input n but not larger than 2500
        '''
        self.assertRaises(OutofRangeError, factorial, 2501)
        
    def testInRangeBigInput(self):
        '''
        factorial can handle big input n beyond set max recursion limit
        '''
        self.assertEqual( factorial(100),93326215443944152681699238856266700490715968264381621468592963895217599993229915608941463976156518286253697920827223758251185210916864000000000000000000000000)

class NotIntegerError(ValueError):
    pass

class OutofRangeError(ValueError):
    pass


def factorial(n):
    import sys
    
    if not isinstance(n,int):
        raise NotIntegerError('Factorial can only accept integer input')
    if not (0<=n<=2500):
        raise OutofRangeError("Factorial is defined only with positive integer input between 1 to 2500")
    
    limit=sys.getrecursionlimit()
    if n>limit:
        sys.setrecursionlimit(n+10)
    
    if n==1 or n==0:
        return 1
    else:
        return factorial(n-1)*n

if __name__=="__main__":
    unittest.main(argv=['first-arg-is-ignored'], exit=False) # parameters is ignored to run in Jupyter

.........
----------------------------------------------------------------------
Ran 9 tests in 0.007s

OK

factorial(100)

93326215443944152681699238856266700490715968264381621468592963895217599993229915608941463976156518286253697920827223758251185210916864000000000000000000000000

type(factorial(250))

int

import sys
sys.getrecursionlimit()

3000

sys.setrecursionlimit(3500)
factorial(2500)

16288884241692635468966810574743966336539994283436659333376117059851739595300666601568118117109111430182218994996706377540737964295726648036084914477398269956576650394995303908153606931358938562424868716863336511787772831963234651490597845804707452080712773761945183179002366243765637991536689969242581709947395573553799155162061020587956162836453609056109182552093352343844029882417375246821954281460020336896525591606956233891343329496954631026393022945474865068966259267963805071707264234749398946807274223651874046023994635224545104061309775665397330572064502645799793490535692439961861758186037617483580487420516854225746700866725272078424896992597788322485750313103767538280635190313055438652113070059895360069459016503698021402127430434703720577454603684221486207712971570279183098247144580669751192292412687570776382442783145813125272512987140013465430577373695416037438604330731495427723748498601316777072913720020200624759285687594697103942902831458433117148104802139150255844954156372702572242931979348640772104241935322544694355717741028042721831057393383946811950229862119018492668601533950515675995793861869111189410513752442848879659001774939446410165714053104744903131715021128531205114521790600044832229285647606408017904177251780563861670452217895698401839016268343830469429772772782341220769473426587820287290019473077524695825215527904355576391305600088839325393721013677844373796989572057534519771031549187963257721208029673279152430652933276800258223453219383978743812269682334913717476068767081112170724712287720561807845229060596372853438939340670348358259624827210411996569765719571305348561907445521649287971976375847487178355765492815778069121838364685540983459992106337314470299659462768807774194455026719275830902631301620632068053005745274643641270818310893189040468508343150208376066332465734970601526332798266648668957684928388346914251393674102236838190309415765024962992701286434254040733064624752399588405701518471706282680092033896216655874206291783633993514147758055661610275976159918807613941637566649034779587069377199455553747637235892554444579113470055333978002998933446236448649956338643549877097069790252117694271543914179639916424071991406456604783333397965866797905100968905477558448660543042454554471492045598502849277515838640500208365860739763710206685971849678108935761798782539066278141381636294637082189768125799193702797967538238466562473387279176788278704807481230413644276139720229104456308083258037763826781395687638241302508020291782679358425712165041212352088250542961656610307562083717426864028254048045585013278396707312988098509307199244525251413018638107871271406375801619527964709310126699327425652342396160313371140810226949214136412603864243886523013717112551532688276164952934427157810894957954046837445796764595217297020162001470343757782370085850953552320637100882919579912163108370028314403969241003234290634568270458955949171264334907057977769908075381921113966351587586648467738374135641552139894953507856890412402614641785184184650269635082032520382461666556052083240749659841927331974627710176727263009232886075400144727578901134034342119214962884370001625512726452523206152157166541752489388503280463130706903614053713733296237361667312991010932983656540560377330832262770042696095731040694487906848645462190989961711109989113241974798068964703059871195609328565827196434230198178800412242371942746686047154961984072073558094313894903724884222066777831669419732898160360633747237482986968369023008889690448245258289105706876230750084254201797244121746320131347525589214486094781766265733538907918016852228868499907315181338394080723321126032440189828823699970328255861187143922082019147768883662612191302509135461511051477630808296519282900741066316050077242544314881058045728870693282326830433019004661600521723836651817381529898440636383917095475899004094206317468376377314153856018840069377218558903334939371343395777264426365318130887683599836088345497158322556553595094840894654614406383376396868199531042942079405403474627428650274595771829568255995466465003325664472293654101201924430539079190443194487046380416281749159026300696294947812447714612418298771332692848211154125929312681287195951419960699391722211424311898584816200565373258799897110855302098610980845978364362685231061369005215230788471798866541648729889538439509890111560301062943923923769988997147648363950708508382034563798637202146673551309938897540440786998003118907883241416258343188784000679367284896011745823289734561911522255979545433492376668285434347783001266108600121473498779217963921384546755156766188951735866744108093964636023471349527019498920540070752394649104708510770999131507315610483632330982876174119389012968417760250231722884408641474340374893362833795392300949100935853949419952854923937872507275773843355108735204114864752466565493366395162830913421138498633186453324268762572668445390770903011717202242312950408723851830364004678106656009078222984968635868416766693788940980240045058068275614302839043567118028698385054274108921811298084233976240006277809181325997330000786683674194376654268377368369949823431206244095885960702059149846318232687179678572748082487793549844548187548311400556064744050666472476457947299489568330636742840865513804654776149494225932236964194164784786218640251626514525549920894810545056499250214652147224638246034892251401029945773476251196919612592752826016419187902895630887737745713281219681141023586882321633118519558748181198803719862314967759867112077291645583600510531017148754676649505951293767894337326791942862414631993126449392270145239108425380056269282654566300820253116701666036949374826952156733286262700190206665725355635411559887439781283350874285109852181238143636938256407098028863909426094536431398247852543854757537942646791855344906017659039384905122375234004320416026882823286655632286053156546748028668750196973803337027786891236116711910780819707846081218679760982625290715984522150204916425232591826590887410401320568045430558979943145617193692059705465421939829497586908594951642657711985208269462425640322372314303330475772671495014049400654155619566942191085558759378416790160458835621823599743742665152596726797511419955382867531134959437002011587472231152746047820309830039699792147174883667855760287780241438038062954704157981072533480950728570115662921849376586317866043009133972729549191452640224225320222817060812601144385749818368781373067888827129923654130858902252804622762751192618314790547633755929057344102888830740499275949997990026813425216747458578654838647236496508876074530642629714199563056483975900965902520059913821418901761372054615449749487163282905699188377895083782877246773269374715942460000414707728522199826307191493572953655749106244370767768017669677808468909623091993519298870943235516530358122980923396832082235382703392292864000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

References

• Refactoring Python Applications for Simplicity
• Using Python unittest in IPython or Jupyter

last update: April 2021

Cite this tutorial as: Teknomo, K. (2021) Debugging, Refactoring and Unit Test using Python (http://people.revoledu.com/kardi/tutorial/Python/)

See Also:

• Python for Data Science list of many python tutorials

Visit www.Revoledu.com for more tutorials in Data Science

Copyright © 2021 Kardi Teknomo

Permission is granted to share this notebook as long as the copyright notice is intact.