Handling a File Error
Description
As already pointed out in the exceptions lesson, Python contains a vast set of built-in exceptions. One important class has to do with file handling. The example presented in the lesson will help to review file handling methods and put them in the context of handling exceptions.
Table of contents
Learning (not) to Handle Exceptions
Learn how to deal with exceptions in Python
When you develop code, it is almost impossible not to run into an error. Some problems are going to arise as soon as you start your program, for example, if you forgot to close a parenthesis, or forgot the :
after an if-statement. However, errors at runtime are also very frequent and harder to deal with. In this article, you are going to learn how to handle exceptions, i.e. how to avoid program crashes when you can anticipate that an error may appear.
We are going to cover from the basics of error handling to defining your own exceptions. You will learn why sometimes it is better not to catch exceptions and how to develop a pattern that can be useful for future users of your code. Exceptions are a crucial part of any code, and dealing with them elegantly can improve a lot the value of your code.
As always, you can check the example code for this article and the original text in case you have any suggestions to improve it.
Source: Aquiles Carattino, https://www.pythonforthelab.com/blog/learning-not-to-handle-exceptions/
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 License.
Try ... Except
Imagine that you are reading data from a file. You can simply write something like this:
If you run the code, you will see a message like the following:
When an error is raised, the code that follows will not be executed. That is why you don't see the statement Data loaded
on the console. This is a somewhat innocent problem but imagine that you are communicating with a device. If there is an error in your program, you won't have the chance to close the communication with the device, or you won't be able to close a shutter to prevent damages to the detectors, etc.
Dealing with this kind of errors is normally done within a try
/ except
block. This means that if there is an error inside the try
, the block except
will be executed. In the example above, we can do the following:
If you run the code, you will see a nice message being printed to screen saying Data not loaded
. This is great! Now our program is not crashing and we can close the shutter, or stop the communication with our device. However, we don't know the reason why the data was not loaded.
Before continuing, create an empty file called my_file.dat, and run the script again. You will see that data is not being loaded, regardless whether the file exists or not. With this trivial example, you are seeing the risks around unspecific except
blocks. If we make a simpler script:
The output will be:
Which is telling us that the problem is the method that we tried to use, readfile
doesn't exist. When you use a plain try/except
block, you are sure you are handling all possible exceptions, but you have no way of knowing what actually went wrong. In simple cases like the one above, you have only two lines of code to explore. However, if you are building a package or a function, some errors will propagate downstream, and you don't know how they are going to affect the rest of the program.
For you to have an idea of the importance of correct handling of errors, I will tell you what I have witnessed using a program that ships with a very sophisticated lab instrument. The program that controls the Nano Sight has a very nice user interface. However, when you are saving data, if the filename you choose has a dot in it, the data will not be saved. Sadly, the data will also be lost and the user will never know that the problem was having a simple .
in the filename.
Handling all possible errors in a graceful way is very complicated and sometimes almost impossible. However, you can see that even the software that ships with very expensive instruments (in this case I mean instruments with a price tag similar to a small apartment), also has to deal with all kinds of exceptions, and not always in the most user-friendly way.
Catching Specific Exceptions
The proper way of handling exceptions in Python is to specify what exception are we expecting. In this way, we know that if the problem is that the file doesn't exist, we can create it, while if the problem is of a different nature, it will be raised and displayed to the user. We can alter the above examples like this:
If you run the script, and the file my_file.dat doesn't exist, it will print to screen that the file doesn't exist and the program will keep running. However, if the file does exist, you will see the exception with readfile
. Of course, you are not limited to printing a message when an exception happens. In the case of the non-existing file, it is easy to create one:
If you run the script once, you will see that the file is being created. If you run the script a second time, you will see the exception with the readfile
method. Imagine that you don't specify which exception you are catching, and you have the following code, what will happen when you run it?:
If you look carefully, you will realize that even if the file my_file.dat exists, an exception is going to be raised because of the readfile
method. Then the except
block is going to be executed. In this block, the program is going to create a new my_file.dat, even if it already existed, and therefore you are going to lose the information stored in it.
Re-raising Exceptions
A very common scenario is that when an exception appears, you want to do something but then raise the same exception. This is a very common case when writing to a database or to different files. Imagine the case where you are storing information in two files, in the first one you store spectra and in the second one the temperature at which you acquire each one. You first save the spectra and then the temperature, and you know that each line on one file corresponds to one file on the second file.
Normally, you save first a spectrum and then you save the temperature. However, once in a while, when you try to read from the instrument, it crashes and the temperature is not recorded. If you don't save the temperature, you will have an inconsistency in your data, because a line is missing. At the same time, you don't want the experiment to keep going, because the instrument is frozen. Therefore, you can do the following:
What you can see here is that we try to read the temperature and if anything happens, we will catch it. We remove the last line from our data file, and then we just call raise
. This command will simply re-raise anything that was caught by the except
. With this strategy, we are sure that we have consistent data, that the program will not keep running and that the user will see all the proper information regarding what went wrong.
Exceptions in Exceptions
Imagine that the code is part of a larger function, responsible for opening a file, loading its contents or creating a new file in case the specified filename doesn't exist. The script will look the same as earlier, with the difference that the filename is going to be a variable:
To run the code above, the only thing you have to do is to specify the filename before, for instance:
If you run this code, you will notice that it behaves exactly as expected. However, if you specify an empty filename:
You will see a much longer error printed to screen, with one important line:
If you look carefully at the error, you will see that it outputs information regarding that an error occurred while the code was already handling another error. This is, unfortunately, a common situation, especially when dealing with user input. The way around it would be to nest another try/except block or to verify the integrity of the inputs before calling open
.
Several Exceptions
So far we have been dealing with only one possible exception, FileNotFoundError
. However, we know that the code will raise two different exceptions, the second one being an AttributeError
. If you are not sure about which errors can be raised, you can generate them on purpose. For instance, if you run this code:
You will get the following message:
The first string is the type of exception, AttributeError
, while the second part is the message. The same exception can have different messages, which describe better what has happened. What we want is to catch the AttributeError
, but also we want to catch the FileNotFound
. Therefore, our code would look like this:
Now, you are dealing with several exceptions. Remember that when an exception is raised within the try
block, the rest of the code will not be executed, and Python will go through the different except
blocks. Therefore, only one exception is raised at a time. In the case where the file doesn't exist, the code will deal only with the FileNotFoundError
.
Of course, you can also add a final exception to catch all other possible errors in the program, like this:
In this case, if the file exists but it is empty, we are going to have a problem trying to access data[0]
. We are not prepared for that exception and therefore we are going to print a message saying Unhandled exception. It would be, however, more interesting to let the user know what exception was actually raised.
We can do the following:
Which will output the following message:
The exception also has a type
, which you can use. For example:
Which will print the first letter of Information
, i.e. I
. The pattern above has a very important drawback, and is that important_data
may end up not being defined. For example, if the file my_data.dat doesn't exist, we will get another error:
The else Block
To prevent what we just saw in the previous section, we can add one more block to the sequence: finally
. This block is always going to be executed, regardless of whether an exception was raised or not. For example:
This is, in the end, a very silly example, because we are setting important_data
to a special value, but I hope you can see the use of finally
. If there is something that you must absolutely be sure that is executed, you can include it in a finally statement.
finally
is very useful to be sure that you are closing a connection, the communication with a device, closing a file, etc. Generally speaking, releasing the resources. Finally has a very interesting behavior, because it is not executed always at the same moment. Let's see the following code:
First, run the code when the file my_data.dat doesn't exist. You should see the following output:
So, you see you went from the try
to the except
to the finally
. If you run the code again, the file will exist, and therefore the output will be completely different:
What you can see here is that when an unhandled exception is raised, the first block to be executed is the finally
. You close the file immediately. And then, the error is re-raised. This is very handy because it prevents any kind of conflict with downstream code. You open, you close the file and then the rest of the program has to deal with the problem of the IndexError
. If you want to try a program without exceptions, just write something into my_data.dat and you will see the output.
The Traceback
There is only one more block to discuss in the exception handling pattern, the else
block. The core idea of this block is that it gets executed if there are no exceptions within the try
block. Is very easy to understand how it works. You could, for example, do the following:
The most difficult part of the else block is understanding its usefulness. In principle, the code that we have included in the else
block could have also been placed right after opening the file, as we have done earlier. However, we can use the else
block to prevent catching exceptions that do not belong to the try
. It is a bit of a far-fetched example, but imagine that you need to read a filename from a file and open it. The code would look like this:
Since we are opening two files, it may very well be that the problem is that the second file doesn't exist. If we would put this code into the try
block, we would end up triggering the except
for the second file even if we didn't mean to. At first, it is not obvious the true use of the else
block, but it can be very useful and therefore it is important that you are aware that it exists.
Of course, it is possible to combine everything that you have learned so far:
You are very encouraged to play around and try to find different usages for each block. If you have worked enough with Python, you probably encounter plenty of exceptions that force you to re-run your script from the beginning. Now you know that there may be workarounds. A great resource, as almost always, is the Python Documentation on Exceptions.
We're not done yet. There are many more things that can be done with exceptions.
Raising Custom Exceptions
When you are developing your own packages, it is often useful to define some common exceptions. This gives a great deal of flexibility because it allows other developers to handle those exceptions as they find appropriate. Let's see an example. Imagine that you want to write a function that calculates the average between two numbers, but you want both numbers to be positive. This is the same example that we have seen when working with decorators. We start by defining the function:
And now we want to raise an Exception
if either input is negative. We can do the following:
If you try it yourself with a negative input, you will see the following printed:
Which is great, it even points to the line number with the issue, etc. However, if you are building a module and you expect others to use it, it would be much better to define a custom Exception, that can be explicitly caught. It is as easy as this:
An exception is a class, and therefore it should inherit from the general Exception
class. We don't really need to customize anything at this stage, we just type pass
in the body of the class. If we run the code above with a negative value, we will get:
If you want to catch that exception in downstream code, you will do it as always. The only difference is that custom exceptions are not available by default and you should import them. For example, you would do the following:
If you have worked long enough with packages, probably you have already encountered a lot of different exceptions. They are a great tool to let the user know exactly what was wrong and act accordingly. Sometimes we can be prepared for some exceptions and is very appreciated when custom ones are included into the package and not just a generic one that forces us to catch any exception, even if it is something that we were not actually expecting.
Best Practices for Custom Exceptions
When you are developing a package, it is very handy to define exceptions that are exclusive to it. This makes it much easier to handle different behaviors and gives developers a very efficient way to filter whether the problems are within your package or with something else. Imagine, for instance, that you are working with a complex package, and you want to write to a file every time an exception from that specific package appears.
This is very easy to achieve if all the exceptions inherit from the same base class. The code below is a bit longer, but it is built on top of all the examples above, so it should be easy to follow:
We first define an exception called MyException
, which is going to be our base exception. We then define two errors, NonPositiveIntegerError
and TooBigIntegerError
which inherit from MyException
. We define the function average
again but this time we raise two different exceptions. If one of the numbers is negative or larger than 10.
When you see the different use cases below, you will notice that in the try/except
block, we are always catching MyException
, but not one of the specific errors. In the first two examples, when passing -1
and 11
as arguments, we successfully print to screen the error message, and the program keeps running. However, when we try to calculate the average between two letters, the Exception
is going to be of a different nature, and is not going to be caught by the Except
. You should see the following on your screen:
Adding Arguments to Exceptions
Sometimes it is handy to add arguments to exceptions in order to give a better context to users. With the example of the average, let's first define a more complex exception:
What you can see is that the exception takes two arguments, x
and y
and it generates a message based on them. They can be both negative or only one of them is negative. It doesn't only give you that information, but it actually displays the value that gave problems. This is very handy to understand what went wrong exactly. The most important part is at the end of the class: the __str__
method. This method is responsible for what appears on the screen when you do print(e)
in the except
block. In this case, we are just returning the message generated within the __init__
, but many developers choose to generate the message in this method, based on the parameters passed at the beginning.
Conclusions
Exceptions are something nobody wants to see but they are virtually unavoidable. Maybe you try to read a file that doesn't exist, the user of your code has chosen invalid values, the matrix you are analyzing has different dimensions than expected, etc. Handling exceptions is a sensitive topic because it can lead to even more problems downstream. An Exception is a clear message that there is something wrong going on and if you don't fix it properly, it is going to become even worse.
Handling exceptions can help you to avoid having inconsistent data, not closing resources such as devices, connections or files, etc. However, not handling exceptions correctly can lead to even more problems later on. The try/except
block is very handy when you know what kind of exceptions can appear and you know how to handle them. Imagine you are performing several steps of a complex operation, like writing to a database. If an error happens, you can revert all the changes and avoid inconsistencies.
As with almost any other Python topic, the best way to learn is to look closely at other's code and judge by yourself. Not all packages define their own exceptions, nor handle them in the same way. If you are looking for inspiration, you can see the errors of Pint, a relatively small package, or the exceptions of Django, a much more complex package.
Remember, that both the code and the text of the article is available, in case you have any comments or suggestions to improve it.
https://learn.saylor.org/mod/book/tool/print/index.php?id=29251
Last updated