All in One View

Content from Python Basics Recap

Last updated on 2026-10-08 | Edit this page

Overview

Questions

  • Python Refresher

Objectives

  • Understanding key concepts in Python

This course follows on from the python introduction course. To ensure that we are starting from similar positions, there follows a short multiple choice quiz on key python concepts that we will be building on through this course.

Variables and Lists

Challenge

1. Assigning a value to a variable

We wish to store the string Cat as a value in the variable animal, which of these lines of code will do this for us?

  1. animal = 'Cat'
  2. animal = Cat
  3. Cat = animal
  4. animal(Cat)

Answer 1 is correct

Challenge

2. Assigning values to a list

We wish to create a list of values, which of these lines of code is valid to do this?

  1. varlist = [34, 57, '2d']
  2. varlist = (12, 'vr', 95)
  3. varlist = 'xcf', 12, 97

Answer 1 is correct

Challenge

3a. Indexing characters in a string

Lists and strings both contain multiple indexed values (in the case of strings these are specifically individual characters rather than other values). If we have a variable animal which contains the string penguin, which of these options will print the first character (p) for us?

  1. print(animal[0])
  2. print(animal[1])
  3. print(animal['p'])

Answer 1 is correct

Challenge

3b. Indexing characters in a string (slicing)

We can also select whole sections of lists and strings, not just single elements. Using the same variable animal, containing the string penguin, as above, which of these options will print the last 2 characters for us? (Note that there is more an one correct answer)

  1. print(animal[5:])
  2. print(animal[6:])
  3. print(animal[6:7])
  4. print(animal[5:7])
  5. print(animal[-2:])
  6. print(animal[:-2])

Answers 1, 4, and 5 are correct

Loops

Challenge

4. Constructing a for loop

Please write a simple for loop which will print out each of the characters in the animal variable one at a time.

PYTHON 

for char in animal:
    print(char)

Software Modules

Challenge

5. Loading new functions

We want to use the functions in the numpy library in our code. How do we open this library in our code?

  1. import numpy
  2. load numpy
  3. open numpy

Answer 1 is correct

If statements and conditionals

Challenge

6. Conditionals

Which of these conditional tests returns a True result?

  1. 4 > 3
  2. 'a' != 'b'
  3. 6 >= 6.0
  4. '3' == 3
  5. 3 > 'c'

Answers 1, 2, and 3 return True results. 4 returns False. 5 raises an Error.

Challenge

7. If statements

What is printed when this if statement is used?

PYTHON 

if 4 > 5:
    print('A')
elif 4 <= 5:
    print('B')
elif 4 < 5:
    print('C')
else:
    print('D')
  1. A
  2. B
  3. C
  4. B and C
  5. D
  6. A and D

B. Both 4 <= 5 and 4 < 5 would return a True result, but the if statement is exited as soon as the True result is returned.

Callout

Formatting Variables and Objects in Strings

Throughout this course, we will be printing the values of variables inside strings to demonstrate what the code does. There are a number of ways to this in Python; in this course we will be using f-strings which are the recommended string interpolation syntax from version 3.6 onwards.

To use f-strings simply start a string literal with f or F, then embed whatever you want interpolating into the string within curly braces {}:

PYTHON 

two_int = 2
two_str = "2"
f_string = f"{two_int} + {two_str} = {2 + 2}"
print(f_string)

OUTPUT 

2 + 2 = 4

Numerical values can be formatted using : inside the curly braces. For example, an integer i can be made to have a width n using {i:nd}, or {i:0nd} to pad it out with zeros:

PYTHON 

print(f"One with four leading spaces {1:5d}")
print(f"One with four leading zeros {1:05d}")

OUTPUT 

One with four leading spaces     1
One with four leading zeros 00001

Floating point variables can have their precision specified with a number after a decimal point and an f:

PYTHON 

pi = 3.141592653589793
print(pi)
print(f"pi to three decimal places is {pi:.3f}")

OUTPUT 

3.141592653589793
pi to three decimal places is 3.142

Or made to use scientific notation with an e:

PYTHON 

print(1e9)
print(f"A billion in scientific notation is {1e9:.1e}")

OUTPUT 

1000000000.0
A billion in scientific notation is 1.0e+09

This is only scratching the surface of what you can do with f-strings, they are often the most powerful and concise way to format variables inside strings. However, there are occasions when other methods are preferable, and you should be careful using them with Python before version 3.12.

To learn more, see: Python f-strings.

Generative AI

We would like to reiterate the Carpentries stance on Generative AI in coding which you may have seen when preparing for this course.

To summarise the position, the recommendation is that you avoid getting help from generative AI during the workshop for several reasons:

  • For most problems you will encounter at this stage, help and answers can be found among the first results returned by searching the internet or looking on e.g. StackOverflow.
  • The foundational knowledge and skills you will learn in this lesson by writing and fixing your own programs are essential to be able to evaluate the correctness and safety of any code you receive from online help or a generative AI chatbot. If you choose to use these tools in the future, the expertise you gain from learning and practising these fundamentals on your own will help you use them more effectively.
  • As you continue programming, the mistakes you make will be the kinds that have also been made – and overcome! – by everybody else who learned to program before you. Since these mistakes and the questions you are likely to have at this stage are common, they are also better represented than other, more specialised problems and tasks in the data that was used to train generative AI tools. This means that a generative AI chatbot is more likely to produce accurate responses to questions that novices ask, which could give you a false impression of how reliable they will be when you are ready to do things that are more advanced.
Key Points
  • Assign values to variables using =
  • Generate lists using square brackets []
  • Use indexes inside [], starting at 0, to select characters from strings and items from lists
  • Use for to loop through items in iterable objects
  • Make conditional expressions using ==, !=, >, <, >= and <=
  • Use f'{}' to embed formatted variables inside strings
  • Avoid use of generative AI for this course

Content from Dictionaries

Last updated on 2026-10-08 | Edit this page

Overview

Questions

  • How can I work with relational datasets?
  • How can I access remote data directly in my scripts?

Objectives

  • Learn about JSON format
  • How to create dictionaries
  • Loading JSON data and working with it
  • Using web API’s for accessing remote data

In previous lessons we have learnt about lists and numpy multi-dimensional arrays, which are designed for working with structured, tabular, datasets. But much of the data that we use in our day to day lives, such as that in data catalogs or the communications between modern digital services, does not fit nicely into these tabular datasets. Instead unstructured data formats, which use labels or ‘keys’ to identify each data object are needed. One of the most common formats for such data is the JavaScript Object Notation (JSON) file format. This format was originally developed to fulfil the need for a self-contained, flexible format for real-time server-to-browser communication, and is now used as the basis for many unstructured data formats.

One example of such usage in research is the storage of metadata for data, programs, workflows, or any other such object in a Research Object Crate (RO-Crate). These metadata records take the form:

JSON 

{
  "@context": "https://w3id.org/ro/crate/1.1/context",
  "@graph": [
    {
      "@id": "ro-crate-metadata.json",
      "@type": "CreativeWork",
      "about": {
        "@id": "./"
      },
      "conformsTo": {
        "@id": "https://w3id.org/ro/crate/1.1"
      }
    },
    {
      "@id": "./",
      "@type": "Dataset",
      "mainEntity": {
        "@id": "tracking_workflow.ga"
      },
      "hasPart": [
        {
          "@id": "tracking_workflow.ga"
        },
        {
          "@id": "object_tracking_pipeline.png"
        }
      ],
      "author": [

      ],
      "provider": [
        {
          "@id": "#project-1"
        }
      ],
      "license": "Apache-2.0",
      "sdPublisher": {
        "@id": "#person-1"
      },
      "sdDatePublished": "2021-01-01 00:00:00 +0000"
    },
    {
      "@id": "#galaxy",
      "@type": "ComputerLanguage",
      "name": "Galaxy",
      "identifier": {
        "@id": "https://galaxyproject.org/"
      },
      "url": {
        "@id": "https://galaxyproject.org/"
      }
    },
    {
      "@id": "#project-1",
      "@type": "Organization",
      "name": "Science Workflows"
    },
    {
      "@id": "#person-1",
      "@type": "Person",
      "name": "Alice Smith"
    }
  ]
}

The layout of this object is organised using key:value pairs, where the key is a unique string, and the value can be any data type, including other data structures. This simple layout allows quite complex data objects to be constructed.

Dictionary

In python this structure is implemented using the ‘dictionary’ object. Below we will go through the principles of creating and working with these objects. Then we will introduce a library for working with JSON files.

Creation

Lists are created by using square brackets [ ].

Dictionaries are created by using curly brackets { }, e.g.:

PYTHON 

d = {}

The simplest way to create a dictionary with some value is:

PYTHON 

d = {'keyname': 'keyvalue'}

Following the previous example, we can create a python dictionary using the name of a person as the key and their age as the value:

PYTHON 

d = {'alice': 35, 'bob': 18}
print(d)

OUTPUT 

{'alice': 35, 'bob': 18}

Alternatively, a dictionary object can be created using the dict function, in a similar manner to using the list function. When using the dict function we need to indicate which key is associated with which value. This can be done in a number of ways, firstly with tuples:

PYTHON 

d2 = dict([('alice', 35), ('jane', 24), ('bob',18)])

or with direct association:

PYTHON 

d3 = dict(bob=18, alice=35, jane=24)

or using the special zip function, which can be used to create a set of tuples from the given iterable lists:

PYTHON 

d4 = dict(zip(['jane','alice','bob'],[24,35,18]))

Accessing elements

To access an element of the dictionary we must use the key:

PYTHON 

print(f'The age of alice is: {d["alice"]}')

OUTPUT 

The age of alice is: 35

We can also use a variable to index the dictionary:

PYTHON 

key = 'alice'
print(f'The name of the person is used as key: {key}')
print(f'The value associated to that key is: {d[key]}')

OUTPUT 

The name of the person is used as key: alice
The value associated to that key is: 35

Adding and deleting elements

Adding an element to a dictionary is done by creating a new key and attaching a value to it.

PYTHON 

print(f'Original dictionary: {d}')
d['jane'] = 24
print(f'New dictionary: {d}')

OUTPUT 

Original dictionary: {'alice': 35, 'bob': 18}
New dictionary: {'alice': 35, 'bob': 18, 'jane': 24}

To add one or more new elements we can also use the update method:

PYTHON 

d_extra = {'tom': 54, 'david': 87}

d.update(d_extra)
print(f'Updated dictionary: {d}')

OUTPUT 

Updated dictionary: {'alice': 35, 'bob': 18, 'jane': 24, 'tom': 54, 'david': 87}

To delete an element, use the del method:

PYTHON 

del d['tom']
print(f'Dictionary with item deleted: {d}')

OUTPUT 

Dictionary with item deleted: {'alice': 35, 'bob': 18, 'jane': 24, 'david': 87}

Alternatively, the pop function can be used to take an element out of a dictionary and assign it’s value to a new variable:

PYTHON 

david_age = d.pop('david')
print(f'Age of David: {david_age}')
print(f'Dictionary with david popped out: {d}')

OUTPUT 

Age of David: 87
Dictionary with david popped out: {'alice': 35, 'bob': 18, 'jane': 24}
Callout

Dictionary Concatenate Warning

Unlike lists it is not possible to use the + operator to concatenate dictionaries:

PYTHON 

{'alice': 35} + {'bob': 18}

OUTPUT 

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-39-a6305e6df312> in <module>
----> 1 {'alice': 35} + {'bob': 18}

TypeError: unsupported operand type(s) for +: 'dict' and 'dict'
Callout

Key Uniqueness Warning

Keys have to be unique; you cannot have two keys with the same name. If you try to add an item using a key already present in the dictionary you will overwrite the previous value.

PYTHON 

print(f'Original dictionary: {d}')
d['alice'] = 12
print(f'New dictionary: {d}')

OUTPUT 

Original dictionary: {'alice': 35, 'bob': 18, 'jane': 24}
New dictionary: {'alice': 12, 'bob': 18, 'jane': 24}

Equality between dictionaries

To be equal, all the elements which compose the first dictionary must be present in the second, and only those elements.

The position (ordering) is not important.

PYTHON 

d1 = {'alice': 12, 'bob': 18, 'jane': 24, 'tom': 54, 'david': 87}
d2 = {'tom': 54, 'david': 87}
d3 = {'bob': 18, 'alice': 35, 'jane': 24}
d4 = {'alice': 35, 'bob': 18, 'jane': 24}
print(f'Dictionary 1 and dictionary 2 are equal: {d1 == d2}')
print(f'Dictionary 1 and dictionary 3 are equal: {d1 == d3}')
print(f'Dictionary 3 and dictionary 4 are equal: {d3 == d4}')

OUTPUT 

Dictionary 1 and dictionary 2 are equal: False
Dictionary 1 and dictionary 3 are equal: False
Dictionary 3 and dictionary 4 are equal: True

Splitting out keys and values

Dictionaries have some special methods. Two of the most useful are keys and values. These return the keys and the values of the dictionary respectively.

PYTHON 

d.keys()

OUTPUT 

dict_keys(['alice', 'bob', 'jane'])

PYTHON 

d.values()

OUTPUT 

dict_values([12, 18, 24])

Note that the dict_keys and dict_values objects are iterable but are not lists. This means that they can be used somewhere like a for loop but you can not index them directly.

PYTHON 

d.values()[0]

OUTPUT 

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'dict_keys' object is not subscriptable

If you want to index keys or values directly, you can convert them to lists with the list function.

PYTHON 

list(d.values())[0]

OUTPUT 

12

It is also possible to iterate through the keys and items in the dictionary at the same time using the items function, which returns a dict_items object containing key, value pairs:

PYTHON 

d.items()

OUTPUT 

dict_items([('alice', 35), ('bob', 18), ('jane', 24)])

This is very useful when using a dictionary in a for loop:

PYTHON 

for key, value in d.items():
    print(f'Name: {key}, Age: {value}')

OUTPUT 

Name: alice  Age: 35
Name: bob  Age: 18
Name: jane  Age: 24

Presence (or not) of an element inside a dictionary

It is possible to test if a key is present in the dictionary (or not) using the keyword in, just as we did at the start of this lesson for values within a list:

PYTHON 

'alice' in d

OUTPUT 

True

PYTHON 

'mark' in d

OUTPUT 

False

Note, however, that we can’t directly test for the presence of values:

PYTHON 

12 in d

OUTPUT 

False

Instead we would have to use the values method to search these:

PYTHON 

12 in d.values()

OUTPUT 

True

JSON files

Because JSON files are such a widely used format, python has a built in package for working with JSON, called json: This package provides the method json.load() to read JSON data from a file and and convert it to a python dictionary:

PYTHON 

import json

with open('data/ro-crate-metadata-1.json') as f:
    data = json.load(f)

The closely related method json.loads() (s for “string”) reads a string containing JSON and turns it into a Python dictionary:

PYTHON 

json_string = '{"numbers": [1, 2, 3]}'
d = json.loads(json_string)
d['numbers']

OUTPUT 

[1, 2, 3]

HTTP requests

Although lot of information is available on the internet for general use, without automated tools for accessing this data it is difficult to make full use of it. Python has a number of libraries for making HTTP requests, to help with this automation, of which the requests library is the most commonly used. This library provides a streamlined application process interface (API) for carrying out these tasks, and has built in JSON support, for easy digesting of the retrieved data.

The basic interaction for making a HTTP request is:

PYTHON 

import requests

source_url='https://api.datacite.org/dois/10.48546/workflowhub.workflow.56.1'
requests.get(source_url)

OUTPUT 

<Response [200]>

The HTTP request returns a response code - a value of 200 indicates the request was successful. There are a wide range of possible response codes. Those starting as 2XX generally indicate success, whereas those starting with 4XX indicate a failure of some sort (including the most common: 404 Not Found).

The HTTP request we made returned more than just the response code, there will also be the attached content that we requested. In this case our request was made to an API which returns citation information associated with the DOI 10.48546/workflowhub.workflow.56.1. Rather than being presented as a complex webpage, this information is returned as a machine-readable string, similar to the JSON file we read earlier, so we can read this in a similar manner:

PYTHON 

response = requests.get(source_url)
record = response.json()

Once the data is in a dictionary we can start exploring it - first step is to check the keys available:

PYTHON 

record.keys()

OUTPUT 

dict_keys(['data'])

The upper level of the dictionary is simply data - so we can move to the second level:

PYTHON 

record['data'].keys()

OUTPUT 

dict_keys(['id', 'type', 'attributes', 'relationships'])

The id contains the DOI that we used to find this entry, while the attributes contains the metadata for the object referred to by the DOI. By digging further into the dictionary we can extract information about the object.

Challenge

Find the title

What is the path to find the title of the object?

PYTHON 

record['data']['attributes']['titles'][0]['title']
Challenge

Find the title (part 2)

Assuming that all JSON objects returned by this API follow the same layout as this record, write a simple function that will return the title of any DOI it is given.

Test that your function works using the DOI: 10.5281/zenodo.4416028

PYTHON 

def doi_title( doi_string ):
    source_url = f'https://api.datacite.org/dois/{doi_string}'
    response = requests.get(source_url)
    record = response.json()
    return(record['data']['attributes']['titles'][0]['title'])

Testing the function:

PYTHON 

doi_title('10.5281/zenodo.4416028')

OUTPUT 

'Britain Breathing 2016-2019 Air Quality and Meteorological Dataset'

Speed Tests…

Sequences are a great tool but they have one big limitation. The execution time to find one specific value inside is linear, as can be shown by fruitless searches for the string x within increasingly long lists of integers.

PYTHON 

short_list = list(range(100_000))
long_list = list(range(1_000_000))

We use the built-in %timeit function, to test the speed of these searches:

PYTHON 

%timeit -n100 'x' in short_list

OUTPUT 

2.02 ms ± 529 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

PYTHON 

%timeit -n100 'x' in long_list

OUTPUT 

17.4 ms ± 1.35 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)

Note that the increase in search time is (very roughly) linear.

This is a real problem because the membership test is a very useful and common procedure. So we would like to have something which is not dependent on the number of elements.

Challenge

Testing access time for large dictionaries

Create two dictionaries, one with 100,000 key:value pairs, the other with 1,000,000 key:value pairs, using the lists created at the start of this lesson. Then use these to test the access times for dictionaries using the %timeit function. How do the access times compare with those for the lists, are they quicker or slower, and do the access times scale linearly with the size of the dictionary?

The dictionaries can be created using the zip method:

PYTHON 

short_hash = dict(zip(short_list,short_list))
long_hash = dict(zip(long_list,long_list))

The access times for the dictionaries are 100-1,000 times faster than for lists, and the search time does not increase with dictionary size.

BASH 

%timeit -n100 'x' in short_hash

OUTPUT 

34.2 ns ± 2.31 ns per loop (mean ± std. dev. of 7 runs, 100 loops each)

BASH 

%timeit -n100 'x' in long_hash

OUTPUT 

48.4 ns ± 5.39 ns per loop (mean ± std. dev. of 7 runs, 100 loops each)

Note: it is likely that your first report back from the test will look like this:

OUTPUT 

The slowest run took 308.65 times longer than the fastest. This could mean that an intermediate result is being cached.
2.52 µs ± 5.91 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

This is because the secret behind the speed of searching a dictionary is the caching of the keys after the first access to them. Running the tests a second time will give above test results.

This caching behaviour is very useful for datasets for are accessed regularly.

Key Points
  • JSON is a simple, easy to interpret format for unstructured data
  • Dictionaries are defined using key:value pairs
  • Dictionaries can be nested, and mixed with lists
  • Web API’s can be accessed using the requests library

Content from Defensive Programming

Last updated on 2026-10-08 | Edit this page

Overview

Questions

  • How can I catch errors in my data?
  • How can I make best use of error messages?

Objectives

  • Learn how to catch and deal with errors within your code.
  • Learn how to catch problematic data before it is used in your code.

In this episode we are going to learn about error handling, a python structure (also common in other programming languages) which will help us to properly manage the anomalies that can be encountered when executing our code.

In science most developers write code for themselves and do not feel the need to use error handling. However, nowadays scientists have to share their work as more funding agencies are also asking for the code used in experiments to be published.

More importantly, applications are not expected to crash and we are going to learn how to avoid this non desirable behaviour.

Challenge

What is the problem with this test?

Please look at the following code. Can you find the fundamental problem in this test?

PYTHON 

val = 1

if val > 0:
    print(f'Value: {val} is positive.')
elif val == 0:
    print(f'Value: {val} is zero.')
else:
    print(f'Value: {val} is negative.')

The test assumes that val is a number, and throws an uncontrolled error if it is not.

PYTHON 

val = 'a'

if val > 0:
    print(f'Value: {val} is positive.')
elif val == 0:
    print(f'Value: {val} is zero.')
else:
    print(f'Value: {val} is negative.')

OUTPUT 

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-99c0e25bf5e9> in <module>()
      1 def check_sign(val):
----> 2     if val > 0:
      3         print(f'Value: {val} is positive.')
      4     elif val == 0:
      5         print(f'Value: {val} is zero.')

TypeError: '>' not supported between instances of 'str' and 'int'

We can avoid problems like this by wrapping our code in an if statement. To make things simpler, we will first write the test as a function:

PYTHON 

def check_sign(val):
    if val > 0:
        print(f'Value: {val} is positive.')
    elif val == 0:
        print(f'Value: {val} is zero.')
    else:
        print(f'Value: {val} is negative.')

Then wrap the function call in an if statement:

PYTHON 

if type(val) is int or type(val) is float:
    check_sign(val)
else:
    print('val is not a number')

However, this becomes tedious to implement very quickly, especially when trying to cover a number of possible errors, which will all need their own if clauses. This can lead to a lot of duplicated code, making the code difficult to maintain.

Python provides the try-except structure to avoid this issue, enabling developers to properly manage any possible errors. The basic try-except structure is:

PYTHON 

try:
    check_sign(val)
except:
    print('Val is not a number')
    print('Enter a new number')

At the top of the statement is the code that we are interested in executing, which is run in the try statement. If that fails then the except statement comes into effect, (hopefully) returning helpful information to the user about what happened and giving them some guidance on how to avoid the problem in future.

Using try-except statements results in clearer, easier to understand code by following the common Python coding style of EAFP (it’s easier to ask for forgiveness than permission). This style shows the code we want to execute first, assuming that the incoming data is correct, before dealing with exceptions if the assumptions prove false.

The except statement will catch all errors and so we do not, initially at least, need to know exactly what errors we are trying to avoid. However, python does provide error codes, which we can use to expand the structure to capture specific error types. For the example above, we would want to capture a TypeError:

PYTHON 

try:
    check_sign(val)
except TypeError as err:
    print('Val is not a number')
    print('But our code does not crash anymore')
    print(f'The run-time error is: {err}')

As with if statements, multiple except statements can be used, each with a different error test. These can be followed by an (optional) catch-all bare except statement (as we started with) to catch any unexpected errors. Note that only one try statement is allowed in the structure.

try-except structures may also contain else and finally statements (following in order after the except statements). The else statement will be executed if no except statements are executed, while the finally statement will be executed after all other statements are finished.

PYTHON 

try:
    check_sign(val)
    reciprocal = 1/val
except TypeError as err:
    print('Val is not a number')
    print(f'The run-time error is: {err}')
except Exception as err:
    print('Some error other than a TypeError occured')
    print(f'The run-time error is: {err}')
else:
    print(f'The reciprocal of the value = {reciprocal}')
finally:
    print('release memory')

The typical use of the finally statement is to deal with the release of external resources (such as files or network connections) whether or not the attempted action has been successful.

More details about errors and exceptions can be found in the Python tutorials.

Challenge

HTTP errors

Earlier we were introduced to the requests library. This is used to access web content, and returns a code indicating the request status. The raise_for_status function is provided for testing the request status, which returns a request.HTTPError if there is an error. For example, if we request a missing webpage we get this error:

PYTHON 

import requests

source_url='https://api.datacite.org/dois/10.48546/workflowhub.workflow.56.1000'
response = requests.get(source_url)
response.raise_for_status()

OUTPUT 

Traceback (most recent call last):

  File "<ipython-input-108-06c23faa067a>", line 3, in <module>
    response.raise_for_status()


  File "/Users/mbessdl2/anaconda3/envs/myenv/lib/python3.9/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)

HTTPError: 404 Client Error: Not Found for url: https://api.datacite.org/dois/10.48546/workflowhub.workflow.56.1000

Write a try-except structure that uses raise_for_status and deals with the error without crashing.

PYTHON 

import requests

try:
    source_url='https://api.datacite.org/dois/10.48546/workflowhub.workflow.56.1000'
    response = requests.get(source_url)
    response.raise_for_status()
except requests.HTTPError as err:
    print(err)

OUTPUT 

404 Client Error: Not Found for url: https://api.datacite.org/dois/10.48546/workflowhub.workflow.56.1000

Assert

The try-except structure is useful for controlling small chunks of code, where we might reasonably expect that the user can fix the problem (where it is a program which takes a short time to run), or where our program itself will fix the problem (such as substituting NaN values for a failed calculation).

Sometimes, however, it is better to check the data before it is passed into the code and to trigger an early crash in the program, which returns a meaningful error. For example, this can be extremely useful in the case of large numerical calculations that take a long time to run and which will only use some of the provided variables after a significant amount of the computational work has already been carried out. In such situations your user will appreciate being warned early on that they have provided invalid data.

To conduct such a test we can use an assert statement. This follows the structure: assert <test>, <error message>. As an example of this, we can write the test for non-numerical values as:

PYTHON 

val = 'a'

assert type(val) is float or type(val) is int, 'Variable has to be a numerical object'

check_sign(val)

OUTPUT 

Traceback (most recent call last):

  File "<ipython-input-38-52d27526daab>", line 1, in <module>
    assert type(val) is float or type(val) is int, "Variable has to be a numerical object"

AssertionError: Variable has to be a numerical object

Like the earlier if statement, this catches the problematic variable before it reaches our calculation. However it is far less intrusive code, and assert tests can be added in sequence, so that we can quickly and easily increase the coverage of such tests.

Callout

Warning when using optimisations

Using asserts when testing and running code while debugging is an invaluable way of checking that your code is working. However, be aware that if you are using assert in conjunction with optimization, then there are some built-in variables which change value when optimization is requested which have a knock-on effect on the behaviour of an assert; for more information look at the documentation on the assert statement.

Challenge

Testing for NaN’s (Not a Number)

Numpy provides Not a Number (nan) values, which can be used to indicate missing data. These are not caught by the test above:

PYTHON 

import numpy as np

val = np.nan

assert type(val) is float or type(val) is int, "Variable has to be a numerical object"

check_sign(val)

OUTPUT 

Value: nan is negative.

However numpy does provide the isnan() function for testing if a value is NaN. Add an assertion test using this which will raise and error if the value is NaN.

Because the test checks if a value is NaN, we need to add not before the test, to ensure that the assert statement fails if it is a NaN:

PYTHON 

import numpy as np

val = np.nan

assert type(val) is float or type(val) is int, "Variable has to be a numerical object"
assert not np.isnan(val), "Variable must not be a NaN"

check_sign(val)

OUTPUT 

Traceback (most recent call last):

  File "<ipython-input-52-553a9c93f77a>", line 4, in <module>
    assert not np.isnan(val), "Variable must not be a NaN"

AssertionError: Variable must not be a NaN
Key Points
  • try-except structures are useful for catching errors as they occur
  • assert structures are useful for forcing errors early, to avoid wasted effort

Content from Software Package Management

Last updated on 2026-10-08 | Edit this page

Overview

Questions

  • How can I load new python libraries?
  • How can I deal with libraries which have competing software requirements
  • How can I record the python libraries that I use for my work?

Objectives

  • Learn how to install a conda package
  • Learn how to use different conda channels
  • Learn how to create a new conda environment
  • Learn how to activate / deactivate conda environments
  • Learn how to record the setup of a conda environment

Virtual environments are a useful tool for isolating and managing the software packages you use. Doing this enables you to track the packages you use for your work, letting you create reproducable environments so that others can use your code. It also allows you to use packages which might have conflicting requirements (or even different versions of the same package) without the hassle of having to install and uninstall these each time.

Python has built in tools for managing virtual environments (venv) and packages (pip). However, we will not cover these tools today. While venv and pip are very useful for managing pure python packages, they are not very flexible when we want to use packages which are built on other languages (or perhaps do not use python at all). Another tool, conda, has been built to extend their functionality to cover these mixed-language packages for a wide range of computing platforms, and it is this which we will cover today.

More details on the differences between conda and pip are given in this summary. Other tools for package management exist as well as conda - popular ones include uv and poetry. See here for a full list of packaging projects.

The lesson below is not conducted using a Python interpreter, but instead using the unix shell. When you are asked to type in code below, please do this in a terminal window and not in a Jupyter notebook or other Python interpreter.

Installing Conda

The conda package manager is available to install by itself (using the miniconda installer), or with a preprepared set of commonly used packages and a graphical user-interface (GUI) for managing your packages (using the anaconda installer). Miniconda is a good choice if you have limited bandwidth or disk space, and are happy to install packages as you need them using the command line interface (CLI). Anaconda is a good choice if you have more disk space (it requires >3Gb) and want to use a GUI.

We will not cover installation in this lesson but more information is available on both of these conda installers, as well as links for downloading the installers, here: https://docs.conda.io/projects/conda/en/latest/user-guide/install/download.html.

Package Channels

Conda uses “channels”, locations where software packages are stored. Conda has a default set of these channels that it searches when you request a package or update. You can modify this list of channels, to add other public or private channels that you wish to use. One of the most commonly added is conda-forge, a community channel with thousands of contributors who use its unified, automated build infrastructure to share their software with the world. Another widely-used channel is bioconda, which is dedicated to bioinformatics software.

We can see what channels are being used by the command:

BASH 

conda config --show channels

This will (likely) list the the following:

OUTPUT 

channels:
  - defaults

The channel order is important for determining priority when installing packages. Those from channels higher in the list will be given priority over those lower in the list. After this the newer versions of a package are given priority over older versions of a package (and then build numbers, for a given version). The order in which priority is determined can be changed, we won’t cover this today, but more information can be found in the conda managing channels documentation.

To add a channel we can use:

BASH 

conda config --add channels conda-forge
conda config --show channels

OUTPUT 

channels:
  - conda-forge
  - defaults

We can repeat this to add as many channels as we need:

BASH 

conda config --add channels bioconda
conda config --show channels

OUTPUT 

channels:
  - bioconda
  - conda-forge
  - defaults

This adds the new channel to the start of the list. Which gives that channel highest priority. The documentation for bioconda, however, recommends a priority order of conda-forge, bioconda, and finally defaults. To correct the channel order we can simply re-add the conda-forge channel:

BASH 

conda config --add channels conda-forge

OUTPUT 

Warning: 'conda-forge' already in 'channels' list, moving to the top

When we do this we should be warned the channel is already in the list, but it is being moved up to the highest priority position.

Callout

removing or appending channels

Channels can be removed using the --remove flag instead of --add. To add a channel to the end of the list, rather than the front, we can use --append instead of --add (and, if we wish to be explicit in our intent to put a channel at the front of the list we can use --prepend, which behaves in the exact same way as --add).

Searching for Packages

Once you’ve added the channels you wish to use, you can now search them for the software that you need. This is simply carried out using conda search. For example, if we wished to find out what versions of Gromacs are available, we can use:

BASH 

conda search gromacs

OUTPUT 

# Name                       Version           Build  Channel
gromacs                        4.6.5               0  bioconda
gromacs                       2018.3      h470a237_0  bioconda
gromacs                       2018.4      h470a237_0  bioconda
gromacs                       2018.5      h04f5b5a_0  bioconda
gromacs                       2018.6      h04f5b5a_0  bioconda
gromacs                       2018.6      h04f5b5a_1  bioconda
gromacs                       2018.6      hd895feb_2  bioconda
gromacs                         2019      h04f5b5a_0  bioconda
gromacs                       2019.1      h04f5b5a_0  bioconda
gromacs                       2019.1      h04f5b5a_1  bioconda
gromacs                       2019.1      h04f5b5a_2  bioconda
gromacs                       2020.5      hd895feb_1  bioconda
gromacs                       2020.5      hd895feb_2  bioconda
gromacs                       2020.5      hd895feb_3  bioconda
gromacs                       2020.5      hd895feb_4  bioconda
gromacs                       2020.5      hd895feb_5  bioconda
gromacs                       2020.6      h94ec9d8_0  bioconda
gromacs                         2021      hd895feb_0  bioconda
gromacs                         2021      hd895feb_1  bioconda
gromacs                       2021.1      hd895feb_0  bioconda

This shows you the name of the package you searched for, then the version of the software, build number (which can increment up as bugs are fixed, see above for version 2020.5), and the channel that the package is in.

If we know the version of the software we are interested in we can narrow this search down. For example, if we are only interested in gromacs versions greater than 2020 we can use:

BASH 

conda search "gromacs>=2020"

OUTPUT 

# Name                       Version           Build  Channel
gromacs                       2020.5      hd895feb_1  bioconda
gromacs                       2020.5      hd895feb_2  bioconda
gromacs                       2020.5      hd895feb_3  bioconda
gromacs                       2020.5      hd895feb_4  bioconda
gromacs                       2020.5      hd895feb_5  bioconda
gromacs                       2020.6      h94ec9d8_0  bioconda
gromacs                         2021      hd895feb_0  bioconda
gromacs                         2021      hd895feb_1  bioconda
gromacs                       2021.1      hd895feb_0  bioconda

Note that we’ve put the search string in quotations, in order to ensure it is passed to conda correctly.

We can also use wildcards in the version string. For example, if we wish to find all numpy packages with version 1.20:

BASH 

conda search "numpy==1.20.*"

OUTPUT 

# Name                       Version           Build  Channel
numpy                         1.20.0  py37h3795f5d_0  conda-forge
numpy                         1.20.0  py37ha9839cc_0  conda-forge
numpy                         1.20.0  py38h64deac9_0  conda-forge
numpy                         1.20.0  py39h3c955ea_0  conda-forge
numpy                         1.20.1  py37h3795f5d_0  conda-forge
numpy                         1.20.1  py37h43259c0_0  pkgs/main
numpy                         1.20.1  py37ha9839cc_0  conda-forge
numpy                         1.20.1  py37hd6e1bb9_0  pkgs/main
numpy                         1.20.1  py38h43259c0_0  pkgs/main
numpy                         1.20.1  py38h64deac9_0  conda-forge
numpy                         1.20.1  py38hd6e1bb9_0  pkgs/main
numpy                         1.20.1  py39h3c955ea_0  conda-forge
numpy                         1.20.1  py39h43259c0_0  pkgs/main
numpy                         1.20.1  py39hd6e1bb9_0  pkgs/main
numpy                         1.20.2  py37h84c02c4_0  conda-forge
numpy                         1.20.2  py37hc415c66_0  conda-forge
numpy                         1.20.2  py38had91d27_0  conda-forge
numpy                         1.20.2  py39h7eed0ac_0  conda-forge

Note that the build string now starts with pyXX, where XX indicates the version of python the package was built against. When conda packages are dependent on python then they usually will be built against a number of different versions of python - which gives more flexibility when trying to create an environment. Also note that two channels are listed here, the default pkgs/main and conda-forge. These duplicate some versions, but also note that conda-forge has the most recent version of the numpy software, whereas the default channel does not. conda-forge tends to be updated more frequently than the default channels.

Creating Environments

As the range and variety of software used increases, it is becoming very common for users require a number of different software programs for their work which need access to different versions of the same software library. In addition there could be conflicts between the operating system requirements (as these tend to change slowly, and be built on reliable software libraries) and the latest version of the required software package. Usually, installing all this software on the same system would be very difficult, but if we use virtual environments to separate these programs and their dependencies, then the task becomes a lot more simple.

We will start with listing the current environments:

BASH 

conda env list

OUTPUT 

# conda environments:
#
base                  *  /Users/mbessdl2/anaconda3

This shows you the environments you have installed, and the path where they are installed. In addition the * indicates which environment you currently have loaded. In the example above, anaconda is installed by the user, and is only accessible by that particular user. However, if anaconda has been installed by the administrator of that computer then you might something like:

OUTPUT 

# conda environments:
#
base                  *  /opt/apps/anaconda

In this case the anaconda install will be accessible for all users, however no-one other than the administrator will be able to install new packages into the base environment.

You can list the packages you have installed in your current environment using:

BASH 

conda list

OUTPUT 

# packages in environment at /Users/mbessdl2/anaconda3:
#
# Name                    Version                   Build  Channel
_anaconda_depends         2019.03                  py37_0
_ipyw_jlab_nb_ext_conf    0.1.0                    py37_0
alabaster                 0.7.12                     py_0    conda-forge
anaconda                  custom                   py37_1
anaconda-client           1.7.2                      py_0    conda-forge
anaconda-navigator        1.9.12                   py37_0
anaconda-project          0.8.3                      py_0    conda-forge
...
zip                       3.0                  h1de35cc_1    conda-forge
zipp                      3.1.0                      py_0    conda-forge
zlib                      1.2.11            h0b31af3_1006    conda-forge
zstd                      1.4.4                he7fca8b_1    conda-forge

When you begin using conda, as you can see, you are given a default environment named base. Avoid installing programs into your base environment, it is better to create separate environments to keep your programs isolated from each other. A new conda environment can be created using the command:

BASH 

conda create --name myenv

This will create an environment named myenv (once you press [y] to proceed). Check this using the conda env list command.

OUTPUT 

# conda environments:
#
base                  *  /Users/mbessdl2/anaconda3
myenv                    /Users/mbessdl2/anaconda3/envs/myenv

This environment will be empty of all packages though, as we can show by specifying the myenv environment when we list the installed packages:

BASH 

conda list myenv

OUTPUT 

# packages in environment at /Users/mbessdl2/anaconda3:
#
# Name                    Version                   Build  Channel

We can also specify the packages we want to install when creating an environment. When this is done conda will work out all the necessary supporting packages for you. For example, we will create the myenv environment again, this time installing the spyder IDE package, as well as the pandas and matplotlib packages, which we will need later:

BASH 

conda create --name myenv spyder pandas matplotlib

This will warn you that you are about to overwrite another environment, press [y] to continue. Conda will then workout the new environment setup (which can take a little time), and then will list the packages which are to be downloaded, and those which will be installed, before asking if you wish to continue. Press [y] to continue, and then check that the environment exists and that the packages you expect are installed using conda env list, and conda list myenv as before.

Activating and Deactivating Environments

Now that we have installed spyder we would like to run it. However it is not available to us yet:

BASH 

which spyder

(if this cannot find the spyder program it will return nothing)

Before we can use this software, we must activate the environment it is in. This can be done using:

BASH 

conda activate myenv

Once this done you should see the name of the environment you are using in brackets before your command prompt. (If you have used pip, this will be familiar.) And you will now be able to access the program:

BASH 

which spyder

OUTPUT 

/Users/mbessdl2/anaconda3/envs/myenv/bin/spyder
Callout

What if conda hasn’t been configured yet?

It is possible that conda activate <ENV> will give you this error:

OUTPUT 

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.

This will often happen on computers with a central installation of anaconda, where the anaconda environment management tools will have not been initialised (as many users will not need them). This can be corrected by following the instructions in the error message (assuming that your current shell is bash, change the last word if not):

BASH 

conda init bash

This will modify your personal shell configuration (in file /home/mbessdl2/.bashrc or similar). You will have to close that shell, and then open another shell for the changes to take effect.

Installing Software into an Existing Environment

During your work you often will find that you need to add new software packages to your environment. For example, we will need the astropy and pint packages for the units and quantities lesson, and so we should add these now.

To install these packages (into the current environment we are in) we can use:

BASH 

conda install astropy pint

If you are installing packages into an environment which is not active (or if you want to be certain that you are installing into the correct environment), you can add -n <ENV>:

BASH 

conda install -n myenv astropy pint

Uninstalling software can be carried out in a similar manner, using conda uninstall.

Recording and Reproducing Environment Setups

A big advantage of using virtual environments is the ability to record the environment in which you carry out your work, so that this can be shared with others in order to help them replicate your work.

The most basic way to obtain this information is using:

BASH 

conda list --export

OUTPUT 

# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: linux-64
_libgcc_mutex=0.1=conda_forge
_openmp_mutex=4.5=1_gnu
alabaster=0.7.12=py_0
alsa-lib=1.2.3=h516909a_0
appdirs=1.4.4=pyh9f0ad1d_0
argh=0.26.2=pyh9f0ad1d_1002
arrow=1.1.0=pyhd8ed1ab_1
astroid=2.5.6=py39hf3d152e_0
...
zeromq=4.3.4=h9c3ff4c_0
zipp=3.4.1=pyhd8ed1ab_0
zlib=1.2.11=h516909a_1010
zstd=1.4.9=ha95c52a_0

This contains similar information to the standard conda list command, but in a machine readable formation. The output information can be saved to a text file using conda list --export > envfile.txt, which can then be used to recreate the environment elsewhere, using the command conda create --name <env> --file <this file>. However, it should be noted that this will be restricted to the operating system it is created on, and the channel information is not recorded in this output - so if you have used more than just the default and conda-forge channels it may be difficult for someone else to replicate.

A better method is to use:

BASH 

conda export

OUTPUT 

name: myenv
channels:
  - conda-forge
  - bioconda
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=1_gnu
  - alabaster=0.7.12=py_0
  - alsa-lib=1.2.3=h516909a_0
  - appdirs=1.4.4=pyh9f0ad1d_0
  - argh=0.26.2=pyh9f0ad1d_1002
  - arrow=1.1.0=pyhd8ed1ab_1
  - astroid=2.5.6=py39hf3d152e_0
 ...
  - zeromq=4.3.4=h9c3ff4c_0
  - zipp=3.4.1=pyhd8ed1ab_0
  - zlib=1.2.11=h516909a_1010
  - zstd=1.4.9=ha95c52a_0
prefix: /home/mbessdl2/.conda/envs/myenv

This will present the same information, but in the more readable YAML format, and it will also include the channel information in the output. This information can be saved as a txt or yml file as before e.g. conda export > envfile.yml, and used to create a new environment using the command conda env create (see below). However it is still using operating system specific information.

To create a higher-level record of the environment you can use:

BASH 

conda export --from-history > envfile.yml
more envfile.yml

OUTPUT 

name: myenv
channels:
  - conda-forge
  - bioconda
  - defaults
dependencies:
  - pandas
  - spyder
  - astropy
  - pint
prefix: /home/mbessdl2/.conda/envs/myenv

This builds a list of only the packages that you explicitly requested to be installed in that environment. Where you have specified a particular version of a package, then this will be recorded, but otherwise the version number will be unconstrained. If you wish to constrain the version of a package (for example, if a certain feature you use has been removed in more recent versions of the package), then you will need to edit this information to specify that (using the information given by the standard conda list command).

To create a new environment from either of these YAML environment files, you can use the command:

BASH 

conda env create --file <env file> --name <env name>

The name that you give the environment does not have to be the same as the name specified in the file, but if you don’t explicitly specify a name then the name in the file will be used.

Key Points
  • conda virtual environments are useful for installing programs with differing requirements
  • conda config --add channels <channel> adds a new channel to your list of sources
  • conda search <package> will find all available versions of a package in your list of sources
  • conda create -n <env> <package(s)> can be used to create a virtual environment from a list of packages
  • conda install -n <env> <pacakge(s)> installs packages in a pre-existing environment
  • conda activate <env> activates the named environment, giving access to the software installed there
  • conda deactivate deactivates the current environment
  • conda export --from-history > <file.yml> creates a portable record of your current environment
  • conda env create --file <file.yml> <env> creates a new environment from an environment file

Content from Object oriented programming and classes in Python

Last updated on 2026-10-08 | Edit this page

Overview

Questions

  • How do you do object oriented programming in Python?
  • How do you use objects such as classes in Python?

Objectives

  • Learn about object oriented programming in general
  • Learn about classes

Add an introduction here

Object oriented programming in Python

Object oriented programming in Python is

Challenge

First challenge

Description first challenge

PYTHON 

print("hello world")

The correct answer is

Classes in Python

Classes are

Callout

Classes in other languages

Classes in other programming languages

Key Points
  • OOP
  • Classes

Content from Structuring a Python project

Last updated on 2026-10-08 | Edit this page

Overview

Questions

  • How do you do structure a medium-to-large project in Python?
  • How do you use an IDE for Python?

Objectives

  • Learn about structuring a python project
  • Learn about using IDEs

Add an introduction here

Structuring a project

Project

Challenge

First challenge

Description first challenge

PYTHON 

print("hello world")

The correct answer is

Python IDEs

Integrated Development Environment

Callout

Other IDEs

Examples of IDEs

Key Points
  • Projects
  • IDEs

Content from Testing in Python

Last updated on 2026-10-08 | Edit this page

Overview

Questions

  • How do you do (simple) testing in Python?
  • What frameworks are available for testing in Python?

Objectives

  • Learn about testing using asserts
  • Learn about testing frameworks

Add an introduction here

Using asserts

Asserts

Challenge

First challenge

Description first challenge

PYTHON 

print("hello world")

The correct answer is

Testing Frameworks in Python

pytest, unittest etc.

Callout

Testing in other languages

Testing in other programming languages

Key Points
  • assert
  • testing frameworks