Description of how the DPU (Data Processing Unit) code functions.
This page aims to explain how the server on the Data Processing Unit (DPU) functions. The repository can be found on Github.
The DPU is where actual experiment logic and data collection occurs. DPU code is typically ran on a lab computer or laptop connected to the the same router as the eVOLVER. Setpoints for experimental parameters, deciding when to actuate fluidics, and deciding how to vary experimental parameters as a function of time or based on feedback from experimental data is programmed in the DPU.
Within the DPU are 3 main components, the experimental code for running experiments, the calibration code for making fits relating sensor measurements to what they physically are measuring, and the graphing code for the deprecated experiment viewing web app.
There are two main files in the experiment DPU code, eVOLVER.py
, which is the file that is run to execute an experiment, and custom_script.py
which is where experimental logic goes.
This file handles all of the initialization (creating data directories and initializing files), communications with the server, and lower level details of sending commands and receiving data. Additionally, it handles data transformations based on the calibrations whenever data is received. It then makes a call to custom_script.py
to carry out the user-desired experiment routine based on the received data.
The DPU uses websockets via socketio to communicate with the eVOLVER server. It functions as a client, connecting to the server when the experiment is running, receiving data periodically, and sending commands to the server based on the experimental design. The EvolverNamespace
class (which is the socketio client class) has functions which are called when the corresponding event occurs over the websockets connection. For example, the function on_broadcast
is called when a broadcast
event occurs from the server.
This file is where the experimental culture routine is located. At the top of the file is commented code directing the user to modify constant variables to configure the experiment (experiment name, eVOLVER to run on, etc.) Then there are two default experiment functions for running either a turbidostat or a chemostat. It is also possible to write your own custom function to run by changing the OPERATION_MODE
variable to your custom function name.
This is the deprecated Django based web-app for viewing experiment data in real-time. To run it, open a terminal and cd
to /graphing/src/
. Then execute the following:
python3 manage.py runserver 0.0.0.0:8080
You should then be able to open a web browser and navigate to localhost:8080
to see the application. The app should also be accessible via the internet if the computer is connected to the internet by navigating to <ip_of_computer>:8080
.
The GUI now allows viewing of the data in real-time - you should not need to use this unless.
An overview of what happens in the software during a calibration
For GUI calibration guides see here.
Communication between the computer and the server is done via websockets.
For more about commands and microcontroller - server communication via serial see here.
GUI connects to the server via websockets
The user has the GUI sends a temperature off command: tempi,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,4095,_!
The server passes this to the Arduinos and the temperature Arduino enacts that command
The server receives temperature data broadcast from the server
The user waits for temperatures to equilibrate in the vials
The user measures values from an independent temperature probe for each vial and records these in the GUI
The user has the GUI record 3 values at this temperature from the server
Steps 2 - 7 are repeated for two higher temperature values
When 3 values for 3 different temperatures have been collected, the GUI saves the raw sensor data and the temperature probe data on the server in calibrations.json
The GUI uses calibrate.py (guide here) to fit raw sensor data from the eVOLVER to experimental temperature probe values (in this case a linear regression)
calibrate.py uses websockets to fetch the sensor and experimental data from the server for a previously run calibration, then does the fit according to the users specifications
The GUI will automatically run this script upon completion of a calibration, but it is also possible to run manually via the command line to fetch .png files for the calibrations if desired.
The GUI adds the calibration fit values to calibrations.json
The user can now select this calibration as the active one via the GUI Setup page
Detailed description of the custom_script.py file.
This file contains all experimental design parameters and logic. It is commented to help guide users in customizing their experiment with minimal actual programming required beyond setting some variable values.
The logic of running a continuous culture experiment should be structured such that any actions are based on the elapsed_time
of the experiment and on the latest received data. The functions are not called continuously in a loop - they are only called when new data is received from the server, which is set by the broadcast_timing
on the RPi (default is every 20s).
For example, if a sinusoidal temperature profile is desired, the custom_function
should check the elapsed_time
, use that time to calculate the appropriate temp, then send that command via the update_temperature
function in the eVOLVER
object that is passed to the function.
It is possible to change this file mid experiment! Simply stop the eVOLVER script (press ctrl + c twice on the terminal running the experiment or stop the experiment in the GUI) and restart it. The experiment will continue where it left off with any changes you made to the script now being implemented.
It is best practice to calculate the values desired for all vials before sending the command so they can all be updated at once as opposed to one at a time. This minimizes the number of messages sent to the server and how many commands the server has to send via serial to the Arduiono microcontrollers. Serial communications are shared between all the parameters and are relatively slow, so best to be efficient with the commands!
At the top of the file are a block of variables, which includes the name of the experiment, and the port (not necessary for most users to modify unless you have a more complicated setup or IT needs). eVOLVER IP is specified on the command line (or experimental params if running through the GUI).
Be sure to set the VOLUME
variable! It is needed to accurately calculate the appropriate amount of time to run the pumps to hit a desired OD wiht a dilution in the t-stat and to hit a desired rate (volumes / hour) in the c-stat.
The next block of variables are where the user should set any initial values such as STIR
, TEMP
, and any other parameters. This is also where the OPERATION_MODE
is set. This value should match the name of another function within the file, which will be called every broadcast
event from the server, i.e. it should notate the name of the users desired experimental routine.
One of the default custom_functions
provided, this function executes logic to perform a turbidostat on the eVOLVER.
time_out
refers to the amount of time to run the efflux pump extra - this prevents overflows by ensuring the efflux pump runs at least a few seconds longer than the influx. If you have a large turbidostat window (i.e. big dilutions > 10ml) you could consider increasing this value for extra safety.
pump_wait
is the minimum amount of time to wait between pump events. This is to allow enough time for averaging of the OD to occur to determine if the OD dropped below the necessary threshold for the t-stat window, preventing over-dilutions.
MESSAGE
holds the fluidic message that will be sent to the server. The default length should be 48, only change this if you also change the value in conf.yml
on the server. By default, positions 0 - 15 correspond to the influx pump array, 16 - 31 to the efflux pump array, and 32-47 are for a third array if you have one installed or for any other connected fluidic devices (like solenoids for millifluidics).
One of the default custom_functions
provided, this function executes logic to perform a chemostat on the eVOLVER.
rate_config
refers to the flow rate for each vial, in volumes per hour. It should be a list of length equal to the number of vials (default 16).
start_OD
refers to the minimum OD required before starting the chemostat algorithm. Set to 0 to start at any positive OD.
start_time
refers to the minimum amount of time that needs to pass before starting the chemostat. Set to 0 to start immediately.
bolus
refers to the amount of liquid that will pump every pump event in mL. For example, a bolus of 0.5 means that eVOLVER will dilute 0.5 mL every pump event, evenly dispersed throughout the hour at a period that hits the desired V/h set by the rate_config
variable.
eVOLVER does not send a command every time one of the pumps needs to actuate for the c-stat! Instead, we send a single command that contains the periodicity and amount of time to run for each vial. This information is handled by the microcontroller. This minimzes the communications between the DPU, server, and microcontrollers and ensures accurate pump timings for each vial.
If you want to make your own custom function, write it in this file and set the OPERATION_MODE
variable at the top of the file to match your function name. Your function will be called every broadcast
by the EvolverNamespace
class in eVOLVER.py
.
ODSet
is the variable keeping track of what OD the algorithm should currently be targeting. Once the OD reaches the upper range, ODSet will be set to the lower range, which will trigger subsequent logic to dilute the vial to hit that lower target. Upon the OD reaching this target, it will set the OD to the upper limit again. See this for more information.
Detailed description of the eVOLVER.py file.
This file is where the main
function resides for the experimental DPU code, ie it is the script that is run to carry out an experiment on eVOVLER.
To start an experiment on the command line, run the following command:
If you would like to see the help menu:
Which will bring up something similar to the following:
If an eVOLVER_parameters.json
file is present, eVOLVER will use this file for all t-stat/c-stat paremeters, including temp and stir. If not, it will default to the variable inside the script that the user can specify manually.
This file contains the lower level functions that deal with communications and carrying out commands that the user specifies in the custom_script
. Additionally, this file handles experiment initialization and data collection, setting up data directories, and doing data transformations based on calibrations.
When the script is ran, a connection is first established with the server, then the initialize_exp
function is run which creates all data directories and requests the active calibrations from the server. Active calibrations are calibrations with the active
field set to true
. There should be only 1 active calibration of a given type. Active calibrations are set via the GUI, but they can be set programatically (see the setactivecal event on the server).
The eVOLVER code then enters a infinite loop which does nothing except reset the socketio conection occasionally to prevent buildup of broadcast messages. Because the custom_functions
are tied to the broadcast
even to the server, the main thread does not need to do anything besides wait for server websocket events.
This is a socketio class responsible for handling communications with the server on the RPi. Each function beginning with on_
is called when the corresponding event occurs on the websocket connection. For example, if the connect
event occurs, the on_connect
function will run.
This class is also being used as the eVOLVER experiment class. TODO: Ideally this would be a second class and not be tacked onto the socketio class - it will be updated in a future version.
The start time and OD blank information is stored as a class variable. Additionally, all low level functions for interacting with the eVOLVER RPi are located here.
Upon receipt of a broadcast event and new data from the eVOLVER RPi, the on_broadcast
function is called. This will log the experiment time, do data transformations based on calibration data, save the resulting data, and then call the custom_functions
in custom_script.py
. This function is set within custom_script.py
.
In each experiment folder, you’ll find a number of different sub-folders and files. Here, I’ll describe the folders and files as of the dpu 1.0.1 version. If you are using a different version, your files and folders may be slightly different.
One important note is that not all folders are used in all experiment types. You can also create new folders to record other types of data for custom types of experiments (more on that in another post). These folders are created in initialize_exp() in evolver.py, if you want to see how that works.
Common Folders:
OD: contains .txt files where each row is [ timepoint (h) , OD ] using the calibration to transform photodiode readings to OD600 equivalents. It is updated every loop through the software (~20 secs).
OD90 (or OD135): contains .txt files where each row is [ timepoint (h) , raw photodiode value ] without transforming the data. This could be used to apply a different calibration after the experiment (e.g. if you started the experiment with the wrong calibration selected). It is updated every loop through the software (~20 secs).
temp: contains .txt files where each row is [ timepoint (h) , temperature ] using the calibration to transform thermistor readings to Celsius equivalents. It is updated every loop through the software (~20 secs).
temp_config: contains .txt files where each row is [ timepoint (h) , temperature target ] in Celsius. It is only updated when a new temperature command is sent from the custom_script, so it won’t have very many lines for most experiments.
Tubidostat Related Folders: These folders are only updated in turbidostat mode, and are not used for chemostat mode.
ODset: contains .txt files where each row is [ timepoint (h) , target OD ]. It is updated once when the upper OD threshold is reached (changes target to lower threshold), and once when the lower density threshold is reached (changes target to upper threshold). This makes it a good choice for segmenting turbidostat data into growth curves.
pump_log: contains .txt files where each row is [ timepoint (h) , pump time(s) ]. It is updated at every pump event, and generally records the amount of time the influx pump was turned on for, though in some older versions a bug caused the efflux pump time to be recorded instead.
Since it records when a pump event is sent to eVOLVER, the dilution won’t show up in the OD data right away, making it a poor choice to use to segment OD data into growth curves. Furthermore, if two dilutions are needed to reach a lower threshold (i.e. you switched out a bottle and there’s some air in the lines) pump log will be updated twice in short succession, complicating the process further.
Chemostat pump events do not get listed here, as they use a recurring function on the Arduino, to reduce the load of commands sent to eVOLVER. Pump commands sent from outside the DPU (i.e. from the Electron app or MATLAB script) also do not get recorded in pump_log files.
growthrate: contains .txt files where each row is [ timepoint (h) , growth rate (h^-1) ] using ODset to segment data and a linear fit to log transformed OD data. It is updated every time the upper OD threshold is reached.
The first growth rate measurement is fit on the data from the first timepoint to the first dilution, which means inoculation and lag phase can throw off this value.
You may want to calculate growth rate offline after the expt. The growth rate calculated here has been optimized for robustness over accuracy, in order to enable feedback control.
Chemostat Related Folders: These folders are only updated in chemostat mode, and are not used for turbidostat mode.
chemo_config: contains .txt files where each row is [ timepoint (h) , chemo_phase, period (s) ]. It is updated every time the chemostat dilution rates are changed. The second value can be used to keep track of updates if designing a custom chemostat experiment that changes or fluctuates between dilution rates. The third value is the period of time, in seconds, between pump events. Note that these pump events do not get recorded in pump_log.
Other Files: evolver.log: some versions of the DPU will store a log of things that happen during the experiment, like errors or verbal descriptions of certain commands (i.e. changes to chemostat rates). XX_cal.txt : some versions of the DPU will store .txt versions of the calibrations locally on your computer. EXP_NAME_DATE_TIME.txt: this is a .txt record of your custom script, created each time you restart an experiment. This way if you change the code partway through, you have a record of each iteration of the code.
From the :