Description of how the Raspberry Pi eVOLVER server code functions.
This page aims to explain how the server on the Raspberry Pi functions. The repository can be found on Github.
General Information
The server acts as the brain for the physical eVOLVER unit - it manages all of the SAMD21 Arduino microcontrollers and communications from the user via the DPU or GUI interface.
In addition to managing communications between connected components and the user, it also stores and manages configuration and calibration data for a given eVOLVER. The RPi also can have other utilities run on it such as this , which can detect if the server gets hung up and restart it.
File Explanations
evolver.py and evolver_server.py
The two main python scripts for the server are evolver.py and evolver_server.py. The first, evolver.py sets up the websockets server and get it running in it's own background thread so that in the main thread the server can communicate with the SAMD21 micro controllers continuously and send off data as needed. The main logic of the server lives in evolver_server.py.
The timing of broadcasts is handled in evolver.py, while all subsequent logic is handled in evolver_server.py.
conf.yml
The server uses this file to notate what kinds of experimental parameters are connected, how they should be used, and configurations for running the server itself.
Read more about this file .
You can modify this file manually (it is meant to be human readable), but be careful not to break the structure or the server will not run.
calibrations.json
This file contains all calibrations for the eVOLVER. Calibrations have a generalized format, so theoretically you can calibrate any number or types of parameters and have the downstream DPU or GUI be able to fetch and use them.
The file is a list of dictionaries. Each item has a name, measured data (the values collected from a platereader or thermometer for example), the calibration type, and a key-value pair for raw and fits. Raw and fits both contain lists of raw data collection and fits, respectively. Fits has an element called coefficients which contains the fit parameters for whatever function was used, as well as the params that it needs.
The schema can be found on . To read more about the file, read .
Don't confuse this file with calibration.json - this is a legacy file that will be phased out soon.
Server Serial Communication
Check guide to view the serial communications in real time. This is a useful troubleshooting tool.
Serial Message Structure
The RPi communicates to and from the SAMD21 microcontrollers via serial. The structure of the message :
<ADDRESS><MESSAGE_TYPE>,<VALUES>,<END_CHARACTER>
<ADDRESS> is the address or name of the SAMD21. It allows all of the connected microcontrollers to decide if the message is for them.
<MESSAGE_TYPE> is typically a single character designating how the message should be handled.
r: recurring message - sent every broadcast iteration on the RPi to ensure that the correct values are being maintained by the SAMD21 microcontrollers.
i: immediate command - sent when a one-off command is to be executed. Functionally handled the same on the SAMD21 microcontrollers. The server will not re-send these commands in a recurring manner (useful for something like a pump event).
e: echo response - Sent from the SAMD21 to the RPi as a way to acknowledge receipt of a command or message. The values should match what the server sent.
b: data response for broadcast - Sent from thhe SAMD21 to the RPi as a response to getting a recurring or immediate command if the SAMD21 has data to report (temp, OD). Values will contain raw ADC values typically.
a: acknowledge response - Sent from the RPi to the SAMD21, signalling that it has registered success of serial communications and signalling whatever previous command was sent to be executed.
<VALUES> is a comma separated list of values either being sent to the SAMD21 as inputs to set an experimental condition level, or as raw ADC values from the SAMD21 to the RPi.
<END_CHARACTER> is a way to denote the end of the message. For messages going to the SAMD21 from the RPi, this will be _!. For messages going to the RPi from the SAMD21, this will be end.
Serial Communications Process
We use a communication structure similar to the .
The RPi will first send a message to the SAMD21 containing the desired values to be set (<MESSAGE_TYPE> of i, or r).
The SAMD21 then needs to respond with a <MESSAGE_TYPE> or b or e.
The RPi will respond with a <MESSAGE_TYPE>
This process is used in order for the server to have a way to know if commands are successfully being sent to the desired SAMD21. For each response from the SAMD21, the lengths of the <VALUES> is checked to be consistent with what is in the server configuration file. The server also checks that messages it sends out conform to the proper length. If any anomalies or problems are detected, an EvolverSerialError is thrown.
Failed commands will be communicated back to the DPU via websockets in a future version.
Example Annotated Server Logs as a Command Comes in
Recurring od_90 Command Set in the conf.yml File
Immediate Stir Off Command from the DPU
Websockets Communications
The eVOLVER server uses websockets via to communicate with the DPU and GUI interfaces. Please see their documentation for more information on how websockets work. In brief, functions in the server are decorated with
to indicate that the function should be run when a given event occurs over the websocket. The function definition itself also needs the async keyword in front of it (asyncronous).
To send out messages to connected clients (DPU/GUI), the server uses a socketio.AsyncServer object sio. For example, to send data to the clients:
where data is a dictionary (json) containing the data to send. Note that any function that calls the emit function needs to have the keyword async in it's definition. This also propogates up to all functions that call that function as well.
Be careful with the async/await keywords - read up on the if you want to write your own custom communication events. Improper usage of these keywords and functions can cause the server to hang up or not run at all.
Relevant information
of a (for acknowledge). When the SAMD21 receives this message it will execute the command from step 1.
# Recurring (i) od_90 command, requesting average of 500 values for each Smart Sleeve:
od_90r,500,_!
# Broadcast (b) data response from the SAMD21
od_90b,53722,48267,50671,41662,62813,63373,60965,60209,50271,49000,51695,56800,61598,62685,60486,62862,end
# Acknowledge (a) the server tells the SAMD21 Arduino board it received the data
od_90a,,_!
Connected dpu as server # Connection to the DPU (GUI or experiment script)
Received COMMAND # The following command was from the DPU (not conf.yml)
stiri,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,_! # Immediate (i) stir command, each vial is turned off
Disconnected dpu as Server # DPU disconnects after its job is done
stire,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,end # Echoed (e) response from the SAMD21 Arduino board. It got the command
stira,,,,,,,,,,,,,,,,,_! # Acknowledge (a) the server tells the SAMD21 Arduino board it can execute the command
This is the configuration file for the eVOLVER server. The server uses this file to notate what kinds of experimental parameters are connected, how they should be used, and configurations for running the server itself.
It is a YAML file. YAML is a "human-friendly, cross language, Unicode based data serialization language designed around the common native data types of dynamic programming languages". You can read more about YAML .
Parameter Explanations
Other Parameters in conf.yml
broadcast_timing (in seconds)
How often the server will cycle (run through all of parameters in its list)
Customizing conf.yml
Edit the conf.yml file at your own risk. It is made to be human-readable, but small changes in the formatting will cause the server not to run.
Adding 'wait' commands or increasing number of parameters greatly
If you use 'wait' commands (see below) or add many commands for the server to run through, you will need to increase the broadcast_timing or the server will not finish its cycle before starting a new one. This will cause your last few commands to be never run.
Advanced conf.yml: Subcommands
More complex conf.yml files including 'pre' and 'post' commands are complicated to repeatedly alter, so storing them allows easy switching to files for different purposes.
subcommand = a command added to the command queue when another command is run
An example subcommand:
'pre' or 'post' command = a list of subcommands to be added to the command queue before or after the main parameter command
'values' = a keyword referencing the current value of a parameter in the conf file (for example, the subcommand)
'wait' = a special type of subcommand that makes the server pause for that many seconds (for example so that a command can execute)
Examples of potential alternate conf files
Calibration conf - (default conf) broadcast_timing parameter should be low to speed up calibration
stir_pause_on_od - prevents stirring from affecting OD. Important if bubbling or in low volume applications
odled_normally_off - OD LED will be normally off unless OD is read to prevent the IR light affecting other light sensors
