Hello again PowerBroker!

The lab is happy with how the Serial Transfer works and has asked me to try and implement a transfer of all experimental variables to the Arduino. I'm wondering how I should best transfer many different variables using the package. Should I send things using individual functions one at a time or should I open communications once and send it all during the same transmission? Do you think it matters which way I do it?

Hello PowerBroker!

My name is Jeremy and I just discovered this package for reading/writing to an Arduino using Python. I'm trying to send a list of numbers that indicate a trial type for an experiment I'm working on. Generating the list externally through Python and sending it for use in the experiment would be so helpful for my team!

I've successfully gotten your package to communicate back and forth to the Arduino, but I'm not sure how to make sure the data I sent is actually saved and usable in the Arduino's memory. I'm not sure if this is the appropriate place to ask about this kind of testing since the package seems to work, but I didn't see a different way of trying to contact you.

Any advice would be greatly appreciated! It would be cool to contribute to the project as I learn and provide examples relevant for my team's use that others could use.

Sincerely,

Jeremy Delahanty Salk Institute for Biological Studies, La Jolla CA

Hey there, I firstly want to say that this library (and its Arduino companion) is awesome and makes communicating with Arduinos so much easier.

I've been running into some issues reading data from multiple Arduinos at the same time. If I just read from one Arduino, it only takes 1-2ms for transfer.available() to complete (measured using time.perf_counter). But if I connect 2 devices that are both sending data to the computer at 10hz, the readings sometimes only take 1-2ms, but sometimes they take 20-70ms. Usually it starts at 1-2ms for a few seconds, then it starts taking 20-70ms for the rest of the time it is running.

The 2 transfer objects are on separate threads, so I imagine this is the root of the issue somehow... I am sending data on a separate thread, but I tried adding locks and that didn't fix the problem.

If I never send any data and just receive, then it always runs at 1-2ms. If I send data for a few seconds and then stop sending data, then it runs at 20-70ms, even well after I stopped sending data.

Each transfer receives in its own thread in this method:

    def run_serial_receiver(self, remote: transfer.SerialTransfer, lock):
        while True:
            time.sleep(.001)
            with lock:
                t0 = time.perf_counter()
                if remote.available():
                    dt = (time.perf_counter() - t0)
                    print(dt)
                    if(dt>.01):
                        print(f"EXCESSIVE TIME: {dt}")
                    sliced_buff = remote.rxBuff[0:remote.bytesRead]
                    rxbytes = bytes(sliced_buff)
                    self.process_RemoteMessage(rxbytes)

And data is sent using this method on the main thread:

    def send_output(self, output):

        packet_size = len(output)
        for i, remote in enumerate(self.remotes):
            with self.serial_locks[i]:
                remote.txBuff = output
                remote.send(packet_size)
        if self.send_callback:
            self.send_callback()

I'm running on Windows 10, with the SerialTransfer library running on a Teensy 4.0.

Hi, I'm trying to send a structure from a microcontroller to my computer. The structure has 7 fields and each field is a "packetSize" element array. Here is the structure initialization on the microcontroller:

const int packetSize = 5; struct STRUCT { int ax[packetSize]; int ay[packetSize]; int az[packetSize]; int gx[packetSize]; int gy[packetSize]; int gz[packetSize]; int ts[packetSize]; } sendStruct;

The python code on my computer looks like this: data = link.rx_obj(obj_type=list, obj_byte_size=457, list_format='i')

In the above python code, 4 is the bytes per number, 5 is the "packetSize," and 7 is the number of fields in my structure. I'm trying to maximize the "packetSize" here. When I upload my arduino code, I can put any number for packetSize, but if I want to receive the data through python the largest packetSize I can use is 9. A packetSize of 9 or less works perfectly, but anything greater than 9 gives the following error:

// arr = array(list_format, buff) ValueError: bytes length not a multiple of item size //

Is there a way I can use a packetSize of more than 9 or is this the maximum amount of data that can be sent at once?

Hi, thank you for providing this crucial library! I'm sure I have overlooked something. I appreciate your help in resolving this. Full Arduino code at the bottom of this post.

The Ultimate Goal: Arduino reads one sample from each of 8 sensors, into an array[8], and transmits the array over serial, 80 times/sec. Python parses the incoming array and plots the values on a graph in real time.

The good: I believe everything is correct and working well on the arduino side. When I use Serial.print() to print the array of sensor values instead of using SerialTransfer, the values change each time like they should - so the Arduino is not sending duplicate packets.

The Python Code: (I'm working in Jupyter on Windows10)

import time
from pySerialTransfer import pySerialTransfer as txfer

link = txfer.SerialTransfer('COM7', baud=115200)
link.open()
time.sleep(2) # allow some time for the Arduino to completely reset

def get_packet(st_obj, byte_size):
    while True:
        st_obj.available()
        yield st_obj.rx_obj(obj_type = list,
                       obj_byte_size = 32,
                       list_format = 'l')
        
stream = get_packet(link, 32)

I can print 7 times...

for x in range(7):
    print(stream.__next__())

Prints the same packet 7 times:

[-179442, -161803, -204057, -239409, -198581, -230921, 16002, -176593] [-179442, -161803, -204057, -239409, -198581, -230921, 16002, -176593] [-179442, -161803, -204057, -239409, -198581, -230921, 16002, -176593] [-179442, -161803, -204057, -239409, -198581, -230921, 16002, -176593] [-179442, -161803, -204057, -239409, -198581, -230921, 16002, -176593] [-179442, -161803, -204057, -239409, -198581, -230921, 16002, -176593] [-179442, -161803, -204057, -239409, -198581, -230921, 16002, -176593]

If I try to print an 8th time: print(stream.__next__())

I get an IndexError:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-4-0991908dffd0> in <module>
----> 1 stream.__next__()

<ipython-input-2-d12a92ebb86c> in get_packet(st_obj, byte_size)
      5 def get_packet(st_obj, byte_size):
      6     while True:
----> 7         st_obj.available()
      8             yield st_obj.rx_obj(obj_type = list,
      9                        obj_byte_size = 32,
C:\Users\me\AppData\Local\Continuum\anaconda3\lib\site-packages\pySerialTransfer\pySerialTransfer.py in available(self)
    475                     elif self.state == find_payload:
    476                         if self.payIndex < self.bytesToRec:
--> 477                             self.rxBuff[self.payIndex] = recChar
    478                             self.payIndex += 1
    479 

IndexError: list assignment index out of range

What am I doing wrong here?

Full Arduino Code:

#include "HX711-multi.h"
#include "SerialTransfer.h"

// Hardware Config
#define CLK1 2    // clock pin to the first group of ADCs
#define DIN1 3    // data pin to the first lca
#define DIN2 4    // data pin to the second lca
#define DIN3 5    // data pin to the third lca
#define DIN4 6    // data pin to the fourth lca

#define CLK2 8    // clock pin to the 2nd group of ADCs
#define DIN5 7    // data pin to the fifth lca
#define DIN6 10   // data pin to the sixth lca
#define DIN7 11   // data pin to the seventh lca
#define DIN8 12   // data pin to the eighth lca

// System Config
#define CHANNEL_COUNT 4
#define NUM_GROUPS 2
#define GAIN 128                  // Gain level for HX711 to use ( 32, 64, 128 are your options)

// Construct scale objects
byte DINS1[CHANNEL_COUNT] = {DIN1, DIN2, DIN3, DIN4};
HX711MULTI scales1(CHANNEL_COUNT, DINS1, CLK1, GAIN);

byte DINS2[CHANNEL_COUNT] = {DIN5, DIN6, DIN7, DIN8};
HX711MULTI scales2(CHANNEL_COUNT, DINS2, CLK2, GAIN);

HX711MULTI scales[NUM_GROUPS] = {scales1, scales2};      // Add all scale instances to an array so we can iterate over them

// Program Variables
long raw[NUM_GROUPS * CHANNEL_COUNT];      // Array to hold raw samples collected from each sensor
uint16_t sendSize;
int i;

SerialTransfer myTransfer;

void setup() {
  Serial.begin(115200);
  myTransfer.begin(Serial);
}

void loop() {
  //Read sensor values into an array
  scales[0].readRaw(raw);                  // Read values from first 4 sensors directly into the raw array
  scales[1].readRaw(raw + 4);           // Read values from last 4 sensors directly into the raw array
  // contents of raw looks like: {-180279, -160207, 57267, -238712, -198426, -232254, 14375, -176826}

  // Transfer the contents of raw to Python via serial
  sendSize = myTransfer.txObj(raw, sendSize);
  myTransfer.sendData(sendSize);

  // Loop around, read new values from sensors, and send new values.
}

Hello!

The Example Python Script works well with Arduino Nano 33 BLE Sense.

However, when I try to modify it to send an array of floats, using the "Send a list" way, the tx_obj() -method returns the following error:

Traceback (most recent call last):
  File "C:\Users\Klupi\AppData\Local\Temp\ipykernel_25544\2144615584.py", line 19, in <module>
    list_size = link.tx_obj(list_)
  File "C:\Users\Klupi\miniconda3\envs\deeplearning\lib\site-packages\pySerialTransfer\pySerialTransfer.py", line 272, in tx_obj
    start_pos = self.tx_obj(el, start_pos)
  File "C:\Users\Klupi\miniconda3\envs\deeplearning\lib\site-packages\pySerialTransfer\pySerialTransfer.py", line 289, in tx_obj
    self.txBuff[index + start_pos] = val_bytes[index]
IndexError: list assignment index out of range

Why is this, what am I doing wrong? My sample array is as follows:

[-0.27273, -0.33333, 0.066667, -0.10638, -0.27273, -0.33333, -0.066667, -0.10638, -0.045455, -0.28889, -0.022222, 0.021277, -0.045455, -0.46667, -0.2, 0.10638, -0.045455, -0.2, 0.066667, 0.14894, -0.045455, -0.33333, 0.42222, 0.14894, -0.59091, -0.33333, 0.33333, 0.021277, -0.59091, -0.28889, 0.066667, -0.31915, -0.59091, -0.28889, -0.55556, 0.021277, -0.59091, -0.24444, -0.066667, 0.021277, -0.18182, 0.066667, 0.11111, 0.14894, -0.18182, 0.066667, 0.066667, 0.14894, -0.18182, 0.022222, -0.022222, -0.021277, -0.18182, 0.066667, 0.066667, -0.021277, 0.13636, 0.066667, -0.022222, -0.021277, 0.36364, -0.33333, 0.022222, -0.021277, 0.090909, -0.066667, -0.33333, -0.44681, 0.090909, -0.42222, -0.2, -0.19149, 0.090909, -0.11111, -0.11111, 0.10638, 0.090909, -0.24444, 0.022222, -0.021277, 0.090909, -0.066667, -0.066667, -0.021277, 0.090909, 0.066667, 0.066667, -0.021277, 0.090909, -0.022222, -0.2, 0.021277, 0.090909, -0.46667, -0.33333, 0.021277, 0.090909, -0.066667, -0.33333, -0.3617, 0.090909, -0.066667, -0.2, -0.3617, 0.090909, -0.066667, -0.33333, -0.48936, 0.090909, -0.24444, -0.2, -0.48936, -0.18182, -0.68889, -0.55556, -0.3617, -0.18182, -0.28889, -0.33333, -0.3617, -0.090909, -0.022222, -0.42222, -0.3617, 0.0, -0.24444, -0.86667, -0.3617, -0.045455, -0.2, -0.2, -0.10638, -0.045455, -0.066667, -0.33333, -0.10638, 0.090909, -0.11111, -0.33333, -0.65957, 0.090909, 0.2, -0.73333, -0.3617, 0.090909, 0.33333, -0.73333, -0.3617, 0.13636, 0.066667, -0.91111, -0.3617, 0.22727, -0.15556, -0.68889, -0.44681, 0.22727, 0.24444, -0.55556, -0.44681, 0.090909, 0.28889, -0.6, -0.44681, 0.63636, 0.11111, -0.42222, -0.74468, 0.63636, 0.066667, -0.6, -0.74468, 0.72727, 0.33333, -0.37778, -0.61702, 0.72727, -0.066667, -0.6, -0.61702, 0.81818, 0.2, -0.77778, -0.61702, 0.81818, 0.2, -0.73333, -0.61702, 0.77273, 0.066667, -0.95556, -0.61702, 0.77273, -0.2, -0.82222, -0.61702, 0.81818, 0.022222, -0.95556, -0.61702, 0.77273, 0.066667, -0.6, -0.82979, 0.77273, 0.33333, -0.73333, -0.82979, 0.77273, -0.37778, -0.86667, -0.82979, 0.27273, -0.066667, -0.68889, -0.3617, 0.27273, -0.066667, -0.68889, -0.70213, 0.27273, 0.15556, -0.6, -0.70213, 0.27273, 0.33333, -1.0, -0.70213, 0.77273, -0.15556, -0.82222, -0.61702, 0.77273, 0.6, -1.0, -0.61702, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

Interestingly, sending a smaller array such as [1.2433333333334, 2.34222222221, 3.4255555555555] works fine. How would you proceed on sending the bigger array to Arduino?

Many thanks!

Hello powrebroker

First of all really big thanks to your awesome library.

As I'm using your library for transmitting simulator data (python) to arduino(mega) CRC Error keep occur and pySerialTransfer Thread keep waiting for timeout and send CRC Error. I lower the baud rate to 9600 and provide some delay to work properly

here Is my code

Hello, I am building mobile robot based on nvidia jetson nano and arduino. I have two motor encoders connected to arduino so I need to send data to jetson nano (python) via uart. What is the best and right way to do it with this library? Approximate range of encoder values is -1000000 - 1000000. I will appreciate your help.

Hello PB2 and a big thank you for your work!

I'm trying to trigger some action in the Arduino Leonardo side when a specific char is received from python. I could have use int instead of char message but I take this as an exercice :)

All the manipulation I tried failed... I'm starting to be tired of this and probably missing some perspective from this point.

So, here my Python:

X1 = [0, 0, 0, 0, 0, ... , 0]
X2 = [1, 1, 1, 1, 1, .... , 1] ---> X1 and X2 are long array

try:
    link = txfer.SerialTransfer('COM8', 115200, debug=True)
    link.open()

    start = 0
    step = 10
    counter = 0

    while counter < len(X1):
        sendSize = 0

        X1_= X1[start:start+step] ---> I split it in order to get under 254 bytes per packet.
        X2_= X2[start:start+step]
        message = "notes"
        
        messageSize = link.tx_obj(message) 
        sendSize += messageSize

        X1Size = link.tx_obj(X1_, sendSize) - sendSize
        sendSize += X1Size

        X2Size = link.tx_obj(X2, sendSize) - sendSize
        sendSize += X2Size

        link.send(sendSize, packet_id=0) ---> I recently tried to add packet_id information to don't messing up the receiving part.
        counter += len(X1_)

        #print('SENT: {} {} {}'.format(message, X1_, X2_))

        while not link.available():
            pass

        rec_message = link.rx_obj(obj_type=type(message), obj_byte_size=messageSize)
        rec_X1 = link.rx_obj(obj_type=type(X1_), obj_byte_size=X1Size, list_format='i', start_pos = messageSize)
        rec_X2 = link.rx_obj(obj_type=type(X2_), obj_byte_size=X2Size, list_format='i', start_pos = messageSize + X1Size)

        start += step
        print("Recu:\t{} - {} et {}".format(rec_message, rec_X1, rec_X2))


    while True:
        sendSize2 = 0
        message2 = "click"
        messageSize = link.tx_obj(message2)
        sendSize2 += messageSize

        print("Ready to click...")
        link.send(sendSize2, packet_id=0)
        #print('SENT: {}'.format(message2))

        while not link.available():
            pass

        rec_message = link.rx_obj(obj_type=type(message2), obj_byte_size=messageSize)
        print("Recu:\t{}".format(rec_message))

        raise KeyboardInterrupt

except KeyboardInterrupt:
    try:
        link.close()
    except:
        pass

except:
    import traceback
    traceback.print_exc()

    try:
        link.close()
    except:
        pass

And my Arduino:

#include "SerialTransfer.h"
#include "Keyboard.h"
#include "Mouse.h"

SerialTransfer myTransfer;

int32_t X1[10];
int32_t X2[10];

int32_t X1G[] = {};
int32_t X2G[] = {};

char message[5];

#Stuff I tried....
char notes[] = "notes";
char clickOk[] = "click";

void setup()
{
  Keyboard.begin();
  Mouse.begin();
  Serial.begin(115200);
  myTransfer.begin(Serial, true);
}

void loop()
{
  if(myTransfer.available())
  {
    uint16_t recSize = 0;
    recSize = myTransfer.rxObj(message, recSize);

    if (strcmp(message, clickOk) == 0) ---> Here my problem. How to check the received message and do action?
    {
      recSize = myTransfer.rxObj(X1, recSize);
      recSize = myTransfer.rxObj(X2, recSize);
  
      memmove(X1G + sizeof(X1G), noteTime, sizeof(X1));
      memmove(X2G + sizeof(X2G), noteType, sizeof(X2));
  
      uint16_t sendSize = 0;
      
      sendSize = myTransfer.txObj(message, sendSize);
      sendSize = myTransfer.txObj(X1, sendSize);
      sendSize = myTransfer.txObj(X2, sendSize);
      
      myTransfer.sendData(sendSize);
     
      delay(50);
    }
    else if (strcmp(message, clickOk) == 0)
    {
      uint16_t sendSize2 = 0;
      sendSize2 = myTransfer.txObj(message, sendSize2);
      myTransfer.sendDatum(sendSize2);
      Mouse.click();
    }
  }
}

My FTDI card will be coming soon, it should be easier for debugging, but in the meantime any help will be appreciated :)

I'm a beginner in this wide world. Maybe my mechanism is completely broken, so feel free to suggest another solution to get what I want.

Bye, Vouncat

Hello PowerBroker2,

We are a high school robotics team that came across your code. One of our goals is to run serial communication between Python on a laptop and an Arduino over RS485. We have tried your pySerialTransfer code to verify functionality.

We are able to get Python to send and receive data while using the USB connection to the Arduino, but not over RS485. At first, we thought our RS485 communication could be a hardware issue. However, all of our hardware appears to work individually.

1. We have tested our USB to RS485 adapter using a second adapter and two instances of "AccessPort" from Serialcomm.com. We are able to communicate between two laptops with this method.

2. We replaced the USB to RS485 adapters using two Arduinos connected through their serial pins. The Arduinos both run an "empty" sketch with pins 0 and 1 set as inputs. We are able to communicate between two laptops with AccessPort as above.

3. We have tested two Arduinos communicating over RS485 using TTL to RS485 converters. These work as well.

4. We have tried using a USB to RS485 adapter with A, B and ground running to the TTL to RS485 converter. We are using Serial1 for RS485 communication and Serial to send debug statements to the Serial Monitor. The code never enters the "if(myTransfer.available())" loop on the Arduino.

5. To try to simplify the connections, our latest test was to eliminate the RS485 connections. We returned to using two Arduinos. One Arduino used the setup described in "2" above to serve as a USB to TTL converter. The second Arduino ran your sample code. The result is the same as in "4" above. In both of these cases, we modified the code to use Serial1. After the "if(myTransfer.available())" we added "else Serial.println("Not available");" to give us feedback.

After reading through several web pages that you have offered advice on, we came across issue #32 (https://github.com/PowerBroker2/pySerialTransfer/issues/32). We tried several of the code examples you provide there with no luck.

Software is our weakest area this year. The addition of Python has made it more complicated for us. We apologize if there is a simple solution that we are missing.

We are not understanding much of the code as it relates to SerialTransfer and pySerialTransfer, so we do not know how to further analyze the "if(myTransfer.available())" functionality. We have started reading through "Serial Input Basics" and your "Serial Input Advanced" tutorials on the Arduino website.

Another issue that came up for us was in trying one of your examples shared here: (https://forum.arduino.cc/t/arduino-to-arduino-serial/621141/15). Our Arduino would not compile your code and gave us the following error: "class SerialTransfer' has no member named 'txBuff'"

With all that said, we are wondering if you have any suggestions since pySerialTransfer seems like it would be helpful to us.

Thanks in advance for your time.

Hi im got a problem with float on python side (RX), got a this kind of output: b'P'2.1579996350602183e-43 When im sending from uC: testStruct.z = 'P'; testStruct.y = 4.55; Betwen uC is no problem work like a harm, on python floats crazy. 9600 baud / Python 3.9.2 / Debian

Hello,

thank you for this great package!

I wanted to understand your code and manipulated some lines in CRC.py.

But with your example Python script, nothing printed and I didn't get a Python error. After a while, I realized that:

CRC_ERROR = 0

So I think in the example script it should be:

if link.status <= 0:

 while not link.available():
                if link.status <= 0:
                    if link.status == txfer.CRC_ERROR:
                        print('ERROR: CRC_ERROR')
                    elif link.status == txfer.PAYLOAD_ERROR:
                        print('ERROR: PAYLOAD_ERROR')
                    elif link.status == txfer.STOP_BYTE_ERROR:
                        print('ERROR: STOP_BYTE_ERROR')
                    else:
                        print('ERROR: {}'.format(link.status))

Kind regards, 8sawte8

bytes() is its own type in python3, and is common enough IMHO that pySerialTransfer should support it. Since bytes() do not need packing with struct.pack(), the proposed solution is to change the logic in txObj to check for type(val) == bytes and not attempt to pack it.

Currently, byte strings must be unpacked into ints then passed in with one call to txObj per byte, giving val_type_override as: send_size = link.tx_obj(self.left_drive_pct, val_type_override='b')

The alternative - to send values as ints and convert them to bytes on the Arduino side, would consume 4X the bandwidth on the serial link.

Thoughts? I am willing to contribute this fix in a pull request if you support the idea.

On multi-byte wide processors the compiler aligns data items to their native alignment, unless directed otherwise. Thus a float is aligned to a 4-byte boundary. Therefore the testStruct in tx_data is laid out in memory differently between an Arduino Uno & a board like an esp32. Since txObj uses sizeof(T) as the number of bytes to transmit, tx_data sends a different number of bytes from esp32 compared to Uno.

testStruct is shown below with the fix needed for processors with > 8-bit wide data bus struct STRUCT { char z; float y; //} testStruct; } attribute((packed)) testStruct;

The original code (commented out line 4 above) causes 3 empty bytes to be placed after z, and before y on esp32, so txObj sends 8 bytes on an esp32, but only 5 on an Arduino Uno. pySerialTransfer is unaware of the packing difference, so tries to pull the float out of bytes 1-4 (counting byte 0 as z), regardless of the sender-architecture.

The example as provided prints the following correct message on pySerialTransfer + Uno: b'$'4.5 | hello but the following error message on pySerialTransfer with esp32, caused by pySerialTransfer reading the float out of bytes 1-4 then trying to read the char array out of byte 5, which is actually the 2nd byte of the float.: Traceback (most recent call last): File "./rx_data.py", line 30, in arr = link.rx_obj(obj_type=str, File "/home/bouchier/.local/lib/python3.8/site-packages/pySerialTransfer/pySerialTransfer.py", line 359, in rx_obj unpacked_response = unpacked_response.decode('utf-8') UnicodeDecodeError: 'utf-8' codec can't decode byte 0x90 in position 1: invalid start byte

Removing the string transfer part of the example prints the following incorrect message on pySerialTransfer + esp32, : b'$'0.0 | Note the float, which should be 4.5, is 0.0 because pySerialTransfer is reading the float out of the 3 empty bytes plus first byte of the float.

The correct way to eliminate machine/compiler dependent packing is with the packed attribute as shown on line 5 of testStruct above. Making this change produces the correct output on both Uno & esp32: b'$'4.5 | hello

Are you open to pull requests to fix the examples so they work on both Uno & esp32? I would test any such changes on both.

Hey PowerBroker - nice library! Thanks very much - it will save me much work compared with PacketSerial.

The last checkin comment for examples/datum/Arduino/ was that it fixed float precision. But Arduino declares doubles in the datum and data examples, but the python side reads it as 'f' (float). Double size is apparently platform-dependent; ghe Arduino language reference says double is 4 bytes (same as float) on Uno and ATMEGA but 8 bytes on Due. I am testing it on an esp32, where apparently doubles are 8 bytes, because I had to change the format specifier in the python example code to 'd' to get it to read the data from esp32-Arduino correctly.

I would think you should make the types match on the Arduino & python sides. Is there some deeper issue here that made you declare Due unsupported? Would such an issue also apply to esp32?

And on the subject of esp32, does the design take care to keep tx & rx logic in Arduino & python (and everything they use) separate such that a receive thread and a transmit thread could run through the classes simultaneously and not step on each other?

Hey PowerBroker!

Things are moving along pretty smoothly with our project! I can send trial sets of 45 or smaller no problem and the lab is pretty happy with it. A new challenge has appeared for me, though. The lab would like it to be possible to double the amount of trials to 90 total as a definite ceiling.

I had thought that I was doing this correctly with what you have taught me so far but, after some testing, I've come to realize that while a second packet is getting sent by Python, my Arduino code doesn't seem to collecting the second packet's content. Here's an example of what I mean:

(bruker_control) C:\Users\jdelahanty>python Documents\gitrepos\headfix_control\bruker_control\bruker_control.py

First Half of Trials Sent
[1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0]
First Half of Trials Received
[1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0]
First Half Trial Array Transfer Successful!
Second Half of Trials Sent
[0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1]
Second Half of Trials Received
[1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0] # <--- Arduino sends first packet's contents!
Second Half Trial Array Transfer Failure!
Exiting...
Traceback (most recent call last):
  File "Documents\gitrepos\headfix_control\bruker_control\bruker_control.py", line 522, in serial_transfer_trialArray
    sys.exit()
SystemExit
Experiment Over!
Exiting...

Any advice? I had thought it would collect the packet independently but I'm clearly misunderstanding how multiple packets for the same object work. Is this something I can do even or should I create a second array in the Arduino code to store the second half of trials?

Here's some full Python and Arduino code: Python

#### Trial Array Generation ####
# Import scipy for statistical distributions
import scipy
# Import scipy.stats truncated normal distribution for ITI Array
from scipy.stats import truncnorm
# Import numpy for trial array generation/manipulation and Harvesters
import numpy as np
# Import numpy default_rng
from numpy.random import default_rng

# -----------------------------------------------------------------------------
# Functions
# -----------------------------------------------------------------------------
#### Trial Generation ####
# Random Trials Array Generation
def gen_trial_array(totalNumberOfTrials):
    # Always initialize trial array with 3 reward trials
    trialArray = [1,1,1]
    # Define number of samples needed from generator
    num_samples = totalNumberOfTrials - len(trialArray)
    # Define probability that the animal will receive sucrose 60% of the time
    sucrose_prob = 0.5
    # Initialize random number generator with default_rng
    rng = np.random.default_rng(2)
    # Generate a random trial array with Generator.binomial
    # Use n=1 to pull one sample at a time, p=.6 as probability of sucrose
    # Use num_samples to fill out accurate number of trials
    # Use .tolist() to convert random_trials from np.array to list
    random_trials = rng.binomial(
    n=1, p=sucrose_prob, size=num_samples
    ).tolist()
    # Append the two arrays together
    for i in random_trials:
        trialArray.append(i)

    if len(trialArray) > 45:
        split_array = np.array_split(trialArray, 2)
        first_trialArray = split_array[0].tolist()
        second_trialArray = split_array[1].tolist()
    else:
        first_trialArray = None
        second_trialArray = None

    ## TODO: Write out the trial array into JSON as part of experiment config
    # Return trialArray or halves of trial arrays (if needed)
    return trialArray, first_trialArray, second_trialArray

#### Serial Transfer ####
# Import pySerialTransfer for serial comms with Arduino
from pySerialTransfer import pySerialTransfer as txfer


# Trial Array Transfer
def serial_transfer_trialArray(trialArray, first_trialArray, second_trialArray):
    # Check if two packets are necessary. If 'first_trialArray' is None,
    # the message is small enough to fit in one packet. There's no need to
    # check the second_trialArray as it too will be None by design in this case.
    if first_trialArray == None:
        try:
            # Initialize COM Port for Serial Transfer
            link = txfer.SerialTransfer('COM12', 115200, debug=True)

            # Send the trial array
            # Initialize trialArray_size of 0
            trialArray_size = 0
            # Stuff packet with size of trialArray
            trialArray_size = link.tx_obj(trialArray)
            # Open communication link
            link.open()
            # Send array
            link.send(trialArray_size, packet_id=0)

            print(trialArray)

            while not link.available():
                pass

            # Receive trial array:
            rxtrialArray = link.rx_obj(obj_type=type(trialArray),
            obj_byte_size=trialArray_size, list_format='i')

            print(rxtrialArray)

            if trialArray == rxtrialArray:
                print("Trial Array transfer successful!")

            else:
                link.close()
                print("Trial Array error! Exiting...")
                sys.exit()

            # Close the communication link
            link.close()

        except KeyboardInterrupt:
            try:
                link.close()
            except:
                pass

        except:
            import traceback
            traceback.print_exc()

            try:
                link.close()
            except:
                pass

    elif first_trialArray != None:
        try:
            # Initialize COM Port for Serial Transfer
            link = txfer.SerialTransfer('COM12', 115200, debug=True)

            # Send the first half of trials with packet_id = 0
            first_trialArray_size = 0
            first_trialArray_size = link.tx_obj(first_trialArray)
            link.open()
            link.send(first_trialArray_size, packet_id=0)

            print("First Half of Trials Sent")
            print(first_trialArray)

            while not link.available():
                pass

            # Receive the first half of trials from Arduino
            rxfirst_trialArray = link.rx_obj(obj_type=type(first_trialArray),
            obj_byte_size=first_trialArray_size, list_format='i')

            print("First Half of Trials Received")
            print(rxfirst_trialArray)

            # Confirm packet was sent correctly
            if first_trialArray == rxfirst_trialArray:
                print("First Half Trial Array Transfer Successful!")
            else:
                link.close()
                print("First Half Trial Array Transfer Failure!")
                print("Exiting...")
                sys.exit()

            # Send second half of trials with packet_id = 0
            second_trialArray_size = 0
            second_trialArray_size = link.tx_obj(second_trialArray)
            link.send(second_trialArray_size, packet_id=1)

            print("Second Half of Trials Sent")
            print(second_trialArray)

            # Receive second half of trials from Arduino
            rxsecond_trialArray = link.rx_obj(obj_type=type(second_trialArray),
            obj_byte_size=second_trialArray_size, list_format='i')

            print("Second Half of Trials Received")
            print(rxsecond_trialArray)

            if second_trialArray == rxsecond_trialArray:
                print("Second Half Trial Array Transfer Successful!")
            else:
                link.close()
                print("Second Half Trial Array Transfer Failure!")
                print("Exiting...")
                sys.exit()

            link.close()

        except KeyboardInterrupt:
            try:
                link.close()
            except:
                pass

        except:
            import traceback
            traceback.print_exc()

            try:
                link.close()
            except:
                pass
    else:
        print("Something is wrong...")
        print("Exiting...")
        sys.exit()

Arduino

// Use package SerialTransfer.h from PowerBroker2 https://github.com/PowerBroker2/SerialTransfer
#include "SerialTransfer.h"

// Rename SerialTransfer to myTransfer
SerialTransfer myTransfer;

const int MAX_NUM_TRIALS = 46; // maximum number of trials possible

int32_t trialArray[MAX_NUM_TRIALS]; // create trial array

boolean acquireTrials = true;

void setup()
{
  Serial.begin(115200);
  Serial1.begin(115200);

  myTransfer.begin(Serial1, true);
}


void loop(){
  trials_rx();
}

int trials_rx() {
  if (acquireTrials) {
    if (myTransfer.available())
    {
      myTransfer.rxObj(trialArray);
      Serial.println("Received Trial Array");

      myTransfer.sendDatum(trialArray);
      Serial.println("Sent Trial Array");

      acquireTrials = false;
    }
  }
}