How to set up a robocar platform with a remote control unit

At itemis, we are involved in automotive software projects in terms of modeling (domain specific languages for architecture and behavior), tooling (architecture, feature models, implementation, machine learning) and concepts/standards (AUTOSAR, Genivi, openADX). At our office in Stuttgart, we are setting up a tangible demonstrator – a robocar platform as a flexible base with an initial showcase of machine learning.

The first version of the car was based on the Sunfounder platform. Our good experiences quickly lead to further ideas and plans, and we ran into the limits of the Sunfounder platform. So we decided to set up a new demonstrator platform based on a Traxxas Slash 4x4 Platinum.

The current milestone

The current goal of the project is to get control of the RC Car with a remote control unit (XBOX360 controller) and to capture video frames from an racing track via the SainSmart 5MP camera module with 175 degree panoramic wide angle. The captured images will run through image processing and will serve as an input to an A.I. algorithm of behavioral cloning. The A.I. also receives the steering angle of the RC Car, and it will be trained to understand the behaviour of steering on a predefined track. So the RC Car should be able to steer autonomously through the whole track.

The RC Car Traxxas Slash 4x4 Platinum Edition (Model 6804R)

The project is based on the Traxxas Slash 4x4 Platinum Edition (Model 6804R) RC racing truck with the original Traxxas NiMH Power Cells with a capacity of 3000 mAh. Its drive system constists of the Traxxas 3355R Velineon VXL-3s waterproof Electronic Speed Control (ESC), which is used to control the connected Traxxas Velineon brushless DC motor.

Not included was the radio system (transmitter and receiver). We bought it for the initialization sequence of the ESC and to analyze the necessary PWM duty cycles. It also came with a digital and waterproof Traxxas 2075 servo, which has a maximum rotation angle of 180° and is intended to use for steering the RC Car. The following picture shows the out-of-the-box state of the car with the above-mentioned important basic components.

Additional components needed to run the RC Car

We also needed some additional components to make the RC Car running for our needs:

• The control unit is the Linux-based single-board computer Raspberry Pi 3 with an Raspbian Stretch Image flashed on its SD card.
  
  
  [Source: https://www.raspberrypi.org/products/raspberry-pi-3-model-b-plus/]
  
  
• We are using two PCA9685 16-Channel 12-Bit PWM Servo Driver from Sunfounder to get control of the ESC and the steering servo. Later on, we will explain why we are using two of these modules.
  
  
  [Source: https://www.sunfounder.com/pca9685-16-channel-12-bit-pwm-servo-driver.html]
  
  
• A DC-DC module that converts a DC power input of 6.5V to 40V to an output of 5V and 2A is also used. It is needed for the input voltage of the PCA9685 modules and is also used to provide 2x5V input for each PCA9685 module.
  
  
  [Source: https://www.sunfounder.com/step-down-dc-dc-converter-module-for-raspberry-pi.html]
  
  
• Furthermore, we are using an original XBOX360 controller for PC purpose with a Bluetooth-to-USB dongle for manual steering and throttle input.
  
  
• The SainSmart 5MP camera module with 175 degree panoramic wide angle for Raspberry Pi 3 is intended to capture video frames for image processing and provides input for the A.I. algorithm.
  
  
  [Source: https://www.sainsmart.com/products/noir-wide-angle-fov160-5-megapixel-camera-module]
  
  
• It is also necessary to have a portable power source, so that the car can drive freely. We bought an Anker Power Core with a capacity of 13000 mAh.
  
  
• It is also recommended to have wireless access to the Raspberry Pi 3. For example, we are using a separate router from TP-Link, and we are able to get remote access to the Raspberry Pi 3 via the VNC Viewer and VNC Server applications. This is easy to install, and there are several online tutorials. It is also very helpful when going to other locations for a Robocar race: We do not have to rely or configure any other Wifi connections, and we can rely on the router giving out static addresses for our devices.

Assembling the components

We assembled and connected all of the components as shown in the following picture:

At this point, we could power up the whole system. First power up the Raspberry Pi 3 with the Anker Power Core and let the system boot. Afterwards, you can plug in the Traxxas Power Cell and press the EZ Button on the ESC until the LED is blinking green. If you have not already done the initialization process, you first need to do the following step.

Initialization process

There are two ways to initialize the ESC to get it controlled with a PWM signal. The initialization sequence needs to register the throttle signal range for driving forwards and backwards. For the initialization sequence it is easier to use the original radio system (receiver and transmitter), which is compatible to the ESC, and follow the instructions from the delivered quick-start guide.

1. Press and hold the Set button on the transmitter while it is not powered on.
2. Turn on the transmitter until the red LED blinks.
3. Press and hold the Link button on the receiver before pressing the EZ button on the ESC until both LEDs, on the transmitter and on the receiver, are flashing green.
4. Calibration mode is now started: If the ESC LED blinks red once, then you need to pull the throttle to full range und hold it until the LED blinks red twice. Then you need to push the throttle to full range and hold it until it blinks red again.
5. If the calibration process was successful, the ESC LED blinks green.
6. Now the initialization process has finished successfully.

This process is quite difficult to manage without the appropriate equipment. But fortunately, you only have to do this process once, unless you want to change the operation mode of the car.
We are using the training mode with the 50:50 range of the throttle to lower the velocity.

Analysis of the PWM duty cycles with an Arduino Uno

After the initialization process is done, we need to figure out the duty cycle in milliseconds of the PWM signals, which are used to steer the servo and to control the throttle.

You can use an oscilloscope to observe the signals, or you can use an Arduino Uno and the following sketch for the Arduino IDE. We chose the latter option, because we had an Arduino Uno in stock.

byte PIN = 3;
int pwm_value;

void setup() {
  pinMode(PIN, INPUT);
  Serial.begin(9600);
}

void loop() {
  pwm_value = pulseIn(PIN, HIGH);
  Serial.println(pwm_value);
}

In this case, you also need the radio system (receiver and transmitter) to observe the signals that have previously been sent from the transmitter to the receiver.

Here is an explanation of how to connect the Arduino Uno to the transmitter and how to read the signals correctly.

First, we had the same results for the duty cycles for the transmitter as shown in that tutorial. But later on, in our software, we observed a small delay caused by functions in the PCA9685 library. A small inaccuracy can have a remarkable impact on the throttle control. Therefore, it is also recommended to observe the generated PWM duty cycles from the used software. You can use the following modified Arduino sketch to figure out the duty cycles of the PCA9685 module for steer and throttle.

You can also compare the transmitter signal to the generated PWM signal from the PCA9685 module. Just additionally connect the PCA9685 module's PWM pin to the PWM IN Pin 5 of the Arduino Uno, and make a ground connection between the PCA9685 module and the Arduino Uno.

byte PIN = 3;
byte PIN_Rpi = 5;

int pwm_value, pwm_raspi_value;

void setup() {
  pinMode(PIN, INPUT);
  pinMode(PIN_Rpi, INPUT);
  Serial.begin(9600);
}
void loop() {
  pwm_value = pulseIn(PIN, HIGH);
  Serial.println((String)"PWM signal from Traxxas Transmitter\t" + pwm_value);
  pwm_raspi_value = pulseIn(PIN_Rpi, HIGH);
  Serial.println((String)"PWM signal from pca9685 module\t" + pwm_raspi_value);
}

Setting up the Raspbian image

We are using the Raspbian Stretch image version which is shown in the following terminal output.

It also has some additional software installed:

1. Activated WiFi (several tutorials can be found online)
2. wiringPi (it was already installed, but it is recommended to check that)
3. wiringPiPca9685 (library for the pca9685 module based on wiringPi)
4. Joystick API (used for gaming device input)
5. OpenCV Version 3.2.0 (intended for camera usage and image processing)
6. Eclipse 3.8 Juno with CDT plugin (IDE for C/C++ code)
7. VNC Server on Rpi 3 and VNC Viewer on PC (intended for remote access to the Rpi3)

After these installations, which can take some time, the image is ready to use. Otherwise, you can use the attached image and flash it on your SD card. Make sure you have at least a 16GB SD card.

Importing the project into the Eclipse IDE

The project can be imported to the Eclipse IDE by the following steps:

1. Open the Eclipse IDE and select File → New → Makefile Project with Existing Code.
   
   

2. Type in the project name and search for the path containing the source code. Then click on Finish.
   
   

Explanation of the project content

Inside the project folder there are several C header and source files. There is also a CMakeLists.txt file which comprises the instructions to build the project, to generate object files, to link the libraries, and to produce the executable. The .project file contains project-specific instructions, which are read when importing the source code to Eclipse IDE. The images folder will contain the captured images for image processing and as an input path for the A.I. algorithm.

Explanation of the general function of the sourcefiles – for more specific explanations look up the source code and the comments:

1. CalculateAngle: Contains two functions which are used to calculate and map angle values.
   
   int calculateSteeringAngle(float mapped_axis_value);
   
   This function maps a duty cycle value to a servo angle and maps the servo angle to the real steering angle of the Traxxas RC Car.
   
   float convertReceivedSteeringAngle(int steeringAngle);
   
   This function converts a received angle (int) from the UDP socket to a duty cycle between [0 .. 100%].
2. CaptureVideo: Contains two functions which are used to open the camera and to capture and save images from the SainSmart camera module.
   
   int initVideo();
   
   Opens the camera module plugged to the camera serial Interface via the 4l2 camera driver.
   
   void *captureVideo();
   
   This function captures a video stream for further image processing. It is implemented as a pthread function. The video capturing can be processed in parallel to the other tasks, because it takes some time to capture and save images and this would have an impact on the performance of the program.
3. Joystick: Contains three functions which open, close, and read the XBOX360 input events via the Joystick API
   
   void Joystick_Init();
   
   Opens the joystick device on the specified path in the filesystem and provides read access in non-blocking mode.
   
   int read_joystick_event(struct js_event *jse);
   
   This function reads in a joystick event and checks its length. An event has to consist of 8 bytes. The content of an event is described by the js_event struct and is defined in the joystick.h file. The input event is saved to a struct js_event, which is passed as an argument to this function. The function returns the number of bytes that have been read from joystick input path /dev/input/js0.
   
   void close_joystick();
   
   Closes the joystick input device on exit of the program.
4. Motor: This file contains seven functions which are responsible for the value mapping for the servo and throttle and the conversion from the value mapping to a pwm duty cycle.
   
   float map_axis_servo(int axis_value);
   
   This function maps the left stick of the Xbox 360 controller to a value in the range [-0.5 .. 0.5]. This represents the duty cycle for left and right steering range in percent, considering the value of 0.5 being the neutral position. So either there is an addition or difference of [0 .. 0.5] to the neutral value. Furthermore, it is possible to change the first axis value on which the axis should react with steering, and you can also increase and decrease the possible maximum steering angle with the macros described in motor.h file.
   
   void steerVehicle(int fd, short value);
   
   This function receives the mapped steering axis value from the previous function as a float value and maps this value to a duty cycle value in milliseconds. After calculating the amount of necessary ticks for a 12-bit channel resolution, the tick value will be written on the I2C1 bus with the PCA9685 module (slave address 0x41) connected and the predefined channel base 0 for the servo motor.
   
   It also uses the int calculateSteeringAngle(float mapped_axis_value) function to calculate the current steering angle for the A.I. process. It is necessary to make a read/write protection for this global resource, because one or more threads need to have an access to this resource.
   
   void steerVehicleAutonomous(int fd);
   
   This function is quite similar to the previous function, but is only used in the state of autonomous driving. It is also responsible for steering, but it does not receive its input from the joystick device but from a UDP Socket, which is connected to the A.I. process. There is an additional global resource which contains the steering angle received by the UDP Socket and therefore it also needs to be protected because there is an access to this resource from one or more other processes.
   
   float map_axis_throttle_forwards(int axis_value);
   
   This function is used to map the input from the RT axis of the Xbox 360 controller to a float value in the range [0.57 .. 1.00]. The lower value is the neutral position of the throttle, and this state remains until the axis value reaches the value 0, which will happen if the button is half-pressed. The maximum upper range is defined in the motor.h file by the THROTTLE_RANGE_FORWARDS macro, which divides the 100% of the forward throttle value by the factor of the macro value. So, a macro value of 20 results in a maximum upper range of 5% added to the neutral_offset, which in this case is the lowest possible value for the car to move forwards. But it is still around 6mph.
   
   void driveVehicle_forwards(int fd, short value);
   
   This function receives the mapped throttle axis value from the previous function as a float value and maps this value to a duty cycle value in milliseconds. After calculating the amount of necessary ticks for a 12-bit channel resolution, the tick value will be written on the I2C1 bus with the PCA9685 module (slave address 0x40) connected and the predefined channel base 4 for the ESC.
   
   float map_axis_throttle_backwards(int axis_value);
   
   This function is quite similar to the float map_axis_throttle_forwards(int axis_value) function. The differences are that it uses the LT axis of the Xbox 360 controller and a float value in the range [0.00 .. 0.56]. The upper value is the neutral position of the throttle, and this state remains until the axis value reaches the value 0, which will happen if the button is half-pressed. The minimum range is also defined in the motor.h file by the THROTTLE_RANGE_BACKWARDS macro, which has the same function as the macro in the forwards mapping function. In this case, the value will be subtracted from the neutral_offset. So, subtracting 5% from the neutral_offset will result in the lowest possible value for the car to move backwards.
   
   void driveVehicle_backwards(int fd, short value);
   
   Similar to the void driveVehicle_forwards(int fd, short value) function, but it receives the mapped axis value from the float map_axis_throttle_backwards(int axis_value) function.
5. mySocket: This file contains two functions, firstly to initialize the socket and secondly to listen on the local loopback address and a port for UDP datagrams.
   
   void init_socket();
   
   This function opens a socket for UDP/IP communication with UDP datagrams.
   
   void *receiveAngle();
   
   This function calls void init_socket() to open the socket. It also configures and binds the previously-opened socket. It is implemented as a thread, so it can do the task of listening to the UDP port without interrupting the whole process. This thread will only work if the autonomous driving state is active, otherwise the thread will be cancelled. It is also necessary to protect two global resources for read/write access. On the one hand, it is the button state which can change between autonomous driving state and manual state. On the other hand it protects the received angle from the UDP socket for the steering function used in the autonomous state.
6. PWM: This file contains three functions that are used to initialize the two PCA9685 modules and to generate an appropriate PWM signal.
   
   int calcTicks(float impulseMs, int hertz, int on_off);
   
   This function transforms a duty cycle in milliseconds to an aquivalent tick value of 12-bit resolution with a given frequency for the PCA9685 module output to generate a PWM signal.
   
   float map(float input, float min, float max);
   
   This mapping function maps the input [0 .. 1] to a maximum and minimum value which are given as an argument and returns a resulting float value, which is used as the first argument for the previously-mentioned function.
   
   int PCA9685_Init(const int i2cAdress, const int pinBase, const int freq);
   
   This function initializes each PCA9685 module with its own I2C adress, pin base, and frequency.
7. timeMeasurement: These functions are only used to analyze time consumption for processes. There is no functional task for the program.
8. xbox360: This file contains two functions. The first function is a get helper function to access a static C variable from a C++ file. The second function is a pthread function and is intended to work in parallel to the other thread-based processes mentioned in previous files. It also represents a state machine for Xbox 360 controller input events and a differentiation between manual and autonomous state.
   
   int getControlState();
   
   Simple get function similar to the ones used in object-oriented programming languages. Returns the value of a global static variable which is shared between two threads.
   
   void *read_Xbox360_Event();
   
   This function calls the PCA9685_Init( ) function twice to initialize the two PCA9685 modules. Afterwards, it reads all events from the joystick event buffer and counts the amount of events. These events will be processed after checking the state of the process, which is either "manual" or "autonomous". In manual mode, there is an differentiation between the input events of steering, throttle forwards, and throttle backwards. By entering the autonomous mode for the first time, a thread is started which opens a UDP socket to receive input instructions for steering from an A.I. algorithm. The throttle values are still coming from the Xbox 360 controller. On exiting the autonomous state, the UDP_receive thread will be cancelled to lower CPU consumption.

Functional project overview

In my opinion, it is also a good thing to have an overview of dependencies and the workflow of a project. Therefore I created a process overview diagram and will add some explanations to the following state charts.

The main thread

The main thread initializes the joystick device and the thread resources for synchronization. It then starts two threads, which will perform the predefined tasks in parallel. The main thread also waits for the two threads to exit or for an exit signal from a terminal. In this case, On Program Exit will be executed.

The Video_Capture thread

This thread is responsible to capture images for a video stream. It also reads a steering angle from a global variable and writes its value to the filename of the current video stream image. This reading operation has to be read-write protected via a mutex from the Xbox_Control thread trying to access that global variable simultaneously. After reading, it saves the image to the predefined path and shows the current image on a screen.

With waitkey(15) we get a resolution of almost 60 frames per second, because it captures and holds an image for 15ms, which results in somewhat more than 60fps. However, in reality the speed is less than 60 frames per second, because saving and displaying the image takes processing time of about 100ms. So, an idea is to separate the capturing of images from saving and displaying them and implement the latter as an additional thread.

The Xbox_Control thread

This thread reads joystick events from the Xbox 360 controller and counts the number of events. When done, it processes these events and then checks for the state of the process, which is either "manual" or "autonomous". If there was a button A press, it toggles the controlState variable.

This write operation needs to be read-write protected via mutex from the UDP_receive thread accessing the variable. A condition state checks the current process state and then continues processing the state machine of the current process state. At the end, it sleeps for10ms to reduce CPU consumption.

The state machine's manual mode

In manual mode, the Xbox 360 controller input is used to control the RC Car. It checks for input events concerning the steering or the throttle of the RC Car. It also differentiates between axis or non-axis event type, because the same event numbers are assigned to button events as well as to axis events.

In our case, there are only axis events used to control the RC Car. Therefore, the event number differentiates between events and triggers execution of the associated instructions. In the case of steering, a write operation on a global variable occurs and needs to be read-write protected via mutex from the Video_Capture thread accessing it.

The state machine's autonomous mode

The autonomous mode reacts to changes of the process state through a toggle flag. This flag is used to instantiate the UDP_receive thread, which is responsible for autonomous steering through the A.I. process. This initialization is done only once at the beginning. The thread initializes the UDP socket and then starts processing its predefined task. That task will be discussed in the following section.

Apart from the new thread started at the beginning, the autonomous state machine is quite similar to the manual one. The throttle control is even identical to the manual mode. Only the steering will be processed by the UDP_receive thread.

The UDP_receive thread

This thread needs to check the process state via the getControlState( ) function. This read operation has to be read-write protected via mutex from the Xbox_Control thread to access the global variable.

If the process is in autonomous mode, the thread waits for a UDP datagram on the previously-opened socket. After that, it checks whether a correct UDP datagram has been received and if so, it converts the received angle to an integer value and writes it to a local variable. The read operation from the recvbuff UDP buffer has to be read-write protected via mutex from the A.I. process accessing the UDP buffer.

However, at the moment it is only implemented as a dummy variable and has no synchronization effect, because the A.I. process is not finished yet. The same applies for the steerVehicleAutonomous(fd) function and the synchronization with the A.I. process. The global variable shared with the Video_Capture thread has to be read-write protected via mutex. Eventually, the thread sleeps for 1ms to reduce CPU consumption.

On Program Exit

If there is an exit condition, i.e,

• an exit command has been issued from a terminal,
• a program stopp has been requested by the Eclipse GUI, or
• an error occurred that called the exit( ) function,

Final assembly of the RC Car

Finally, after explaining the software, we can have a look at the current state of the assembled car.

We hope this blog post gives a good overview on the required steps to set up a car like ours. If you discovered any problems or inconsistencies, please do not hesitate to leave a comment!

Troubleshooting

Previously, I promised to explain why we are using two PCA9685 modules. When we were using only one module and only gave steering inputs from the Xbox 360 controller, everything seemed to work fine. But after we gave the first throttle input to the ESC, a subsequent steering input lead to an unpredictable behavior of the throttle.

After some effort spent on analyzing this problem, we were still clueless what the cause was. At the same time, we were preparing for an event and had a short time frame, so we decided to solve the problem by separating servo and ESC control.

We also decided to deal with this issue at a later point, because the priority to solve it was low. Anyway, if someone has an explanation based on our source code, please feel free to share with us!

Further plans

We already have a number of ideas for further improvements and extensions to the car:

• Adding a LIDAR sensor
• Adding an IMU
• Replacing the Raspberry by an Nvidia Xavier
• Implementing some of the functionality according to AUTOSAR (based on the Minnowboard)