At itemis, we are involved in automotive software projects in terms of modeling (domain specific languages for architecture and behavior), tooling (architecture, feature models, implementation, machine learning) and concepts/standards (AUTOSAR, Genivi, openADX). At our office in Stuttgart, we are setting up a tangible demonstrator – a robocar platform as a flexible base with an initial showcase of machine learning.
The first version of the car was based on the Sunfounder platform. Our good experiences quickly lead to further ideas and plans, and we ran into the limits of the Sunfounder platform. So we decided to set up a new demonstrator platform based on a Traxxas Slash 4x4 Platinum.
The current goal of the project is to get control of the RC Car with a remote control unit (XBOX360 controller) and to capture video frames from an racing track via the SainSmart 5MP camera module with 175 degree panoramic wide angle. The captured images will run through image processing and will serve as an input to an A.I. algorithm of behavioral cloning. The A.I. also receives the steering angle of the RC Car, and it will be trained to understand the behaviour of steering on a predefined track. So the RC Car should be able to steer autonomously through the whole track.
The project is based on the Traxxas Slash 4x4 Platinum Edition (Model 6804R) RC racing truck with the original Traxxas NiMH Power Cells with a capacity of 3000 mAh. Its drive system constists of the Traxxas 3355R Velineon VXL-3s waterproof Electronic Speed Control (ESC), which is used to control the connected Traxxas Velineon brushless DC motor.
Not included was the radio system (transmitter and receiver). We bought it for the initialization sequence of the ESC and to analyze the necessary PWM duty cycles. It also came with a digital and waterproof Traxxas 2075 servo, which has a maximum rotation angle of 180° and is intended to use for steering the RC Car. The following picture shows the out-of-the-box state of the car with the above-mentioned important basic components.
We also needed some additional components to make the RC Car running for our needs:
We assembled and connected all of the components as shown in the following picture:
Caution:
Plug in the Anker Power Core and the Traxxas Power Cell only after assembling the components! Assembling while there is a power source connected can cause damage to the components or maybe even harm a person.
At this point, we could power up the whole system. First power up the Raspberry Pi 3 with the Anker Power Core and let the system boot. Afterwards, you can plug in the Traxxas Power Cell and press the EZ Button on the ESC until the LED is blinking green. If you have not already done the initialization process, you first need to do the following step.
There are two ways to initialize the ESC to get it controlled with a PWM signal. The initialization sequence needs to register the throttle signal range for driving forwards and backwards. For the initialization sequence it is easier to use the original radio system (receiver and transmitter), which is compatible to the ESC, and follow the instructions from the delivered quick-start guide.
This process is quite difficult to manage without the appropriate equipment. But fortunately, you only have to do this process once, unless you want to change the operation mode of the car.
We are using the training mode with the 50:50 range of the throttle to lower the velocity.
After the initialization process is done, we need to figure out the duty cycle in milliseconds of the PWM signals, which are used to steer the servo and to control the throttle.
You can use an oscilloscope to observe the signals, or you can use an Arduino Uno and the following sketch for the Arduino IDE. We chose the latter option, because we had an Arduino Uno in stock.
byte PIN = 3;
int pwm_value;
void setup() {
pinMode(PIN, INPUT);
Serial.begin(9600);
}
void loop() {
pwm_value = pulseIn(PIN, HIGH);
Serial.println(pwm_value);
}
In this case, you also need the radio system (receiver and transmitter) to observe the signals that have previously been sent from the transmitter to the receiver.
Here is an explanation of how to connect the Arduino Uno to the transmitter and how to read the signals correctly.
First, we had the same results for the duty cycles for the transmitter as shown in that tutorial. But later on, in our software, we observed a small delay caused by functions in the PCA9685 library. A small inaccuracy can have a remarkable impact on the throttle control. Therefore, it is also recommended to observe the generated PWM duty cycles from the used software. You can use the following modified Arduino sketch to figure out the duty cycles of the PCA9685 module for steer and throttle.
You can also compare the transmitter signal to the generated PWM signal from the PCA9685 module. Just additionally connect the PCA9685 module's PWM pin to the PWM IN Pin 5 of the Arduino Uno, and make a ground connection between the PCA9685 module and the Arduino Uno.
byte PIN = 3;
byte PIN_Rpi = 5;
int pwm_value, pwm_raspi_value;
void setup() {
pinMode(PIN, INPUT);
pinMode(PIN_Rpi, INPUT);
Serial.begin(9600);
}
void loop() {
pwm_value = pulseIn(PIN, HIGH);
Serial.println((String)"PWM signal from Traxxas Transmitter\t" + pwm_value);
pwm_raspi_value = pulseIn(PIN_Rpi, HIGH);
Serial.println((String)"PWM signal from pca9685 module\t" + pwm_raspi_value);
}
We are using the Raspbian Stretch image version which is shown in the following terminal output.
It also has some additional software installed:
After these installations, which can take some time, the image is ready to use. Otherwise, you can use the attached image and flash it on your SD card. Make sure you have at least a 16GB SD card.
The project can be imported to the Eclipse IDE by the following steps:
Inside the project folder there are several C header and source files. There is also a CMakeLists.txt file which comprises the instructions to build the project, to generate object files, to link the libraries, and to produce the executable. The .project file contains project-specific instructions, which are read when importing the source code to Eclipse IDE. The images folder will contain the captured images for image processing and as an input path for the A.I. algorithm.
Explanation of the general function of the sourcefiles – for more specific explanations look up the source code and the comments:
int calculateSteeringAngle(float mapped_axis_value);float convertReceivedSteeringAngle(int steeringAngle);int initVideo();void *captureVideo();void Joystick_Init();int read_joystick_event(struct js_event *jse);void close_joystick();float map_axis_servo(int axis_value);void steerVehicle(int fd, short value);void steerVehicleAutonomous(int fd);float map_axis_throttle_forwards(int axis_value);void driveVehicle_forwards(int fd, short value);float map_axis_throttle_backwards(int axis_value);void driveVehicle_backwards(int fd, short value);void init_socket();void *receiveAngle();int calcTicks(float impulseMs, int hertz, int on_off);float map(float input, float min, float max);int PCA9685_Init(const int i2cAdress, const int pinBase, const int freq);int getControlState();void *read_Xbox360_Event();In my opinion, it is also a good thing to have an overview of dependencies and the workflow of a project. Therefore I created a process overview diagram and will add some explanations to the following state charts.
The main thread initializes the joystick device and the thread resources for synchronization. It then starts two threads, which will perform the predefined tasks in parallel. The main thread also waits for the two threads to exit or for an exit signal from a terminal. In this case, On Program Exit will be executed.
This thread is responsible to capture images for a video stream. It also reads a steering angle from a global variable and writes its value to the filename of the current video stream image. This reading operation has to be read-write protected via a mutex from the Xbox_Control thread trying to access that global variable simultaneously. After reading, it saves the image to the predefined path and shows the current image on a screen.
With waitkey(15) we get a resolution of almost 60 frames per second, because it captures and holds an image for 15ms, which results in somewhat more than 60fps. However, in reality the speed is less than 60 frames per second, because saving and displaying the image takes processing time of about 100ms. So, an idea is to separate the capturing of images from saving and displaying them and implement the latter as an additional thread.
This thread reads joystick events from the Xbox 360 controller and counts the number of events. When done, it processes these events and then checks for the state of the process, which is either "manual" or "autonomous". If there was a button A press, it toggles the controlState variable.
This write operation needs to be read-write protected via mutex from the UDP_receive thread accessing the variable. A condition state checks the current process state and then continues processing the state machine of the current process state. At the end, it sleeps for10ms to reduce CPU consumption.
In manual mode, the Xbox 360 controller input is used to control the RC Car. It checks for input events concerning the steering or the throttle of the RC Car. It also differentiates between axis or non-axis event type, because the same event numbers are assigned to button events as well as to axis events.
In our case, there are only axis events used to control the RC Car. Therefore, the event number differentiates between events and triggers execution of the associated instructions. In the case of steering, a write operation on a global variable occurs and needs to be read-write protected via mutex from the Video_Capture thread accessing it.
The autonomous mode reacts to changes of the process state through a toggle flag. This flag is used to instantiate the UDP_receive thread, which is responsible for autonomous steering through the A.I. process. This initialization is done only once at the beginning. The thread initializes the UDP socket and then starts processing its predefined task. That task will be discussed in the following section.
Apart from the new thread started at the beginning, the autonomous state machine is quite similar to the manual one. The throttle control is even identical to the manual mode. Only the steering will be processed by the UDP_receive thread.
This thread needs to check the process state via the getControlState( ) function. This read operation has to be read-write protected via mutex from the Xbox_Control thread to access the global variable.
If the process is in autonomous mode, the thread waits for a UDP datagram on the previously-opened socket. After that, it checks whether a correct UDP datagram has been received and if so, it converts the received angle to an integer value and writes it to a local variable. The read operation from the recvbuff UDP buffer has to be read-write protected via mutex from the A.I. process accessing the UDP buffer.
However, at the moment it is only implemented as a dummy variable and has no synchronization effect, because the A.I. process is not finished yet. The same applies for the steerVehicleAutonomous(fd) function and the synchronization with the A.I. process. The global variable shared with the Video_Capture thread has to be read-write protected via mutex. Eventually, the thread sleeps for 1ms to reduce CPU consumption.
If there is an exit condition, i.e,
Finally, after explaining the software, we can have a look at the current state of the assembled car.
We hope this blog post gives a good overview on the required steps to set up a car like ours. If you discovered any problems or inconsistencies, please do not hesitate to leave a comment!
Previously, I promised to explain why we are using two PCA9685 modules. When we were using only one module and only gave steering inputs from the Xbox 360 controller, everything seemed to work fine. But after we gave the first throttle input to the ESC, a subsequent steering input lead to an unpredictable behavior of the throttle.
After some effort spent on analyzing this problem, we were still clueless what the cause was. At the same time, we were preparing for an event and had a short time frame, so we decided to solve the problem by separating servo and ESC control.
We also decided to deal with this issue at a later point, because the priority to solve it was low. Anyway, if someone has an explanation based on our source code, please feel free to share with us!
We already have a number of ideas for further improvements and extensions to the car: