Friday, August 26, 2022
Edge Computing
Edge Computing
Autonomous
Autonomous
AI for Wildlife
AI for Wildlife

Autonomous Flight and the Landing of a Fixed-Wing UAV

Developing a model for autonomous landing using a ‘traditional’ technique using GPS beacons a more experimental method of Reinforcement Learning.

This is an account of the 10-week teamwork of the AI for Wildlife Challenge 3 engineers working on a poacher-detecting model running to edge hardware of an autonomous drone. The group was split into 4 teams: Model Optimization, CI/CD, Hardware and Autonomous Flight. Learn how the Autonomous Flight and Hardware teams addressed the problem.

The Threats of Autonomous Landing in South African Bush

Although the plane is guided by a mission planner in-flight, the landing is not executed on auto-pilot. Extreme caution has to be taken when landing a fixed-wing plane with a span of four meters in a densely packed landscape of the south aAfrican bush. Roads are only slightly wider than the plane - there are high risks involved when landing, including hitting overhanging tree branches, uneven terrain and passing animals. Because of this, basic auto-landing features were originally replaced by manual landing with expert drone pilots.

But manual landing requires many resources. SPOTS wanted to implement a smart auto-landing feature as soon as possible. Our team set out to develop a prototype for landing the plane autonomously.

We first set out to list all the constraints the plane has to adhere to. Things like weather conditions, runway conditions and plane measurements have to be encoded accurately when giving the control of the plane over to our software. 

We approached the landing problem from two perspectives: 

  • developing a ‘traditional’ technique using GPS beacons 
  • try a more experimental technique that uses Reinforcement Learning

GPS Beacons Pointing the Drone to Safe Landing

The first technique involves setting up two GPS beacons at the start and at the end of the runway, which serve as pointers for the plane. When the drone approaches the first runway pointer, a script calculates the desired landing approach direction, and uploads a flight plan to a PX4 Flight Control Unit (FCU). The FCU monitors the sensors onboard the plane and executes a landing manoeuvre. This ensures that the landing is soft enough and does not divert too much from the middle of the runway. 

Runway model including GPS locations

Using a Gazebo simulation environment, we tested the auto-landing script with success. The only setback we faced was a discrepancy in the simulation, causing the plane to land below the surface of the map. After fixing this, the method is ready to be tried on a test-version of the real plane in South Africa.

Architecture of the PX4 modes
“The diversity of the tasks involved challenged me to learn a lot of new things in the overlap of hardware and software. The interactive team environment and interaction with the people from SPOTS was a great experience for someone learning to apply AI in real-life scenarios. Moreover, working on a case as beautiful as wildlife conservation has been inspiring, it showed how applying engineering knowledge can have a positive impact in the real world.” - Emile Dhifallah, Autonomous Flight Team

Reinforcement Learning for Autonomous Landing

We started to build a second simulation environment in parallel, where we could train a model of the plane to land with the help of a Reinforcement Learning agent. In a highly realistic environment built in Unreal Engine 4, we could model the descent of the plane using an Airsim simulation. The goal here was to connect the plane’s controls to OpenAI’s Gym, which is an extremely useful toolkit for running RL experiments with drones.

Programmatically controlled quadrotor model in the Airsim environment

Training a RL agent to land the plane involves setting definitions of what is considered ‘good’. Specifically, we had to define:

  1. Reward function; probably the trickiest one, as we needed to encode what it means to land safely and at the predefined location in this function. The former means incorporating technical constraints and collision dangers, while the latter means calibrating the runway location as to measure when it has reached its destination, through GPS and lidar usage. 
  2. Observations - snapshots of the continuously changing environment
  3. Possible actions - these are defined as the input that is accepted by the PX4 controller, which translates to roll, pitch, yaw and throttle
  4. Conditions for termination; i.e. when should the agent start over the descent?

After encoding all of this information, the agent could start learning the landing process. We made a first attempt at doing so using Deep Q Learning. The progress was unfortunately halted because of the lack of compute resources. Other obstacles we faced in the RL pipeline included a lack of definition in the Unreal Engine environment, which is essential to transferring the capabilities of the reinforcement learning agent to the real-life plane.

Both of our development trajectories need more refinement before real-life testing, but they’ve shown to be a promising replacement for manual landing of the SPOTS plane. We’re looking forward to seeing the techniques above implemented on the drone in the future.

Autonomous Flight Team

Thanasis Trantas, Emile Dhifallah, Nima Negarandeh, Gerson Foks, Ryan Wolf 

Running ML Models on Edge Hardware, Mid-flight  

The goal of the Hardware Team was to connect different hardware components on the drone, which included:

  • Pixhawk: the computer responsible for flight control 
  • Jetson Nano: a companion computer equipped with a GPU which handles the poacher detection model
  • A thermal camera
  • Herelink antenna sending information to the ground station
  • A laptop on the ground receiving information and interacting with the drone
The diagram showing the connected components

We had to make sure that the information flows smoothly in the whole system, with as little delay as possible. Within seconds, the drone could have already moved tens of meters, so the detection results had to be processed and sent to the ground really quickly. 

Our goals were: 

  • To figure out how to run the YOLO detection model on the Jetson Nano
  • To reduce the computation load teaching the model to ignore every couple of frames from the video feed 
  • To find a way to send the model results to the ground and to clearly display them

“Being a part of the AI for Wildlife Challenge was a great learning opportunity. My background is in machine learning, so I was able to learn about different parts of the pipeline required to create a successful real-life solution. The biggest challenge was to work on the hardware remotely (I was in Amsterdam, 9000 km from the actual drone), but I still managed to contribute to the project. It was great to meet people from all over the world and work together towards a common goal for a good cause.” - Maks Kulicki, Hardware Team

We were able to run the model on the Jetson Nano, but we didn’t find a way to skip the frames. We managed to set up the system to send the information to the ground. It required plugging an HDMI cable between the Jetson Nano and the Herelink transmitter. We tried to set up the ground display to show the original video alongside the model output on the same screen, but it turned out to be more difficult than expected. 

The model output sent to the ground station successfully

Hardware Team of AI for Wildlife 3

Kamalen Reddy, Estine Clasen, Maks Kulicki, Thembinkosi Malefo, Michael Bernhardt

Model Optimization: Sahil Chachra, Manu Chauhan, Jaka Cikač, Sabelo Makhanya, Sinan Robillard

CI/ CD: Aisha Kala, Mayur Ranchod, Sabelo Makhanya, Rowanne Trapmann, Ethan Kraus, Barbra Apilli, Samantha Biegel, Adrian Azoitei

Hardware: Kamalen Reddy, Estine Clasen, Maks Kulicki, Thembinkosi Malefo, Michael Bernhardt

Autonomous Flight: Thanasis Trantas, Emile Dhifallah, Nima Negarandeh, Gerson Foks, Ryan Wolf

Find out how the Model Optimization and CI/CD teams tackled their part of the Challenge

Read the full story in the AI for Wildlife 3 case study

Read full version
Subscribe to our newsletter

Be the first to know when a new AI for Good challenge is launched. Keep up do date with the latest AI for Good news.

Thank you!

We’ve just sent you a confirmation email.

We know, this can be annoying, but we want to make sure we don’t spam anyone. Please, check out your inbox and confirm the link in the email.

Once confirmed, you’ll be ready to go!

Oops! Something went wrong while submitting the form.

Previous publication

You are reading the most recent publication!

Return to publications page

Next publication

You have reached the end of our publications, you must love our content!

Return to publications page
GIS
GIS
Remote Sensing
Remote Sensing
AI for Earth
AI for Earth
Explainable AI
Explainable AI
Time Series Forecasting
Time Series Forecasting
AI for Health
AI for Health
Autonomous
Autonomous
TinyML
TinyML
MLOps
MLOps
Edge Computing
Edge Computing
AI for Wildlife
AI for Wildlife