Student project: Implementing a Digital Twin of an orchard using color point cloud over time

Periodic measurement and prediction of apple tree’s architectural attributes from LiDAR point clouds to provide important insights on different levels of the tree’s structure (Tree level, branch, sub-branch).

What you will do

In our Digital Orchard program, we are interested in implementing a digital twin of an orchard. The Orchard will be recorded by combining LiDAR and common (RGB) cameras data into a  “colored point cloud”. We are working on multiple recording devices which will be improved during the project. Current, we have performed multiple measurement campaigns allowing us to follow fruit tree development in the orchard over time. Our aim is to measure the tree growth/development over time by processing the recorded point clouds and images.

You will also evaluate and define what measurable attributes to study in order to monitor the orchard plant development, where our final goal is to monitor multiple aspect, i.e., leaf growth, blossoming, fruit growth, and harvest. Given the internship timeline, we will probably pick a single aspect to study for the student. Measurable attributes should be understood by farmers and readable by automatic systems (e.g., canopy height, canopy area, canopy density, tree spacing).

The final goal is that the measurable attributes per tree/branch are stored in a digital twin representation, where you can experiment with already existing digital twin frameworks (e.g., Microsoft Azure’s Digital Twin services). On the basis of a digital twin, further decision making can be enabled, increasing the automation in the digital orchard.

During the internship you will model and evaluate predictions of these attributes based on the annotated (colored) point clouds. In this case, it is important to obtain good prediction, but it is also important to be aware of uncertainty margins for these predictions because of external factors (e.g., pests, drought, late frosts). We would like to be able to make predictions at different levels: on a branch, subbranch and tree level. These predictions should deliver valuable insights for farmers.

Our vision for this project is to use state of the art Deep Learning in point clouds.  This requires techniques like PointNet++, KPConv, PointGAN, Point Cloud GAN, or other architectures for subsequentially segmentation of the different classes (wood, leaves, flowers, fruits) and over-time prediction of growth. Knowing that this is a challenging assignment, we will scope the internship activities to make them feasible within the internship timeframe.

Main tasks:

• Literature review of related solution and technology.
• Prepare a plan containing a timeline with deliverable.
• Actively participate and plan measurement campaigns.
• Process and annotate point clouds making the suitable for Machine Learning.
• Create a machine learning algorithm to predict/measure attributes of key importance to farmer.
• Evaluate the machine learning solution(s). 

What we do for you

• We have a challenging problem where you have a lot of freedom to come up with solutions.
• We have a diverse team of experts both from the biological and the technical sides to supervise and support you.
• You will join the Data Science team of OnePlanet, which employs state of the art knowledge on machine learning for precision agriculture and the frameworks necessary to perform these big data tasks at huge scale.
• You will be able to exchange views and knowledge with the OnePlanet and Imec community of experts and scientists, widening your professional network.
• We can help you to improve your coding skills up to industry standards.
• You have access to our cloud solutions to solve this problem allowing you to process large amount of data within reasonable time.

Who you are

• Familiarity with LiDAR scanning.
• Knowledge of Python.
• Knowledge of Machine Learning.
• (Advanced) Knowledge of Deep Learning for point clouds.
• (Advanced) Knowledge of Git.