Creating an Augmented Reality Sandbox for the Science Classroom (Part 1)

Creating an augmented reality sandbox for hands-on learning in science classrooms is (maybe?) a bit harder than it might seem. In early 2020, we embarked on a journey to build our own from scratch. Now in 2021, find out what hurdles we’ve faced, where we stand, and what we need to do to finish the project.

Augmented Reality Sandbox

The Augmented Reality Sandbox

Developed by University of California Davis as part of an NSF-funded project, the augmented reality sandbox was created for informal science education of freshwater lake and watershed science.

The project combines 3D visualization applications with a hands-on sandbox exhibit to teach earth science concepts. The augmented reality (AR) sandbox allows users to create topography models by shaping real sand, which is then augmented in real time by an elevation color map, topographic contour lines, and simulated water. The system teaches geographic, geologic, and hydrologic concepts such as how to read a topography map, the meaning of contour lines, watersheds, catchment areas, levees, etc.

The sandbox is capable of rendering a topographical overlay onto sand by levying a Microsoft Kinect camera with various programs on a Linux computer. The Kinect camera measures the depth of field (DoF) of the sand in the sandbox, which can be manipulated by hand to build mountains, valleys, plateaus, rivers, etc. This technology works in a similar way to LiDAR, which measures the distances to various points in an image.

These measurements are then received by the software, converted to topography, and displayed on the sand overlay projection in near real-time, making it a highly interactive addition to any science classroom. Current versions of the 3D rendering software also include water animations for a more interesting and realistic overlay.

Augmented Reality Sandbox at Holston High School

Steve Ahn, science teacher at the school, originally brought this idea to our attention. He had seen one of the sandboxes used a few times in other areas and thought it would be a great idea to create our own!

While the concept isn’t exactly a new one, as the project is several years old, according to the official map sandbox locations on the UC Davis website, a majority of the boxes are at colleges or universities. Being the first public school in Southwest Virginia, Northeast Tennessee, or Northwest North Carolina to build and retain our own augmented reality sandbox is a very exciting prospect!

When the project is completely finished, the sandbox will be housed in a new emerging technology area at the school; more information on this later this summer. This center will provide a place for students at all three Damascus-area schools to be able to come experience and learn from many new technologies they might not have the opportunity to use otherwise.

Regional AR Sandboxes: East Tennessee State University and Hands on Museum in blue; Holston High School in red

Pre-Made Augmented Reality Sandboxes

Before we get into the information about our specific sandbox build project, materials needed, and the hurdles we’ve faced, I want to take a moment and discuss the pre-assembled augmented reality sandboxes you will see inevitably see online.

First and foremost, you will notice these sandboxes are, to say the least, expensive. In fact, at the time of this writing, the three that show up in search results range from $6,000 to nearly $10,000. This price doesn’t include the kinetic sand, which depending on your supplier of choice, can be a bit of a cost as well.

Shipping costs, if any, are not immediately available either.

Secondly, while these do come with most everything you need (minus the sand), you will still have assembly to do as well as the setup of the computer software. One description even states the sandbox ships with a Linux laptop (brand, distro, and specs unknown) that will “come with instructions to load the visualization software.”

Having said that, yes, these options are much less work in the long run than sourcing and building the project with your students, but they are more than likely prohibitively expensive for your school and/or school system. And, they might still require new hardware after a short time to power the visualization software as its new iterations of the software continues to get better and more powerful.

Building Your Own Augmented Reality Sandbox

Building your own augmented reality sandbox is a much cheaper option than buying one pre-made. It is, of course, much more time and labor intensive, but serves as the perfect opportunity to teach students about the process and engage in project-based learning.

You can find a full listing of recommended parts on the augmented reality sandbox website, but I am listing the specific parts I used below. As far as the physical sandbox is concerned, consulting with your shop teacher or someone who is good at creating and building should yield a quick price estimate and construction guidance. And, they might even build it for you!

Depending on current prices on components, this project is very doable at around $2000, which is a lot cheaper than the options you can buy online. This includes the Linux computer build, extra needs, and materials to build the physical sandbox itself.

List of Components We Used

*Estimated Price = $1,200; at the time of this writing, GPU’s are very hard to get and prices are fluctuating constantly due to the meteoric rise of crypto currency mining.

List of Miscellaneous Needs

  • Etcher (also known as Balena Etcher) – this is used to write the Linux Mint ISO to a flash drive so you can boot from the medium and install Linux on your AR computer build
  • Microsoft Kinect Adapter/Power Supply – to use a Kinect with a computer, you will have to purchase a power supply/USB adapter to be able to use it. We didn’t realize this up front, so easier to know now so you can purchase this and prepare!
  • Linux Mint with MATE ISO (64 bit) – Linux Mint is specified in the original build needs and the MATE desktop is arguably the best desktop environment on Linux since it’s most similar to what you’re used to on Windows or Mac. You’ll select this ISO when using Etcher to write it to your bootable USB drive.
  • USB Flash Drive (32GB) – used to make the bootable drive for initial installation of Mint on the finished desktop

*Estimated Price = $20

Don’t Use an AMD Processor!

As you’ll see below, our largest issue with this project was getting the motherboard to work. While I don’t directly attribute this to our initial use of an AMD CPU and motherboard, that could have been part of the cause. In fact, I later found in the official documentation NOT to use AMD because it’s unable to run several NVIDIA drivers you’ll need for the newer features of the software, so in the long run it wouldn’t have been a good idea even if it had worked. Unfortunately a bit of a financial loss, but at least it wasn’t an overly expensive one.

Start with the motherboard and Intel processor listed above and you shouldn’t have any problems.

Early Build Hurdles

Sourcing the components for the Linux computer was not a hard task. In the early days of planning, we actually turned to several students who have built their own computers to research and source the components as part of a project-based learning (PBL) approach to the project. We went through several sets of parts on PartPicker before landing on what was as close to the recommended parts as possible.

It should be noted, we did opt for an AMD Ryzen 2600 CPU at first as it should have given plenty of performance capability. However, as mentioned above, do NOT use AMD architecture with this build!

Within a few weeks or less, all of the different components had been delivered and we were ready to begin assembly. We once again turned to this group of students to physically assemble the computer. However, this construction and initial boot turned into the first hurdle we’d run into with this project.

The First Bump in the Road

Simply put, the computer would not POST and display the needed BIOS screen to allow us to boot from the USB flash drive containing the Linux distro. We tried multiple things, including different VGA, HDMI, and display port cables with the various outputs on both the motherboard and the graphics card; no luck. I even ordered motherboard speakers to see if we could get some audible information as to what the problem might have been.

We spent a short amount of time working on this issue and even tried removing the graphics card and RAM to see if there was an issue with either one. Turns out, there wasn’t. It was around this time work on the project ceased completely due to the COVID-19 shutdowns across the state and country.

Picking the Project Back Up… And Putting It Back Down

Around the first of May, I picked the project back up and kept working on troubleshooting the issue with the CPU/motherboard. Even after several days of back and forth with Gigabyte support and reading countless forums online, there was still no signal transmitted from the motherboard to a display. There also, even more oddly, was no sounds coming from the motherboard speaker either.

It was at this time we decided to bite the bullet and order a second motherboard, assuming ours was faulty. Several online forums indicated the particular model of motherboard we had purchased were finicky at best.

A few days later, the second board arrives and we were pleasantly surprised to find out… it didn’t work either. Everything checks out, all components power, LEDs turn on, CPU is compatible, but still no progress. Again, the project gets shelved in favor of other more pressing matters to prepare for the year ahead of us.

Finally Progress in 2021!

This spring, it was time to pick the project back up again. We decided to buy yet another motherboard, this time coupling it with an Intel CPU. As referenced above, use Intel for this computer build!

After getting the new components in and assembling them, the computer was up and running in just a few minutes. Booting from the USB drive and installing Linux was a breeze and a year later, we finally had our Linux computer for the augmented sandbox up and running!

RawKinectViewer showing the two camera outputs from the Kinect camera

Tips for Your AR Linux Computer Build

Tip #1: Update Linux Mint After Boot

After installing Linux Mint, removing the bootable USB drive, and booting the computer for the first time, it’s a good idea to make sure Linux is up-to-date. While the update service should alert you there are updates to be done, you can open Terminal and enter the following code and hit enter.

sudo apt upgrade

Tip #2: Run Terminal Commands One Line at a Time

In the documentation for installing the software needed for the augmented reality software, you’ll see most sets of commands grouped together, like this:

cd ~
wget http://web.cs.ucdavis.edu/~okreylos/ResDev/Vrui/Build-Ubuntu.sh
bash Build-Ubuntu.sh

However, to be safe and make sure Terminal takes the commands correctly, I recommend you complete them one at a time. That turns the example above into this:

cd ~
wget http://web.cs.ucdavis.edu/~okreylos/ResDev/Vrui/Build-Ubuntu.sh
bash Build-Ubuntu.sh

Where Do We Go From Here?

Now that everything checks out with the Linux computer (and it runs!) and the Microsoft Kinect camera is showing readings correctly in RawKinectViewer, we have to do the following to finish the sandbox completely:

  • build and install the mounting bracket for both the projector and the Kinect camera
  • go through multiple calibration steps as outlined in the documentation to align the camera and projector to the physical sandbox corners/edges
  • add the kinetic sand and test the projector output
  • document, share our results, and teach others how to use the system with their classes

Linux Computer Idle Time

Since the Linux computer has plenty of power and will not be used constantly for sandbox projections, we have several other projects planned for it in its idle time. This may or may not include photogrammetry image processing, BOINC work unit processing, and since it has a capable GPU on board, Folding@Home work unit processing!

Currently, Folding@Home is focusing on COVID Moonshot processing and analysis.

Stay tuned for Part 2 of the augmented reality sandbox build!

One thought on “Creating an Augmented Reality Sandbox for the Science Classroom (Part 1)

  1. This is really well written and so much more straightforward than so many other sites that provide advice on the AR Sandbox. Well done!

Leave a Reply to Steve Ahn Cancel reply

Your email address will not be published. Required fields are marked *