Grace Under Pressure - THESIS

Welcome to the TechNICAL BLOG

12/31/2020

Most of my day-to-day activities when I'm not on an exciting expedition are related to developing and testing new ideas and research as well as software and mechanical devices. I think of the process as:

Like so many others, when I know a task should be fairly standard (e.g., yesterday I wanted to convert ten thousand .pgm files into .tiff files.), but I don't know how to do it, I Google it and hopefully an Internet user or blogger-from-heaven has written about it (THANK YOU "Matt"; I installed ImageMagick and used it's 'convert' function).

When Google doesn't have the answer, however, I take extra time to figure out the solution, usually by asking people around the lab or using the trial-and-error-plus-reading-bits-of-the-manual method.

I'm going to try to give back more by recording oddball technical things that might save others time and frustration. It's like take-a-penny leave-a-penny with the Internet. I realize this changes the audience of my blog; so I'm posting these under this new page "Thesis."

Rhino 3D tutorials

2/23/2018

This is a guest post by the incredible Kathryn Ellen Whittey from Cardiff University! The two of us ran a workshop together at the 2017 European Coral Reefs Symposium about how to create and analyse 3D models of coral reefs using PhotoScan and Rhinoceros 3D ("Rhino"). The methods are outlined in my PloS One publication "Cost and time-effective method for multi-scale measures of rugosity, fractal dimension, and vector dispersion from coral reef 3D models," and here they are explained in a slightly friendlier way. If you've issues please comment below and we'll work something out. I've also published 3D modeling teaching notes (link here) and a tutorial on how to rotate and scale models in Rhino (link here).

Onto Kath... !

In this series of video tutorials I have used one of my very own brain coral bommies from my study site in Tobago. The model has been rendered using structure-from-motion techniques with an Olympus tough TG-4 camera and rendered in Agisoft Photoscan.

In these demonstrations I show how to rotate, delete, fill holes in the model, scale the model, take measurements and use the python script coded by Grace Young to calculate rugosity. For each video the commands used are highlighted in bold, these can be typed straight into the Rhino interface.

Video 1: Rotate & Tidy

Topics covered:

Rotating
Filling holes
Deleting
Rhino Commands: Rotate, FillMeshHoles, Explode, SelAll, Join

In this video the coral model has come straight out of Agisoft Photoscan, and as you can see it isn’t perfect, there are abnormalities such as floating pieces of sand and holes in the model mesh which are not representative of the true coral bommie. Before I begin clearing up these abnormalities I first rotate the model in rhino into the correct orientation (this makes it easier to view and also is important for taking measurements such as rugosity later on). To rotate the model, begin by selecting it and typing into the command box the command “rotate”. In the video I rotate my bommie twice, aligning the sandy seabed with the x axis. Alignment is sometimes more fiddly than this example and you may need to be correct orientation several times in the different viewport windows.

Once you’re happy with the alignment you can move on to filling the holes in the model. To do this, simply type the command “FillMeshHoles” and rhino automatically fills any holes in the model with ‘artificial’ mesh. Rhino usually figures out where the holes need to be filled pretty accurately but sometimes is creates mesh where you don’t really need it. If you are not familiar with your coral, using still photographs as a reference is useful. To delete any non-representative mesh pieces, we first need to break the model into the constituent meshes by using the command “Explode”. You can now select the individual undesired mesh pieces and delete them, including any incorrect mesh pieces rhino has added. Once you’re happy that the model is tidied up you can then select all (Command: SelAll) and re-join the meshes (Command: Join).

Note that this video has NO AUDIO.

Video 2: Scaling & Measuring Distances

Topics covered:

Scaling
Measuring
Rhino Commands: Point, OnMesh, DimAligned, SelAll, Group, Scale, Area

To scale your model you will need to use a reference object of a known size (this needs to be in the original video). For my models I used a weighted Lego® cube measuring 64 mm3. Using one edge of the cube I add two points (Command: Point) to each end making sure to select the “OnMesh” command. Usually doing this in top view is best. I then need to get the dimension from one point to another. I use the "DimAligned" command and Rhino generates the artificial length of that measurement. To ensure the maximum accuracy, in settings under Dimensions I select the maximum number of decimal places. As my scale cube length is in mm I also ensure that the units are in mm when scaling. I can then change this to meters afterwards.

After creating the dimension I select all and “Group” to make sure rhino uses the measurement I just created to scale the model. Typing “scale” now allows you to scale the model, select a point of origin, a point on the model and then type into the command box the number you wish to scale the model by. For me, as one length of my cube is 64mm I divide 64 by the number previously generated. In the video this is 64/2.0348929. When I then change the units, I can see that the new measurement is displayed as 0.064 m, 64 mm. You may also want to change the spacing of lines in your grid, which makes it easier to see whether the model has been correctly scaled.

Now that your model is scaled you can use the same commands “Point” and “DimAligned” to make measurements of your coral (and they will now be real life size). For example you may wish to measure the area of dead coral vs the total “Area” of your coral. Go to town, having a 3D model is like being able to breathe underwater – you can take as many measurements as you want!

Note that this video has NO AUDIO.

Video 3: Measuring Rugosity

Topics covered:

Rugosity
Rhino Commands: Copy, Paste, MeshIntersect, Delete, Rotate, RunPythonScript

Now that we’re happy with the model we can start to take more complicated measurements. One of the most common measurements of habitat complexity is rugosity. This is classically measured by laying a chain of known length along the sea floor and comparing the distance covered on a flat surface. We can now do this using virtual chains in Rhino! The first step is to create a mesh grid of desired size, or use the template available at https://github.com/gracecalvertyoung/Rhino-Python-3D-Coral-Reefs. Then we simply “Copy” the grid directly on to the model and move it into the desired position. Then using the command “MeshIntersect” rhino calculates where the grid makes contact with the coral and this creates the effect of chains being laid on the reef.

We then delete the mesh grid and move the model out the way to isolate the chains. In this example there are overhangs in the coral and so the MeshIntersect command has created extra lines. This means that a little bit of work is needed to isolate the correct chain, once isolated, rotate all the vertical lines by selecting them and typing “Rotate”, click an area in the middle of the lines and the type 90 into the dialog box and rhino will rotate the lines by 90 degrees. Now the lines are horizontal they are ready to be used for rugosity measurements. Type in “RunPythonScript” and select the code for rugostiy available at https://github.com/gracecalvertyoung/Rhino-Python-3D-Coral-Reefs, you will then be prompted to select the chains and press enter. Depending on the number of chains this could take a little processing time. When finished Rhino will create an output of rugosity values which can be copied into excel and you can calculate the average rugosity value for your coral by adding a “=” in front of the copied values.

Note that this video has NO AUDIO.

Questions/comments/something still not working? Comment below and we'll work it out.

Offshore Cybersecurity Musings

1/17/2018

Last autumn in Oxford I had an enlightening coffee with Katherine Fletcher, coordinator of Oxford Cybersecurity. Cybersecurity isn’t a department at Oxford. Rather, it's a community of a dozen or so principal investigators and nearly 90 doctoral students spread across 26 departments, faculties, and institutes. It takes a full-time coordinator like Katherine to keep track of everyone. For example, there are computer scientists working on network security, mathematicians working on anomaly detection, criminal philologists working on understanding adversarial mindsets, polity thinkers working on legal implications (e.g., my fellow Marshall Scholar and author of The Cybersecurity Dilemma), and others. If cybersecurity were its own department, it would pull in ~£33 million in funding. Needless to say, it's an incredibly pertinent topic today as there have been massive cyber attacks in the past few years, on land and at sea, causing billions in damage.

As chaotic as this interdisciplinary subject may appear, it reminds me of the situation Oceans@MIT highlighted at MIT: that an ocean researcher is spread somewhat thin across the Ocean Engineering sub-Department, the Mechanical Engineering Department, the Earth, Atmospheric, and Planetary Sciences Department, and our sister institution, Woods Hole Oceanographic Institution, among others.

U.S. Navy ships conducting security exercises off the coast of Southern California. Credit: U.S. Navy labeled for reuse.

What did we talk about? Cybersecurity meets oceans! I’m interested in the unique challenges facing offshore industries, especially those that have environmental consequences. These include security of cargo, security of open-ocean lanes, and offshore drilling and resource-dredging activities. I’m also interested in coordination between companies, including how new data regulations may yield win-win scenarios with research groups. Companies in the EU will soon be taxed on their data through the General Data Protection Regulation. What if it was more beneficial for companies to buy data off each other instead of, say, conducting repeated seismic airgun survey, which causes wide-spread marine disasters. DNV-GL and other insurance companies have already expressed interest in this sector; e.g., DNV-GL, Lloyd’s Register with the Royal New Zealand Navy, and the group “Be Cyber Aware at Sea,” plus not surprisingly most don't publish much on what they're doing in this sector. I have much to learn.

Automatically Masking delay-Doppler Images

8/16/2017

One thing we did during NASA Frontier Development Lab was write a script to automatically mask the signal in delay-Doppler images. Examples below. This is useful for planetary astronomers. If you'd like to access this and its documentation, please contact me using my contact page. The documentation (version 1.0) is also online here.

This was developed while at NASA Frontier Development Lab, working with Agata Rozek, Sean Marshall, Adam Cobb, Justin Havlovitz, Chedy Raissi, Michael Busch, and Yarin Gal.

Python Script for Retrieving Width of Signal

8/14/2017

The following code snippet was written by Justin Havlovitz during NASA Frontier Development Lab. It returns the width of the signal in a numpy array; see test code below for an example.

Theory of Diminishing Returns

8/9/2017

I've recently been thinking about the theory of diminishing returns. It's like the tan function applied to business/economics concepts:

For example, say you're investing loads of money and effort into safety. That's great, but at some point, no matter how much money you throw at the problem, you can't get to a 100% safe system. Something can always go wrong.

#randomthoughts

Blur Image in Python Without openCV

7/19/2017

It's not secret that installing openCV is by far the most annoying part about using it... but once it's running on your machine it's great! That said, to make a simple script super user-friendly I'd rather not require (or assume) that a user already has compiled openCV. I recently found myself only needing the function cv2.blur from openCV. To implement the same function without openCV, you can use the function scipy.ndimage.filters.convolve; e.g.:

import numpy as np
from scipy import ndimage

A = np.ones((10,10))
pix_blur = (5,5)
k = k = np.ones(pix_blur) / float(pix_blur[0]*pix_blur[1])
B = ndimage.convolve(A, k, mode='mirror')

In the above example, the array B would be the same as from:

import cv2

A = np.ones((10,10))
pix_blur = (5,5)
B = cv2.blur(A, pix_blur)

That said, cv2.blur is marginally faster (it's written C++). e.g., I found that with an 255 x 512 array, cv2.blur was on average 0.008 seconds faster than ndimage.convolve as used above (tested on n = 50 different arrays)---and of course that scales for your larger images. So you might consider writing your code with both options -- e.g., in your import statements have something like:

openCV = False
try:
import cv2
openCV = True
except:
from scipy import ndimage

This was developed with Chedy Raïssi. Hope it helps someone. Please post any comments below.

Reading .dat and .hdr Files in MATLAB

7/10/2017

DAT files (.dat) are a common file type containing data. They can be stand-alone files, but are often accompanied by a configuration file such as a header file (.hdr).

This summer my team at NASA is working with .dat and .hdr files containing radar data of near-earth asteroids. The files are generated from an asteroid modelling program called SHAPE.

We are experimenting with preprocessing the data in MATLAB. There wasn't an existing way to straightforwardly ("straightforwardly") work with the .dat files in MATLAB, so we had to write our own script. In that script we use information contained in the .hdr file to read the .dat file as an image. That code is below in case it's useful to somebody.

EXAMPLE DATA:

run20040918175737.dat
File Size:	1400 kb
File Type:	dat

<- example .dat file

run20040918175737.hdr
File Size:	0 kb
File Type:	hdr

<- associated .hdr file

MATLAB Script:

dathdrconv.m
File Size:	1 kb
File Type:	m

<- .m file containing the MATLAB function datHdrConv

EXAMPLE RESULTS:

Run the function on example data:
>> pixval = datHdrConv('data/run20040918175737');

>> imshow(pixval)
Result below.

>> imshow(pixval, [])
Result below.

This code was developed with Agata Rożek. Please post any comments below.

3D Modelling TEACHING NOTES

5/25/2017

The PowerPoint below describes how to start working in Rhino to measure structural complexity. It's based off my notes from teaching students last summer. The method and associated scripts can be found in our journal publication, Cost and time-effective method for multi-scale measures of rugosity, fractal dimension, and vector dispersion from coral reef 3D models.

Additionally, I've written some guides to specific topics:

3D MODELLING CORAL REEFS

4/16/2017

Our paper on creating/ analysing 3D models of coral reefs is out! Link here.

I'll be running workshops on how to use the method over the next several months in various locations (London, Oxford, DC, Venice). Drop me a line if you're interested, or even better please fill out this form:

https://goo.gl/forms/i84lzrkCi0ZeJkkw2