EVReflex
Dense time-to-impact estimation using events and depth.
Description
This has been accepted to the IROS 2021 conference. The paper (currently pending publication) can be found on ArXiV.
A brief introduction can be found in the following video.
Processing is offline and data must be run through two sub-networks sequentially,
- Optical flow estimation
- Time-to-impact (TTI) estimation
Installation
This repository is built with python 3 (>=3.7) on pytorch lightning.
-
Clone this repository.
git clone --recursive https://gitlab.eps.surrey.ac.uk/cw0071/EVReflex.git cd EVReflex
-
Install the requirements.
I recommend first using a miniconda environment and then following the most recent pytorch install instructions.
# NOTE: Customise for your system/development environment/preferences conda install pytorch torchvision cudatoolkit=11.1 -c pytorch -c nvidia pip install -r requirements.txt pip install -e evreflex
Usage
Remember to activate your conda environment if you have one.
Train:
./train_evreflex.py TRAIN_DIR VAL_DIR OUTPUT_DIR
where TRAIN_DIR
and VAL_DIR
can be LMDB directories, or text files containing a list of LMDB directories.
OUTPUT_DIR
is a tensorboard directory, so to view the output:
tensorboard --bind_all --load_fast=true --logdir OUTPUT_DIR
Dataset
Source files
Format | Links | Size |
---|---|---|
LMDBs (preview) | download | (5.1GB) |
LMDBs (full) | download | (364GB) |
Source ROSbags (preview) | download | (6.6GB) |
Source ROSbags (full) | download | (455GB) |
Command-line instructions
wget https://cvssp.org/data/evreflex/lmdb_full.tar
tar --one-top-level -xvf lmdb_full.tar
rm lmdb_full.tar
Using other datasets is possible, but you may need to adapt the data loader. Other datasets will need to include RGB/greyscale and optical flow, as well as an event stream and depth. You may also wish to convert it to LMDB.
Conversion from rosbag:
./evreflex/tools/rosbags_to_danger_lmdb.py PATH_TO_BAG
NOTE: The version of pickle
used for the LMDB datasets affects how they are encoded/decoded.
The python version in your ROS distribution (for converting rosbags to LMDBs) must be compatible with the python version you would run pytorch with.
The downloadable LMDBs were created with python version 3.8.5
(ROS noetic).
Citation
Celyn Walters and Simon Hadfield. "EVReflex: Dense Time-to-Impact Prediction for Event-based Obstacle Avoidance". In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, Sept. 2021.
@inproceedings{Walters2021,
author = {Walters, Celyn and Hadfield, Simon},
title = {{EVReflex: Dense Time-to-Impact Prediction for Event-based Obstacle Avoidance}},
booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
publisher = {IEEE}
year = {2021}
arxivId = {2109.00405},
url = {http://arxiv.org/abs/2109.00405},
}
Contact
Open an issue or contact me directly.