diff --git a/report/bibliography.bib b/report/bibliography.bib index 3590570c4219d543b2a4b0d4e5804266030587c9..eb9bf40d31939a1c4745292124629eaf245100df 100644 --- a/report/bibliography.bib +++ b/report/bibliography.bib @@ -227,4 +227,12 @@ year = {2017}, publisher = {http://photobyte.org}, howpublished = {\url{http://photobyte.org/137mhz-polar-orbiting-weather-satellites/}} +} + +@misc{bintray, + title = {Welcome to JFrog BinTray}, + author = {Rami Honig}, + year = {2018}, + publisher = {jfrog.com}, + howpublished = {\url{https://www.jfrog.com/confluence/display/BT/Welcome+to+JFrog+Bintray}} } \ No newline at end of file diff --git a/report/chapters/abbreviations.md b/report/chapters/abbreviations.md new file mode 100644 index 0000000000000000000000000000000000000000..eeee79ec0c14a1210f46732e535c775d5369abfa --- /dev/null +++ b/report/chapters/abbreviations.md @@ -0,0 +1,18 @@ +# Abbreviations + +| Abbreviation | Definition | +| ------------ | ---------- | +| CI | Continuous Integration | +| REW | Radio Enabled Web | +| SDR | Software Defined Radio | +| GCC | GNU C Compiler | +| LLVM | Apple's Clang Low Level Virtual Machine | +| MSVC | Microsoft Studio Visual C++ | +| x64 | The 64-bit system | +| amd64 | Also refers to the 64-bit system | +| x86_64 | The 64-bit version of the x86 architecture. Also refers to the 64-bit system | +| x32 | The 32-bit system | +| x86 | The 32-bit instruction system, refers to the 32-bit system | +| i686 | The 32-bit instruction set, part of the x86 family | +| win32 | Windows specific 32-bit architecture, used by MSVC | +| win64 | Windows specific 64-bit architecture, used by MSVC | diff --git a/report/chapters/abstract.md b/report/chapters/abstract.md index 8957164d154f051f06cacc5d24c102652fa8994d..b15b9c92d4a40f07a88285cb764ee6a7f4ec85db 100644 --- a/report/chapters/abstract.md +++ b/report/chapters/abstract.md @@ -1 +1,9 @@ # Abstract + +In this report we look at the design and implementation of small scale websites transmission over FM Radio. The project, "Radio Enabled Web", consists of a Encoder and a Decoder parts. These two applications, more precisely shared libraries, can encode a static HTML website into an audio file. This audio file can be then transmitted using conventional FM Transmitter, and then received on the other side. The received audio is then decoded via the Decoder application, and shown to the user via browser. + +This project, Radio Enabled Web, utilizes the FM radio transmitter and a receiver to transmit the contents of the website. However, it is possible to use any other carrier that uses audio, for example an amateur short-band radio. + +The Radio Enabled Web (REW), splits both the Decoder and Encoder into small standalone blocks, which can be used by the user. These blocks are C++ classes, and are purely optional. These blocks can be assembled into a custom pipeline, or integrated into any other viable language. The main part of this project is the GUI application(s) and Command Line Interfaces that can be used out of the box. + +This report mainly consists of software design, implementation, and testing of the Radio Enabled Web. diff --git a/report/chapters/acknowledgement.md b/report/chapters/acknowledgement.md index 132007919c5788af9c9040f0524e7921e6891d90..e415e6e870b7b8147037864194b2b6b3180b7012 100644 --- a/report/chapters/acknowledgement.md +++ b/report/chapters/acknowledgement.md @@ -1 +1,4 @@ # Acknowledgement + +I would like to thank the grocery shop under the University library for a constant supply of energy drinks, which were greatly needed to complete this project. + diff --git a/report/chapters/acronyms.md b/report/chapters/acronyms.md deleted file mode 100644 index e4acb8789430c51d9730823d31d1826ca09f20ea..0000000000000000000000000000000000000000 --- a/report/chapters/acronyms.md +++ /dev/null @@ -1 +0,0 @@ -# Acronyms diff --git a/report/chapters/design.md b/report/chapters/design.md index 9611023cd3640fe58d3b5b4197356d74224f34dd..b1c06681bfed11cee2753675e6a3e0320ce73d47 100644 --- a/report/chapters/design.md +++ b/report/chapters/design.md @@ -2,29 +2,29 @@ ## 4.1 Software -The aim of this project is not only to transmit entire web pages through an FM radio, but also to provide the API for the user. The API, in form of a library, must be easily understood and well documented. The API will provide a basic set of building blocks for the user to arrange them in any desirable pipeline. These blocks in a form of a pipeline can be used to encode or decode the transmission, or further enhance the library. The aim is to also provide a basic CLI (Command Line Interface) with pre-built pipeline that can be used to encode and then decode the transmission. Additionally, to provide a http server for localhost, in order for the user to be able to browse the decoded transmission through a web browser of their choice. +The aim of this project is not only to transmit entire web pages through an FM radio, but also to provide the API for the user. The API, in form of a library, must be easily understood and well documented. The API will provide a basic set of building blocks for the user to arrange them in any desirable pipeline. These blocks, in a form of a pipeline, can be used to encode or decode the transmission, or further enhance the library. The aim is to provide a basic CLI (Command Line Interface) with pre-built pipeline that can be used to encode and then decode the transmission. Additionally, to provide a HTTP server for localhost, in order for the user to be able to browse the decoded transmission through a web browser of their choice. ### 4.1.1 Programming language and environment -C++ was chosen as the implementation language for this project. The main reason behind this decision was performance benefits of compiled languages over an interpreted languages [@languagesbenchmark]. Another reason for the chosen language is that I have past experience working with this language in my own personal projects. C++ is a good choice for an open source library as it can be integrated into interpreter library. For example, using pybind11 [@pybind11] to create python bindings, or by using Node [@nodejsaddons] to create a JavaScript library. C++ libraries can be also easily integrated into C# Net Framework via dllimport [@csharpdllimport], Ruby [@rubyextensions], and many more. +C++ was chosen as the implementation language for this project. The main reason behind this decision was performance benefits of the compiled languages over the interpreted languages [@languagesbenchmark]. Another reason for the chosen language is that I have past experience working with this language in my own personal projects. C++ is a good choice for an open source library, as it can be integrated into interpreted language. For example, using pybind11 [@pybind11] to create python bindings, or by using Node [@nodejsaddons] to create a JavaScript library. C++ libraries can be also easily integrated into C# Net Framework via dllimport [@csharpdllimport], Ruby [@rubyextensions], and many more. -Additionally, the decoding part of this project needs to listen to the raw audio input of the user's machine. Therefore, it is essential that the application needs to run at real-time speed. The use of C++ will greatly improve the efficiency of the decoding algorithm. +Additionally, the decoding part of this project needs to listen to the raw audio input of the user's machine. Therefore, it is essential that the application needs to run at real-time speeds. The use of C++ will greatly improve the efficiency of the decoding algorithm. -The C++ offers a standard library [@stdlib] which contains the basic building blocks for the project needs. However, unlike Java and other high level languages, C++ does not offer out of the box networking library or filesystem library. The standard library, or STD for short, only provides us with language specific additions, such as: strings, vectors (dynamic arrays), optionals, and mainly the smart pointers. The smart pointers are a new addition to the C++ standard library version 11. It is a word of mouth that C or C++ applications are memory leaky, meaning, the application built in C or C++ are eventually leak memory and therefore use all of the available resources. However, these smart pointers are meant to help the programmer to reduce the likelihood of that happening. Normally when a new object is allocated, the standard way of doing it is through malloc (in C) or new operator (in C++). The memory needs to be then deleted via free (in C) or operator delete (in C++). The smart pointers encapsulate the raw pointers in an smart pointer object instance, and delete the memory when the instance fall out of scope. The project will have a heavy use of this feature. +The C++ offers a standard library [@stdlib], which contains the basic building blocks for what the project needs. However, unlike Java and other high level languages, C++ does not offer out of the box networking library or filesystem library. The standard library, or STD for short, only provides us with language specific additions, such as strings, vectors (dynamic arrays), optionals, and the smart pointers. The smart pointers are a new addition to the C++ standard library version 11. It is a word of mouth that C or C++ applications are memory leaky, meaning, the application built in C or C++ will eventually leak memory and therefore use all of the available resources. However, these smart pointers are meant to help the programmer to reduce the likelihood of that happening. Normally when a new object is allocated, the standard way of doing it is through malloc (in C) or new operator (in C++). The memory needs to be then deleted via free (in C) or operator delete (in C++). The smart pointers encapsulate the raw pointers in an smart pointer object instance, and delete the memory when the instance fall out of scope. The project will have a heavy use of this feature. The C++ comes with many versions, the standards, from C++98 (1998) up to C++20 (2020). The project will use the C++ 17 standard. The reason behind this chose is that the smart pointers have been first introduced in C++11 and later enhanced with additional features in C++14. Additionally, C++17 offers better metaprograming features than its predecessors. However, due to issues with the continuous integration and Mac OSX, the standard had to be lowered to C++ 14. -For the unit testing the Catch2 [@catch2] framework was chosen. The framework does not produce any static or dynamic libraries, instead only a single header file is needed for a integration. This Catch2 unit testing framework also offers a real time expression expansion for the each assert macro. This is extremely beneficial when it comes to asserting comparisons between two values in the unit tests. For example, when testing if a `std::vector` contains a specific number of elements, we can call `REQUIRE(vec.size() == 10);`. If this assertion fails, the Catch2 will print it out as an error `8 == 10`. This also works for more complicated types such as strings. +For the unit testing, the Catch2 [@catch2] framework was chosen. The framework does not produce any static or dynamic libraries, instead only a single header file is needed for a integration. This Catch2 unit testing framework also offers a real time expression expansion for the each assert macro. This is extremely beneficial when it comes to asserting comparisons between two values in the unit tests. For example, when testing if a `std::vector` contains a specific number of elements, we can call `REQUIRE(vec.size() == 10);`. If this assertion fails, the Catch2 will print it out as an error `8 == 10`. This also works for more complicated types such as strings. ### 4.1.2 Package Manager and build system -The C++ does not have an official package manager. There are many third party options such as Microsoft's vcpkg [@vcpkg] or conan [@conan]. The vcpkg is available on all three major platforms (Windows, Linux, and OSX) and offers packages in i686, amd64, or arm. This seems as a great choice for a package manager of this project, however there are some issues with it. First, the vcpkg does not offer a static binaries of the library on Linux and OSX, only dynamic binaries. This is also reversed on Windows where the the manager offers mainly static binaries with little or no choice of using dynamic binaries instead. Second, the vcpkg requires user's knowledge about complicated C++ building mechanism. This breaks the idea of this project, the entire project must be easy to understand and build by the user. The conan has been discarded due to a reason that the official documentation lacks the essentials, and the offer of the libraries is not comprehensive as we need. This leaves us with no other choice than building the libraries ourselves. +The C++ does not have an official package manager. There are many third party options such as Microsoft's vcpkg [@vcpkg] or conan [@conan]. The vcpkg is available on all three major platforms (Windows, Linux, and OSX) and offers packages in i686, amd64, or arm. This seems as a great choice for a package manager of this project, however there are some issues with it. First, the vcpkg does not offer a static binaries of the library on Linux and OSX, only dynamic binaries. This is also reversed on Windows where the the manager offers mainly static binaries with little or no choice of using dynamic binaries instead. Second, the vcpkg requires user's knowledge about complicated C++ building mechanism. This breaks the idea of this project, the entire project must be easy to understand and easily built by the user. The conan has been discarded due to a reason that the official documentation lacks the essentials, and the offer of the libraries is not comprehensive as need for this project. This leaves us with no other choice than to build the libraries ourselves. Instead of using a package manager, the Git Submodules [@gitsubmodules] will be used. The Git submodules is a way of adding other Git repositories into our project. The libraries will be simply added as a git submodule and then fetched from the remote git origin into a subfolder of the project. The contents of those libraries will not be pushed to the main repository, instead only the submodule metadata needed to fetch the library. This slightly introduces a complexity of the building system, but allows us to chose a specific commit or compilation flags for each of the library we will need. -The C++ offers no standard way of building the application from sources. There are many different offers of tools when it comes to building and generating project files for C++ applications. One of the popular one is automake [@automake]. The automake was primarly designed to work only on Linux based systems, and has almost non-existing support for Windows without using a virtual machine or a port of Linux tools. Automake is used by many open source C++ libraries, however due to Linux only support it can not be used in the project. Instead, CMake [@cmake] will be used. CMake offers a simple intuitive scripting language which can be used to identify source files, add include directories, and link libraries for the target executable (this project) or a library. CMake fulfils the needs as it is easy to install and the commands to build the project are easy to follow for a beginner. +The C++ offers no standard way of building the application from sources. There are many different offers of tools when it comes to building and generating project files. One of the popular one is automake [@automake]. The automake was primarly designed to work only on Linux based systems, and has almost non-existing support for Windows without using a virtual machine or a port of Unix shell tools. Automake is used by many open source C++ libraries. However, due to Linux only support, it can not be used in this project. Instead, CMake [@cmake] will be used. CMake offers a simple intuitive scripting language, which can be used to identify source files, add include directories, and link libraries for the target executable (this project) or a library. CMake fulfils the needs, as it is easy to install, and the commands to build the project are easy to follow for a beginner. -Additionally CMake offers an optional variables to be added into the building script. For example, it is possible to add `BUILD_EXAMPLES` or `BUILD_TESTS` conditional boolean variable which a user can set to true or false. Using this variables, the project's CMake script file `CMakeLists.txt` will conditionally include examples and tests into the final generated project files. These project files can be anything from a simple Makefile, Visual Studio solution files, or even Mac XCode files. The CMake also offers a GUI which greatly reduces the complexity of user experience. +Additionally, CMake offers an optional variables to be added into the building script. For example, it is possible to add `BUILD_EXAMPLES` or `BUILD_TESTS` conditional boolean variables, which a user can set to true or false. Using this variables, the project's CMake script file `CMakeLists.txt` will conditionally include examples and tests into the final generated project files. These project files can be anything from a simple Makefile, Visual Studio solution files, or even Mac XCode files. The CMake also offers a GUI which greatly reduces the complexity of the user experience.  @@ -32,14 +32,14 @@ Additionally CMake offers an optional variables to be added into the building sc The following third party libraries have been chosen for this project. -* argagg [@argagg] - This library provides a command line argument parsing. Needed for the command line interface part of this project. -* asio [@asio] - A library to create a basic http server. This server will be listening on the localhost and serving the decoded transmission to the user. +* argagg [@argagg] - This library provides a command line argument parsing. Needed for the command line interface part of this project. The library only needs a list of arguments and their description. Additionally, a help page is automatically generated using the description. +* asio [@asio] - A library to create a basic http server. This server will be listening on the localhost and serving the decoded transmission to the user. The main part of the Asio is asynchronous networking, which will be needed when we want to separate decoding and serving via threads. * dirent [@dirent] - A library to wrap Windows specific file system functions to be compatible with Unix functions. The original dirent is a library provided inside of every Linux based system. However due to the differences between Linux and Windows, this library provides a translation for Windows compilers between those two environments. This is only needed when building on Windows and will be enabled or disabled by CMake during configuration. -* zlib [@zlib] - This is a common compression library. It will be used to compress the source website into raw stream of bytes before being transformed into packets and transmitted. -* portaudio [@portaudio] - A cross platform audio input/output library. This library will be used by the Decoder to listen for transmission. -* qt5 [@qt5] - A cross platform GUI library developed by Nokia. +* zlib [@zlib] - This is a common compression library. It will be used to compress the source website into raw stream of bytes before being transformed into packets and transmitted. The zlib is well used compression library among many real world applications. Therefore, its documentation has been perfected, and easily understood, over the course of many years. +* portaudio [@portaudio] - A cross platform audio input/output library. This library will be used by the Decoder to listen for transmission. Many Linux distributions already provides this library by default, which we need to take into consideration in our CMake scripts. +* qt5 [@qt5] - A cross platform GUI library developed by Nokia. This library provides a basic set of UI widgets, alongside with some more complex ones. In this project, only the basic set will be used, to reduce the size of the output executable. The Qt5 has a commercial fee, however because this project is open source, we are allowed to use Qt5 without paying for it. -All of the third party libraries are included as a git submodule. Only the libraries that need compilation step, such as zlib, will be included in the CMake generator. Any other libraries that are header only, meaning they do not require compilation step, are only included as a include directory for the compiler. +All of the third party libraries are included as a git submodule. Only the libraries that need compilation step, such as zlib, portaudio, and Qt5, will be included in the CMake generator. Any other libraries that are header only, meaning they do not require compilation step, are only included as a include directory for the compiler. ### 4.1.4 Integrated development environment @@ -71,7 +71,7 @@ Microsoft Visual Studio 2017 [@visualstudio] with ReSharper C++ [@resharper] has ## 4.2 Continuous Integration -The continuous integration is an essential part when building C++ applications or libraries. Due to a reason that the output binary (either an executable or a dynamic library) is platform dependent, we will need to test the compilation and tests for all of the platforms we wish to distribute to. The continuous integration is an online service which listens to changes within the repository and launches a build for every change. This change can be a commit, new tag, or a pull request. With a use of this service we can ensure that the code written is platform independent. In this project, the three major platforms have been included: Windows, Linux, and OSX. Additionally, the i686 and amd64 versions need to be included in the Windows testing. The status of all of the continuous integration is listed in the README file of the repository. +The continuous integration is an essential part when building C++ applications or libraries. Due to a reason that the output binary (either an executable or a dynamic library) is platform dependent, we will need to test the compilation and unit tests for all of the platforms we wish to distribute to. The continuous integration is an online service which listens to changes within the repository, and launches a build for every introduced change. This change can be a new commit, new tag, or a pull request. With a use of this service, we can ensure that the code written is platform independent. In this project, the three major platforms have been included: Windows, Linux, and OSX. Additionally, the i686 and amd64 versions need to be included in the Windows testing. The status of all of the continuous integration is listed in the README file of the repository.  @@ -79,7 +79,7 @@ The continuous integration is an essential part when building C++ applications o The AppVeyor [@appveyor] is a continuous integration that offers building software on Windows virtual machines. The MinGW and MSVC compilers (both in amd64 and i686 settings) have been included in the building script. During the build, the latest commit is fetched from the repository, and using a custom script provided, the project is built and the output packed into a single artifact zip file. The artifact contains the binary distribution of the command line interface, alongside with the dynamic library and header files. -Due to the CRT [@iteratordebug] in MSVC compiler, the build system had to be adjusted for Visual Studio 2017. While the output binary for MinGW, or Unix system is built in Release mode only, The Visual Studio build needs to output both Debug and Release binaries. The Debug mode produces a non optimized library with debug symbols and the Release mode produces optimized library with no debug symbols attached. These debug symbols are only needed for the development of an application that used this library. However, due to the CRT in MSVC compiler, the CRT library on Windows is not compatible if the CRT used in the built binary does not match the debug level used. For example, if the user decides to download the Release version, and tries to use it in their own custom application built in Debug mode, it will cause compilation error. Therefore, both Debug and Release versions must be published. This does not apply for MinGW, GCC, or Clang compiler. +Due to the CRT [@iteratordebug] in MSVC compiler, the build system had to be adjusted for Visual Studio 2017. While the output binary for MinGW, or Unix system is built in the release mode only. The Visual Studio build needs to output both the debug and the release binaries. The debug mode produces a non optimized library with debug symbols attached. The release mode produces optimized and minimized library with no debug symbols. These debug symbols are only needed for the development of an application that used this library. However, due to the CRT in MSVC compiler, the CRT library on Windows is not compatible if the CRT used in the built binary does not match the debug level used. For example, if the user decides to download the release version, and tries to use it in their own custom application built in debug mode, it will cause compilation error. Therefore, both debug and release versions must be published. This does not apply for MinGW, GCC, or Clang compiler. Only Visual Studio 2017 in Win32 and Win64 mode and MinGW 6.3.0 in i686 and x86_64 mode are used in this continuous integration. @@ -87,7 +87,7 @@ Only Visual Studio 2017 in Win32 and Win64 mode and MinGW 6.3.0 in i686 and x86_ ### 4.2.2 Travis CI -The Travis CI [@travisci] is a continuous integration that offers building software on both Linux and Mac OSX. Similarly as in AppVeyor, the build is launched on every new commit pushed to the repository. The Linux GCC 7.3.0 in x64 mode and Apple Clang LLVM 8.0.0 in x64 mode is used during compilation. +The Travis CI [@travisci] is a continuous integration that offers building software on both Linux and Mac OSX. Similarly, as in AppVeyor, the build is launched on every new commit pushed to the repository. The Linux GCC 7.3.0 in x64 mode and Apple Clang LLVM 8.0.0 in x64 mode is used during compilation.  @@ -95,17 +95,21 @@ The Travis CI [@travisci] is a continuous integration that offers building softw The Circle CI [@circleci] is a continuous integration that only offers building software on Linux. As in both AppVeyor and Travis CI, The Circle CI launches the build on every new commit. For the purposes of this project, only Linux GCC 5.5, 6.4, 7.3, and 8.2 in x64 mode is used. -After the successful build of the software on various version of GCC, the documentation building process is run in two separate jobs. During this, the first job "build_docs" will run Doxygen [@doxygen] which will parse the header files and produces XML output. This XML output is further processed by DoxyBook [@doxybook] and exported as Markdown into a temporary folder. In the second job, the last job, the Markdown documentation is converted into HTML static pages via VuePress [@vuepress], and finally pushed into the `gh-pages` branch on the GitHub mirrored repository. +After a successful build of the software, on various version of GCC, the documentation building process is run in two separate jobs. During this, the first job "build_docs" will run Doxygen [@doxygen] which will parse the header files and produces an XML output. This XML output is further processed by DoxyBook [@doxybook] and exported as Markdown into a temporary folder. In the second job, the final job, the Markdown documentation is converted into HTML static pages via VuePress [@vuepress], and finally pushed into the `gh-pages` branch on the GitHub mirrored repository. These HTML files can be browsed online by accessing the repository.  ### 4.2.4 Distribution of binary artifacts -TBA: Bintray artifacts +When the artifacts are built on the continuous integrations, they are pushed to the BinTray [@bintray] into a custom generic repository. This repository is no more than a simple file storage with versioning labelling system. The reason behind BinTray is that if we wish to distribute the library and the header files to the user, we need to make it available to them. If this was a Java application, we could distribute the library via Maven. However, this is not possible with C++, as there is no official package distribution software to do it for us. We could host the artifacts on University's GitLab, but due to the problem with non-working pipelines, I have decided not to try this feature as it may be a waste of time. Instead, we could use the mirror repository on GitHub to host the artifacts. This can work until we reach the 1GB of used space. After this limit is reached, a friendly warning is sent to the owner forcing them to release some space. This has lead me to BinTray which allows up to 10GB of space and up to 1TB of download traffic, more than needed for these artifacts. + +When the artifact is built, it is compressed into a zip file. The versioning of these zip files is the following this schema: `rew-<version>-<machine>-<compiler>.zip`, where the `<version>` is the version described by the git, the tag, and the `<machine>-<compiler>` are retrieved from the compiler itself. This versioning is useful for the developer, if they wish to integrate this software in their own application. However, for a common user, this is a highly confusing naming scheme. It would be more beneficial to host a website, for example a product website, with three simple options (Windows, Linux, and OSX) for downloading the software. This product website does not exist, but could be part of the future of this project. + + ### 4.2.5 Documentation -The documentation is implemented via Doxygen [@doxygen] and a custom Doxygen xml parser DoxyBook [@doxybook]. The Doxygen parses the header files of this project and outputs an XML structure representing the classes, namespaces, and functions. This XML structure is then processed by DoxyBook, an open source command line utility created and maintained by myself. This utility converts the XML into Markdown files. The markdown files are then processed by VuePress [@vuepress]. The VuePress is a Vue.js [@vue] based platform for creating blogs and static page documentation. The reason for this choice is that the files generated can be browsed offline without a use of a HTTP server. Additionally, the generated static HTML files can be also uploaded to GitHub's gh-pages. The gh-pages is a name of a branch of a repository. If the branch is populated with HTML files, the GitHub will then automatically publish them online, without a use of a server. These files, the documentation, can be then viewed by anyone. +The documentation is implemented via Doxygen [@doxygen] and a custom Doxygen xml parser DoxyBook [@doxybook]. The Doxygen parses the header files of this project and outputs an XML structure representing the classes, namespaces, and functions. This XML structure is then processed by DoxyBook, an open source command line utility created and maintained by myself. This utility converts the XML into Markdown files. The markdown files are then processed by VuePress [@vuepress]. The VuePress is a Vue.js [@vue] based platform for creating blogs and static page documentation. The reason for this choice is that the files generated can be browsed offline without a use of a HTTP server. Additionally, the generated static HTML files can additionally be uploaded to the GitHub's gh-pages. The gh-pages is a name of a branch of a repository. If the branch is populated with HTML files, the GitHub will then automatically publish them online, without a use of a server. These files, the documentation, can be then viewed by anyone. ~~~{.cpp caption="Example of Doxygen comments within the header files"} REW_NAMESPACE{ @@ -133,9 +137,9 @@ REW_NAMESPACE{ ### 4.3.1 System structure -The system was split into two separate components: Encoder and Decoder. The Encoder will be designed to only accept a folder containing all website files, and outputs a raw stream of samples that can be saved into an audio file. These samples, a PCM stream, is a stream of 32bit floating point values representing the entire encoded website as audio. The samples need to be normalized values between -1.0f and +1.0f, so that they have the maximum gain possible, without causing audio clipping. Any values beyond or under +/-1.0f will be wrapped over, causing glitches. Any audio PCM sample of 0.0f is considered to be infinitely silent, while +/-1.0f is considered highest possible gain. The Decoder needs to accept these PCM samples, either through an audio input or from an audio file. The Decoder will then process these samples and extracts the compressed stream of data that holds the website contents. +The system was split into two separate components: the Encoder and the Decoder. The Encoder will be designed to only accept a folder containing all website files, and outputs a raw stream of samples that can be saved into an audio file. These samples, a PCM stream, is a stream of 32bit floating point values representing the entire encoded website as an audio. The samples need to be normalized values between -1.0f and +1.0f, so that they have the maximum gain possible, without causing audio clipping. Any values beyond or under +/-1.0f will be wrapped over, causing glitches - clipping. Any audio PCM sample of 0.0f is considered to be infinitely silent, while +/-1.0f is considered highest possible gain. The Decoder needs to accept these PCM samples, either through an audio input or from an audio file. The Decoder will then process these samples and extracts the compressed stream of data that holds the website contents. -Command Line Interface will be provided alongside of the encoder and decoder libraries, so that the user can use these application without integrating the library in their own software. However, our target demographic may not have the technical skills to operate the CLI. Therefore, a GUI application, one for the Encoder and one for the Decoder, will be part of the project and binary distribution provided for the user. This GUI will be implemented via Qt5 [@qt5], a cross platform library developed by Nokia. GTK+ [@gtk] was considered at the beginning, however due to incompatibility on newer Mac OSX versions, the GTK will not correctly work unless the user enables X11 manually. Therefore, QT5 for its wide compatibility was chosen as the GUI framework for this project. The binary distribution of Qt5 will be distributed alongside the GUI binary distribution of the Encoder and Decoder. This is necessary to ensure that the target user does not need to install Qt5 globally into their machine. +Command Line Interface will be provided alongside of the encoder and decoder libraries, so that the user can use these application without integrating the library in their own software. However, our target demographic may not have the technical skills to operate the CLI. Therefore, a GUI application, one for the Encoder and one for the Decoder, will be part of the project and binary distribution provided for the user. This GUI will be implemented via Qt5 [@qt5], a cross platform library developed by Nokia. GTK+ [@gtk] was considered at the beginning, however due to incompatibility on newer Mac OSX versions, the GTK will not correctly work unless the user enables the X11 manually. Therefore, QT5 for its wide compatibility, was chosen as the GUI framework for this project. The binary distribution of Qt5 will be distributed alongside the GUI binary distribution of the Encoder and the Decoder. This is necessary to ensure that the target user does not need to install Qt5 globally into their machine. All of the dependencies and target binaries are defined inside of the CMakeLists.txt files, one file for each Encoder or Decoder component. @@ -143,9 +147,9 @@ All of the dependencies and target binaries are defined inside of the CMakeLists ### 4.3.2 API Design -Because this project is meant to be open source, it must have a consistent API that is intuitive and easy to integrate to a custom software. Additionally, the algorithm must be adjustable. To satisfy these requirements, the algorithm was split into multiple blocks. Each block, using its own specific input data type, will consume the input and outputs it either in the same data type, or a new data type. Each block only does a single specific work, and is completely standalone. The blocks can be arranged together to form a pipeline (can be also described as flowgraph). This kind of implementation was inspired by GNU Radio [@gnuradio]. +Because this project is meant to be open source, it must have a consistent API that is intuitive and easy to integrate to a custom software. Additionally, the algorithm must be adjustable. To satisfy these requirements, the algorithm was split into multiple blocks. Each block, using its own specific input data type, will consume the input and outputs it either in the same data type, or a new data type. Each block only does a single specific work and is completely standalone. The blocks can be arranged together to form a pipeline (can be also described as flowgraph). This kind of implementation was inspired by the GNU Radio [@gnuradio]. -Each block is a derived class of Input and Output class. However, not every block needs to inherit both Input and Output classes. For example, a file audio source block, does not need to have an input, it only produces an output. An abstract Input class contains a virtual method `process()`. This method is called by the abstract Output class in order to forward the data. Due to the strict static typing of C++, the combination of Input and Output derived classes will always match. +Each block is a derived class of the Input and the Output base classes. However, not every block needs to inherit both Input and Output classes. For example, a file audio source block, does not need to have an input, it only produces an output. An abstract Input class contains a virtual method `process()`. This method is called by the abstract Output class in order to forward the data. Due to the strict static typing of C++, the combination of Input and Output derived classes will always match. ~~~{.cpp caption="The abstract Input with omitted code"} template<typename T> @@ -177,19 +181,19 @@ private: Using this idea, the following blocks have been implemented as listed below: -* `Clamp<typename I, typename O>` - A block that clamps the input based on a specific minimum and maximum variables. This is necessary in order to ensure that the algorithm produces samples in -1.0f and +1.0f range. -* `Merge<typename T, size_t N>` - A block that can merge N number of streams into a single stream for further processing. During this, the data is not converted to another data type, instead it is re-used. -* `Multiply<typename T>` - This block simply multiplies the input stream by a fixed value. -* `AudioSource` - (Decoder only) An abstract block that is used to produce an output of 32bit float PCM samples. This block needs to be further implemented. An implementation of `AudioReader` class, for reading audio files, and `AudioListener` class, for listening to the microphone audio input on the machine. TBA: WavAudioReader + PhysicalAudioListener. -* `FirFiler` - (Decoder only) Filters out a specific frequency from a raw stream of PCM samples. -* `Goertzel` - (Decoder only) Filters the input stream of 32bit float PCM samples and outputs a magnitude based on the frequency being tracked. +* `Clamp<typename I, typename O>` - A block that clamps the input based on a specific minimum and maximum variables. This is necessary in order to ensure that the algorithm produces samples in -1.0f and +1.0f range. At the same time, the output is converted into a different data type, specified by the template parameters. +* `Merge<typename T, size_t N>` - A block that can merge N number of streams into a single stream for further processing. During this, the data is not converted to another data type, instead it is re-used. During this process, the data is temporarily stored within the block. The data is forwarded once the length of all input paths matches, and then released and the input buffer reset. +* `Multiply<typename T>` - This block simply multiplies the input stream by a fixed value. No data type conversion happens. +* `AudioSource` - (Decoder only) An abstract block that is used to produce an output of 32bit float PCM samples. This block needs to be further implemented. An implementation of abstract `AudioReader` class, for reading audio files, and abstract `AudioListener` class, for listening to the microphone audio input on the machine. These two classes are then further implemented as concrete `WavAudioReader` and `PhysicalAudioListener` classes. `WavAudioReader` is meant to read audio samples from a WAV file, while the `PhysicalAudioListener` reads the audio samples from a physical input device. +* `FirFiler` - (Decoder only) This block is meant to filter out noise. It works by listening to a specific frequency from a raw stream of PCM samples. +* `Goertzel` - (Decoder only) Filters the input stream of 32bit float PCM samples and outputs a magnitude based on the frequency being tracked. The block needs to be initialized with a specific window size. This window size greatly changes the output magnitude and the accuracy of the algorithm as a whole. * `ByteAssembler` - (Decoder only) Takes a raw stream of Goertzel processed data and assembles a byte. * `PacketAssembler` - (Decoder only) Takes a raw stream of bytes and converts it into a Packet. An 8bit CRC is applied before the packet is constructed in order to verify its integrity. -* `FileAssembler` - (Decoder only) Takes a stream of packets and assembles the output file if all of the packets have been provided. This block caches the packets based on a file ID, and only outputs a file if all of the packets are provided and can be decompressed. +* `FileAssembler` - (Decoder only) Takes a stream of packets and assembles the output file, only if all of the packets have been provided. This block caches the packets based on a file ID, and only outputs a file if all of the packets are provided and can be decompressed. * `FrameSync` - (Decoder only) Waits for a frame synchronization and forwards a specific number of samples further down the pipeline. The amount of samples to forward is determined by the size of a single packet. This specific amount is necessary in order to ensure that the samples will or will not contain the packet we need to decode. -* `AudioSink` - (Encoder only) Similarly as in `AudioSource` except its role is reversed. An abstract class that is meant to be derived into a custom implementation in order to output the samples into a WAV file or a speaker output on the machine. -* `FileLoader` - (Encoder only) Loads a file into a memory and appends the file name at the start of the byte array representing the contents. Simply forwards the file in memory further down the connected blocks. -* `PacketGenerator` - (Encoder only) Accepts an in-memory file, and using a zlib compression the packets are generated. +* `AudioSink` - (Encoder only) Similarly as in `AudioSource` except its role is reversed. An abstract class that is meant to be derived into a custom implementation in order to output the samples into a WAV file or a speaker output on the machine. This class is further specialized as abstract `AudioWriter`, and a block `WavAudioWriter` has been provided as the concrete class. Additionally, `PhysicalAudioOutput` is provided as well. +* `FileLoader` - (Encoder only) Loads a file into a memory and appends the file name at the start of the byte array representing the contents. Simply forwards the contents further down the connected blocks. +* `PacketGenerator` - (Encoder only) Accepts an in-memory file, and using a zlib compression, the data is compressed and packets are generated. * `SamplesGenerator` - (Encoder only) Constructs samples using a raw stream of bytes. In this case, the stream of bytes are the packets from the `PacketGenerator`. More custom blocks can be implemented by the user using the abstract Input and Output classes. The new blocks will automatically be compatible with the pre-existing blocks as listed above. @@ -198,7 +202,7 @@ The detailed explanation behind the functionality of these blocks is explained i ### 4.3.3 Fast Fourier Transform and Goertzel Algorithm -During the prototyping phase, a Fast Fourier Transform (FFT) was considered in order to listen for the two specific frequencies representing the binary stream of data. After a lot of research done on this topic, a decision was made that using FFT was computationally heavy for a simple work we need. In this project, we are only looking at two specific frequencies, representing high and low bits, and a complete silence, representing no transmission. In the end, Goertzel Algorithm was used to do this. While FFT produces a buckets for a range of frequencies, Goertzel only produces a magnitude of a specific frequency we are trying to listen to. If no frequency is detected, the output of the algorithm is close or equal to zero. Goertzel can be sumarized in the following code: +During the prototyping phase, a Fast Fourier Transform (FFT) was considered in order to listen for the two specific frequencies representing the binary stream of data. After a lot of research done on this topic, a decision was made that using FFT was computationally heavy for a simple work we need. In this project, we are only looking at the two specific frequencies, representing high and low bits, and a complete silence, representing no transmission. In the end, the Goertzel algorithm was used to perform this process. While the FFT produces buckets for a range of frequencies, the Goertzel only produces a magnitude of a specific frequency we are trying to listen to. If no frequency is detected, the output of the algorithm is close or equal to zero. The Goertzel can be sumarized in the following code: ~~~{.cpp caption="Processing samples via Goertzel to produce a magnitude."} void processSamples(const float* samples, const size_t length) { @@ -214,7 +218,7 @@ void processSamples(const float* samples, const size_t length) { } ~~~ -The same algorithm can be represented in less than 23 assembly instructions. Not only Geortzel is faster than FFT, but it also solves the same problem. +The same algorithm can be represented in less than 23 assembly instructions. Not only the Goertzel is faster than the FFT, but it also solves the same problem we have. ~~~{.S caption="Processing samples via Goertzel represented as assembly. Compiled via GCC 8.3 amd64 with -O2."} processSamples(float const*, unsigned long): @@ -244,11 +248,11 @@ processSamples(float const*, unsigned long): ### 4.3.4 Packet design and the Algorithm -In order to transmit the files, or any large data, over an FM radio, we need to convert the files into an audio. There are multiple ways how to do this. For this project, Frequency Shift Keying (FSK) [@fsk] will be used. FSK encodes the stream of bits into two different frequencies. For the purpose of naming simplicity, the frequency to encode the low bit will be called low frequency, and the to encode the high bit will be called high frequency. To implement this, a simple `ToneSynth` class has been constructed. Combined with `SamplesGenerator` block, it creates PCM samples from a raw stream of bytes. Each byte is encoded into a fixed number of samples. This way we ensure that the Decoder can listen to a fix sized block of samples to decode it back into a byte. Additionally, frequencies 2400hz and 4800hz have been chosen for the low and high frequencies. Each byte is additionally delimited by a silence with a width of a single encoded bit. This way, we can easily calculate the expected number of samples for the entire input. +In order to transmit the files, or any large data, over an FM radio, we need to convert the files into an audio. There are multiple ways how to do this. For this project, Frequency Shift Keying (FSK) [@fsk] will be used. The FSK encodes the stream of bits into two different frequencies. For the purpose of the naming simplicity, the frequency to encode the low bit will be called low frequency, and the frequency to encode the high bit will be called high frequency. To implement this, a simple `ToneSynth` class has been constructed. Combined with the `SamplesGenerator` block, it creates PCM samples from a raw stream of bytes. Each byte is encoded into a fixed number of samples. This way we ensure that the Decoder can listen to a fix sized block of samples, in order to decode it back into a byte. Additionally, frequencies 2400hz and 4800hz have been chosen for the low and high frequencies. Each byte is additionally delimited by a silence with a width of a single encoded bit. This way, we can easily calculate the expected number of samples for the entire input. The choice of having the high frequency the double of the low frequency, is to ensure that the start and the end of the sine wave matches. This increases the accuracy of the Goertzel algorithm.  -Next, to decode the audio and make sense of the PCM samples, we apply the Goertzel algorithm. The Goertzel algorithm works in fixed number of samples, a window. The size of the Goertzel window greatly changes the magnitude of the output. After many attempts and tests made, a window size of a length of a single encoded bit has been chosen. Using sample rate of 48Khz and a single bit length of 1.25 millisecond, the window size is 60 samples. This window also slides per single sample. Using this method I have been able to produce the best bit detection results. +Next, to decode the audio, and make sense of the PCM samples, we apply the Goertzel algorithm. The Goertzel algorithm works in a fixed number of samples, a window. The size of the Goertzel window greatly changes the magnitude of the output. After many attempts and tests made, a window size of a length of a single encoded bit has been chosen. Using sample rate of 48Khz and a single bit length of 1.25 millisecond, the window size is 60 samples. This window also slides per single sample. Using this method I have been able to produce the best bit detection results.  @@ -260,7 +264,9 @@ To find the start and the end of the encoded byte, we simply look for the empty  -When we find the start and the end of the magnitude sequence, we can test for the length of the samples. If the does not match the expected length, it is discarded. However, not exact amount of samples may be found, while still being a valid sequence of samples. For example, an audio input of the user's machine may record in 47.9Khz due to chap hardware, while the application is configured in 48Khz. To fix this, a tiny margin has been applied during the length verification. When the byte is constructed, it is simply forwarded down the block pipeline for further processing. +When we find the start and the end of the magnitude sequence, we can test for the length of the samples. If the sequence does not match the expected length, it is discarded. However, not exact amount of samples may be found, while still being a valid sequence of samples. For example, an audio input of the user's machine may record in 47.9Khz due to lower-quality hardware, while the application is configured in 48Khz. To fix this, a tiny margin has been applied during the length verification. When the byte is constructed, it is simply forwarded down the block pipeline for further processing. + +The process of detecting wether the bits in the byte are low or high, we simply count the incoming magnitudes. These magnitudes come from a `Merge` block. This block outputs N number of streams. In our case, we are listening to two frequencies, therefore two streams. This stream is a clamped value of the Goertzel algorithm. Because it is clamped, we either get 1 or 0. To check wether we have a high or low bit, we simply divide the entire byte sequence into 8 equal buckets. Within each of the bucket, we count the number of `1` within each of the high or low stream. The higher count wins and we get a high or low bit. However, to encode a large number of data, we need to split the source data into a fixed sized packets. If we transmit the entire large file, the user's receiver may be interrupted, or simply temporarily blocked due to ambient noise, and the entire file lost. Instead, we use packets. Each packet has a simple fixed sized header that identifies the file ID and a packet sequence number. The packet structure then follows by a raw stream of data, in our case a 32 byte chunk, and finally by a single byte representing the 8-bit CRC of the entire packet. @@ -281,20 +287,20 @@ struct Packet { When we receive the packet, the algorithm, more specifically the `FileAssembler` block, will cache the received packets until the final packet has been received. To check whether we have the final packet, the `FileAssembler` block simply searches for the `{ 0xff, 0xff, 0x00, 0x00 }` sequence. This sequence is the footer of the zlib [@zlib] stream. Before the packets are constructed on the Encoder side, the data is compressed through zlib and the footer is appended automatically. We can use this to detect the end of the stream, and therefore the end of the packet sequence. To decompress the data, we simply feed the 32 byte chunks of data into the zlib deflator in the correct order. The CRC of the packet, and the zlib deflating process, should ensure that the decoded data is valid. -However, to mark the start and the end of the packet, we need to include synchronization. To implement this, we need to consider the following scenarios: +However, to mark the start and the end of the packet, we need to include a synchronization. To implement this, we need to consider the following scenarios: * The receiver, the FM radio, on the user's side may be interrupted by a noise or a hardware defect. * The transmission may be blocked by machine's interrupts (the machine, for example a computer, that transmits the data). * The receiver, the FM radio, may start receiving data in the middle of the packet. (The user has plugged in the audio input in the middle of the transmission). * The Decoder may be blocked by the user's machine interrupts or other tasks. -The Decoder is not meant to run on a dedicated hardware, therefore interruption of the audio input may happen at any time. To ensure that we have the beginning of the packet, we introduce a synchronization bytes. These bytes is a sequence of bytes that are transmitted at the start of every packet. The idea comes from NOAA satellites [@noaa]. More precisely, the synchronization of the weather images sent by the satellites [@fmpicture]. +The Decoder is not meant to run on a dedicated hardware, therefore interruption of the audio input may happen at any time. To ensure that we have the beginning of the packet, we introduce the synchronization bytes. These bytes is a sequence of bytes that are transmitted at the start of each packet. The idea comes from NOAA satellites [@noaa]. More precisely, the synchronization of the weather images sent by the satellites [@fmpicture]. ![Example of the transmission sent by the NOAA satellites. The synchronization is seen on the right as a quick transition between black and white pixels. The credits for the image belong to Mike Richards from his blog post "137MHz Polar Orbiting Weather Satellites" [@noaadecoding]](images/noaa.png) In this project, we use the same idea. The beginning of the packet is marked with a synchronization sequence of `{0xff, 0x00, 0xff, 0x00}` bytes. This was later changed to sequence `{0xf0, 0xf0, 0xf0, 0xf0}`. In the NOAA satellites, the synchronization sequence also marks the speed at which the data is transmitted, the baudrate. However due to the time constraints of this project, this was not implemented. Instead, the receiver must know the baudrate beforehand. This is something that could be added in the future. -When the synchronization sequence is detected, the Decoder then starts to buffer fixed number of samples, before forwarding it into the similar algorithm (explained in the section further). This fixed number of samples is calculated based on the baudrate (width of a single bit in number of samples), the sample rate of the audio input (usually 48Khz), and the spacing (the delimiter) between the bytes. When the samples are buffered, they are forwarded into the next stage, which follors the same algorithm with minor modifications, and packets are generated. If the packet is generated, it is verified for its integrity by calculating the CRC and comparing the value with the CRC value appended in the packet. +When the synchronization sequence is detected, the Decoder then starts to buffer fixed number of samples before forwarding it into a similar algorithm (explained in the section further). This fixed number of samples is calculated based on the baudrate (width of a single bit in number of samples), the sample rate of the audio input (usually 48Khz), and the spacing (the delimiter) between the bytes. When the samples are buffered, they are forwarded into the next stage, which follors the same algorithm with some minor modifications, and packets are generated. If the packet is generated, it is verified for its integrity by calculating the CRC and comparing the value with the CRC value appended in the packet.  @@ -306,11 +312,11 @@ The idea behind the algorithm is simple: accept a raw stream of 32bit float PCM This pipeline as shown above proved to be unsuccessful due to a reason that the packet synchronization breaks down. The reason is that the number of data that enters a block may not match the number of data that comes out of it. For example, when we are converting Goertzel decoded output of raw ones and zeros, it is converted into bytes. This is not a 1:1 conversion. Therefore, the track of time is lost. The algorithm can detect the synchronization bytes, however if the data failed to decode, the system will still have partial data that can affect the next packet. For example, it is possible that the Decoder will receive only the first three bytes of the synchronization sequence, the last byte fails to decode, and then the first byte of the file ID happens to be `0xf0`. The Decoder will mark that as the start of the packet, and the next 32+5 bytes are used, whether they form a real packet or a random sequence of mixed data. -This is a problem that we can solve by adding Frame Synchronization (`FrameSync` class) block. This block internally implements the same pipeline as above, however the track of the time is added into the bytes decoded. The track of time is measured in the number of samples and not via the system clock. The reason for this is that the algorithm does not run in a real time environment, unless a dedicated hardware is used. It is possible for the Decoder to be temporarily interrupted by the system process scheduler, therefore the system time would not match, even if we have received the bytes in correct distance apart. The `FrameSync` block remembers the last N bytes. This N matches the number of bytes in the expected synchronization sequence. This sequence must be the same for the Encoder and Decoder, otherwise no packets would be detected. If the received N bytes matches the expected sequence, the `FrameSync` block starts to buffer the next X number of samples. Additionally, the buffered bytes also need to verified for the distance at which the bytes have been received. The distance is the number of samples it takes to encode a single byte. If the distance is higher, this simply means the bytes received are not neighbours, and the buffer is reset. As mentioned previously, the X number (in the internal buffer for forwarding packet samples) is the amount of samples that is expected to hold the entire packet. This is calculated based on the expected width of a single byte, and the delimiter between the bytes. When the buffer is filled, it is then forwarded into the next pipeline (see figure below). If the packet is decoded, it is added into the `FileAssembler`. If no packet is decoded, for example due to noise or transmission errors, the pipeline is reset and no packet is added. When the pipeline is reset, it simply listens for the next synchronization sequence. +This is a problem that we can solve by adding a frame synchronization (`FrameSync` class) block. This block internally implements the same pipeline as above, however the track of the time is added into the bytes decoded. The track of time is measured in the number of samples and not via the system clock. The reason for this is that the algorithm does not run in a real time environment, unless a dedicated hardware is used. It is possible for the Decoder to be temporarily interrupted by the system process scheduler, therefore the system time would not match, even if we have received the bytes in correct distance apart. Additionally, the raw samples from the audio input device may not come at exact intervals. It all depends on the OS process scheduler. The `FrameSync` block remembers the last N bytes. This N matches the number of bytes in the expected synchronization sequence. This sequence must be the same for the Encoder and the Decoder, otherwise no packets would be detected. If the received N bytes matches the expected sequence, the `FrameSync` block starts to buffer the next X number of samples. Additionally, the buffered bytes also need to verified for the distance at which the bytes have been received. The distance is the number of samples it takes to encode a single byte. If the distance is higher, this simply means the bytes received are not neighbours, and the buffer is reset. As mentioned previously, the X number (in the internal buffer for forwarding packet samples) is the amount of samples that is expected to hold the entire packet. This is calculated based on the expected width of a single byte, and the delimiter between the bytes. When the buffer is filled, it is then forwarded into the next pipeline (see figure below). If the packet is decoded, it is added into the `FileAssembler`. If no packet is decoded, for example due to noise or transmission errors, the pipeline is reset and no packet is added. When the pipeline is reset, it simply listens for the next synchronization sequence.  -This pipeline is provided in the dynamic library of the Decoder. However the user, if they wish to, are able to reconstruct this pipeline in their own specific way. The user is also able to further connect their own custom blocks, for example to forward the data into their own software for further processing. This is optional and not the main usage of this algorithm. The main usage is simply to encode or decode websites. The target user of this application does not have programming skills to implement this. This block design is purely an optional feature. +This pipeline is provided in the dynamic library of the Decoder. However, the user, if they wish to, is able to reconstruct this pipeline in their own specific way. The user is also able to further connect their own custom blocks, for example to forward the data into their own software for further processing. This is an optional feature and not the main usage of this algorithm. The main usage is simply to encode or decode a static websites. The target user of this application may not have programming skills to implement this. This block design is purely an optional feature. ### 4.3.6 Command Line Interface @@ -342,13 +348,16 @@ build\Debug\decoder-cli.exe --help Serve the decoded files on local HTTP server via http://localhost:8080 ~~~ -TBA: ASIO and serving HTTP files. +To be added: ASIO and serving HTTP files. ### 4.3.7 Graphical User Interface -TBA: Qt5 and GUI description + screenshots. +To be added: Qt5 and GUI description + screenshots. ## 4.4 Unit Testing +To be added: Unit testing with Catch2 + ## 4.5 Performance Testing +To be added: Performance testing on different hardware and different OS. diff --git a/report/chapters/introduction.md b/report/chapters/introduction.md index e10b99d01382d9c6527dfa7a9cf1b0c81f3d5307..c8cf3cd687a10d72a6fa54ae02c23e0b05f62d4b 100644 --- a/report/chapters/introduction.md +++ b/report/chapters/introduction.md @@ -1 +1,52 @@ -# Introduction +# 1. Introduction + +The aim of this project is to create a fully working GUI application and CLI application to transmit static web pages over FM Radio. This project will additionally consists of a shared library, which can be used the user. Having a shared library is beneficial for the user, if they wish to modify or enhance the Decoder or the Encoder. But, the target end user of this application is not a developer nor an engineer. Thus, a GUI application will be provided. Additionally, a CLI program will be provided alongside the GUI. The CLI can be used on IoT devices such as Raspberry Pi or simply on machines that do not have GUI capabilities. + +Because of the limitations of the carrier (audio), the throughput is limited to few bytes per second. Therefore this project is intended to be used with a simple static website with little to no embedded images. A great example is a Wikipedia article. + +## 1.1 Background + +There have been many natural disasters around the world, such as the hurricanes Maria and Irma in Puerto Rico in 2017. People have been left without electricity for months. Because of this, the information infrastructure collapsed, people had no access to the internet or media. The idea of this project is to provide an internet access in times of similar crisis. + +This project, Radio Enabled Web, can use already existing hardware, the FM radio, to receive the static website. The FM Radio is an old technology and easily accessible and cheap. On the other had, people do not have FM transmitter laying around in their houses. However, it is possible to construct low power FM transmitter from around 20 passive components and few transistors. The FM transmitter circuit can be found in some beginner electronics DIY books. Therefore, the idea is that it only needs one person to create the FM transmitter, with little to no electronics skills. Additionally, because FM radios are common domestic appliance, the users will only require the application and a computer. + +This project can be used to distribute a static website to the public, when the infrastructure collapses. The contents of the websites can provide a basic guidelines for the public, such as a map of refugee camps or specific instructions given to the citizens from the government. + +## 1.2 Motivation + +The idea of this project came to me when I was working with my brother on a hobby project with Software Defined Radio (SDR). We were interested in receiving raw weather images from a NOAA [@noaa] satellite orbiting around the Earth. The satellite constantly transmits the data, and anyone who listens with the proper hardware can receive these images. I was also interested in decentralized internet, an internet created by the users, that can survive a natural disaster. Thus, the FM radio internet idea was born. The initial idea was to have a specific + + +Additionally, I wanted to create something unique that can benefit the public. + +## 1.3 Aims & Objectives + +This project aims to develop a GUI and CLI applications to encode or decode an audio into a static website. The following criteria are: + +* Design a block based algorithm that can be structured into any kind of pipeline. This pipeline will be used to decode or encode the static website into an audio. +* Create a shared (dynamic) library that consists of pre-build set of blocks and the algorithm itself. Leave the implementation open so that it can be adjusted if needed to. +* Create continuous integration scripts so that the application and the library are tested for every new added feature. The CI also serves as a proof that the application is portable across different platforms. +* Create a CLI that uses the shared library. The CLI can be used on non GUI enabled computers to transmit or decode the data. +* Create a GUI that a user can use to encode or decode the static website. +* Create a HTTP server that is embedded into both the CLI and GUI applications so that the user can browse the encoded website with a web browser. +* Perform tests with real FM hardware and modify the implementation if needed to. + +## 1.4 Success Criteria + +The following are success criteria needed for this project to be marked as success: + +* A working implementation of the encoding and decoding algorithm. +* Ability to decode the data from noisy transmission. A noise filter must be implemented. +* A working Command Line Interface. +* A working Graphical User Interface. +* Cross platform support for Windows, Linux, and Mac OSX. + +## 1.5 Report Structure + +* *1. Introduction* - This section provides a brief introduction and highlights the aims and objectives of this project. +* *2. Literature Review* - This section provides an investigation into already existing or similar technologies. Additionally, relevant findings are highlighted. +* *3. System Requirements* - This section includes a set of features of the system. This section also provides software and hardware specifications. +* *4. System Design and Implementation* - The main part of this report, which consists of detailed description of the algorithm implementation and C++ project structure as a whole. This section is also dedicated to a usability of continuous integration. +* *5. System Testing* - This entire section is dedicated to testing of the algorithm via FM transmitter and receiver. The section is also dedicated to performance testing on a number of different hardware. +* *6. Conclusion and Future Work* - This section highlights the success of this project. Additionally, any changes or improvements that have not been included in the final implementation are described in here. + diff --git a/report/chapters/literature.md b/report/chapters/literature.md index 7984f4fd1d25fb0943de5e1d94af5c384aa2e2f6..66131b07392995e5427bad96941f776ac6c81154 100644 --- a/report/chapters/literature.md +++ b/report/chapters/literature.md @@ -1 +1,8 @@ -# Literature Review +# 2. Literature Review + +This chapter will discuss any relevant technologies and relevant encoding or decoding strategies explained. + +## 2.1 Modulation + + + diff --git a/report/chapters/originality.md b/report/chapters/originality.md new file mode 100644 index 0000000000000000000000000000000000000000..fbcd497bb3f5de5e0cb5f91410477d6e36238591 --- /dev/null +++ b/report/chapters/originality.md @@ -0,0 +1,3 @@ +# Declaration of Originality + +I confirm that the submitted work is my own work and that I have clearly identified and fully acknowledged all material that is entitled to be attributed to others (whether published or unpublished) using the referencing system set out in the programme handbook. I agree that the University may submit my work to means of checking this, such as the plagiarism detection service Turnitin® UK. I confirm that I understand that assessed work that has been shown to have been plagiarised will be penalised. diff --git a/report/chapters/requirements.md b/report/chapters/requirements.md index 104947390e056f674bf890c1150a4c309f1c7d2e..069c09f8cb2f628e79d6d43b5c58950c5115a5d5 100644 --- a/report/chapters/requirements.md +++ b/report/chapters/requirements.md @@ -1 +1,15 @@ -# System Requirements +# 3. System Requirements + +## 3.1 Functional Requirements + +## 3.2 Non-Functional Requirements + +## 3.3 Software and Hardware Specifications + +## 3.4 Technical Feasibility + +### 3.4.1 Legal Feasibility + +### 3.4.2 operational Feasibility + +### 3.4.3 Economical Feasibility diff --git a/report/chapters/research.md b/report/chapters/research.md deleted file mode 100644 index 15d2a044c2a5db914f1d25401b8abade7a83fde1..0000000000000000000000000000000000000000 --- a/report/chapters/research.md +++ /dev/null @@ -1 +0,0 @@ -# Research topic in theory diff --git a/report/chapters/specifications.md b/report/chapters/specifications.md deleted file mode 100644 index 247b1857d5668f869be0d84a0ba75307d38ebffb..0000000000000000000000000000000000000000 --- a/report/chapters/specifications.md +++ /dev/null @@ -1 +0,0 @@ -# Specifications diff --git a/report/chapters/testing.md b/report/chapters/testing.md index f00b526a9875b0f8bec9fa0c97eeed75379a6ddd..10da05e33911316db8699dac03fabda264b633aa 100644 --- a/report/chapters/testing.md +++ b/report/chapters/testing.md @@ -1 +1,3 @@ -# Testing +# 5. System Testing + +To be added: Testing on a real FM transmitter and receiver hardware. diff --git a/report/images/BinTray.png b/report/images/BinTray.png new file mode 100644 index 0000000000000000000000000000000000000000..39682a5b5493d790e595f79fb9de26b833275a04 Binary files /dev/null and b/report/images/BinTray.png differ diff --git a/report/index.mdpp b/report/index.mdpp index cefeba2c4a9623409112dc84088e02e7cebc3b92..e058c24e202d7e620038946baa4aacafcd05e904 100644 --- a/report/index.mdpp +++ b/report/index.mdpp @@ -5,22 +5,22 @@ date: January 1 1970 geometry: "left=3cm,right=3cm,top=2cm,bottom=2cm" --- +!INCLUDE "chapters/originality.md" + !INCLUDE "chapters/abstract.md" !INCLUDE "chapters/acknowledgement.md" -!INCLUDE "chapters/acronyms.md" +!INCLUDE "chapters/abbreviations.md" -!INCLUDE "chapters/introduction.md" +\tableofcontents -!INCLUDE "chapters/research.md" +!INCLUDE "chapters/introduction.md" !INCLUDE "chapters/literature.md" !INCLUDE "chapters/requirements.md" -!INCLUDE "chapters/specifications.md" - !INCLUDE "chapters/design.md" !INCLUDE "chapters/testing.md" diff --git a/report/make.bat b/report/make.bat index 30d085890e47cb9e1739787a98c22b61ae3aef01..168fd3bba7f624e05d21dfe1d1e9f844101d0f97 100644 --- a/report/make.bat +++ b/report/make.bat @@ -1,3 +1,3 @@ mkdir build markdown-pp index.mdpp -o build/report.md -pandoc --filter pandoc-citeproc --bibliography=bibliography.bib --variable papersize=a4paper --csl=ieee.csl --toc --listings --include-in-header mixins/fix-captions.tex --include-in-header mixins/titlesec.tex --include-in-header mixins/table.tex --include-in-header mixins/listings.tex -o build/report.pdf build/report.md +pandoc --filter pandoc-citeproc --bibliography=bibliography.bib --variable papersize=a4paper --csl=ieee.csl --listings --include-in-header mixins/fix-captions.tex --include-in-header mixins/titlesec.tex --include-in-header mixins/table.tex --include-in-header mixins/listings.tex -o build/report.pdf build/report.md