Kamil Jeziorek

2026

1. TRETS

   Hardware-accelerated graph neural networks: an alternative approach for event-based audio classification and keyword spotting on SoC FPGA

   Kamil Jeziorek, Piotr Wzorek, Krzysztof Blachut, Hiroshi Nakano, Manon Dampfhoffer, and 4 more authors

   ACM Transactions on Reconfigurable Technology and Systems, 2026

   Abstract DOI Bib

   As the volume of data recorded by embedded edge sensors increases, particularly from neuromorphic devices producing discrete event streams, there is a need for hardware-aware neural architectures that enable efficient, low-latency and energy-conscious local processing. To address this research gap, we present an FPGA implementation of event-graph neural networks for audio processing. We utilise an artificial cochlea that converts time-series signals into sparse event data, reducing memory and computation costs. Our architecture was implemented on an SoC FPGA and evaluated on two open source datasets. For the classification task, our baseline floating-point model achieves 92.7% accuracy on the SHD dataset—only 2.4% and 2% below the state-of-the-art—while requiring 10 and 67 fewer parameters, respectively. Compared to FPGA-based spiking neural networks, our quantised model reaches 92.3% accuracy, outperforming them by 19.3 pps while reducing resource usage and latency. For the SSC dataset, we report the first hardware-accelerated evaluation, and our models achieve 66.9–71.0% accuracy. Extending our conference work, we further demonstrate the first end-to-end FPGA implementation of the event-audio keyword spotting task, combining graph convolutional layers with recurrent sequence modelling for continuous, streaming operation. The system achieves 95% word-end detection accuracy, with only 10.53 µs latency and 1.18 W power consumption, establishing a strong benchmark for energy-efficient event-driven KWS.

   @article{badap-agh:170304,
     title = {Hardware-accelerated graph neural networks: an alternative approach for event-based audio classification and keyword spotting on SoC FPGA},
     author = {Jeziorek, Kamil and Wzorek, Piotr and Blachut, Krzysztof and Nakano, Hiroshi and Dampfhoffer, Manon and Mesquida, Thomas and Nishi, Hiroaki and Dalgaty, Thomas and Kryjak, Tomasz},
     year = {2026},
     journal = {ACM Transactions on Reconfigurable Technology and Systems},
     volume = {19},
     pages = {45},
     doi = {10.1145/3838800},
   }

2. KU KDM ARTIQ
   Preludium
   

   Towards event-based in-cabin sensing: a synchronized dual-camera dataset pipeline

   Marcin Kowalczyk, Marek Dlugosz, Kamil Jeziorek, Mateusz Komorkiewicz, Tomasz Kryjak, and 4 more authors

   In KU KDM 2026 : eighteenth ACC Cyfronet AGH HPC users conference : Zakopane, 13–15 April 2026 : proceedings, 2026

   Bib

   @inproceedings{badap-agh:167206,
     title = {Towards event-based in-cabin sensing: a synchronized dual-camera dataset pipeline},
     author = {Kowalczyk, Marcin and Dlugosz, Marek and Jeziorek, Kamil and Komorkiewicz, Mateusz and Kryjak, Tomasz and Marchewka, Dariusz and Szelest, Marcin and Wzorek, Piotr and Skruch, Pawel},
     year = {2026},
     series = {KU KDM~2026 : eighteenth ACC Cyfronet AGH HPC users conference : Zakopane, 13–15 April~2026 : proceedings},
     pages = {41–42},
     collection = {KU KDM~2026 : eighteenth ACC Cyfronet AGH HPC users conference : Zakopane, 13–15 April~2026 : proceedings},
     projects = {ARTIQ, Preludium-Kowalczyk},
   }

3. JSA Preludium
   

   Embedded graph convolutional networks for real-time event data processing on SoC FPGAs

   Kamil Jeziorek, Piotr Wzorek, Krzysztof Blachut, Andrea Pinna, and Tomasz Kryjak

   Journal of Systems Architecture, 2026

   Abstract Collaboration DOI arXiv Bib

   • SYEL (Systèmes Électroniques), LIP6 laboratory
     Sorbonne Université, France

   All our collaborations

   The utilisation of event cameras represents an important and swiftly evolving trend aimed at addressing the constraints of traditional video systems. Particularly within the automotive domain, these cameras find significant relevance for their integration into embedded real-time systems due to lower latency and power consumption. One effective approach to ensure the necessary throughput and latency for event processing is through the utilisation of graph convolutional networks (GCNs). In this study, we introduce a custom EFGCN (Event-based FPGA-accelerated Graph Convolutional Network) designed with a series of hardware-aware optimisations tailored for PointNetConv, a graph convolution designed for point cloud processing. The proposed techniques result in up to 100-fold reduction in model size compared to Asynchronous Event-based GNN (AEGNN), one of the most recent works in the field, with a relatively small decrease in accuracy (2.9% for the N-Caltech101 classification task, 2.2% for the N-Cars classification task), thus following the TinyML trend. We implemented EFGCN on a ZCU104 SoC FPGA platform without any off-chip external memory resources, achieving a throughput of 13.3 million events per second (MEPS) and real-time partially asynchronous processing with low latency. Across multiple event-based classification benchmarks, our approach achieves competitive accuracy while providing state-of-the-art computational efficiency per event, small model size, and high scalability, customisability and resource efficiency. We publish both software and hardware source code in an open repository: https://github.com/vision-agh/gcnn-dvs-fpga .

   @article{badap-agh:168430,
     title = {Embedded graph convolutional networks for real-time event data processing on SoC FPGAs},
     author = {Jeziorek, Kamil and Wzorek, Piotr and Blachut, Krzysztof and Pinna, Andrea and Kryjak, Tomasz},
     year = {2026},
     journal = {Journal of Systems Architecture},
     series = {Journal of Systems Architecture},
     volume = {177},
     pages = {103850},
     doi = {10.1016/j.sysarc.2026.103850},
     issn = {1383-7621},
     collection = {Journal of Systems Architecture},
     projects = {Preludium-Wzorek},
   }

4. ARC Preludium
   

   End-to-end keyword spotting on FPGA using graph neural networks with a neuromorphic auditory sensor

   Wiktor Matykiewicz, Piotr Wzorek, Kamil Jeziorek, Tomás Muñoz, Antonio Rios-Navarro, and 2 more authors

   In Applied Reconfigurable Computing : architectures, tools, and applications : 22nd international symposium, ARC 2026 : Cagliari, Sardinia, Italy, April 8–10, 2026 : proceedings, 2026

   Abstract Collaboration DOI arXiv Bib

   • Robotics and Technology of Computers Lab (RTC)
     Universidad de Sevilla, Spain

   All our collaborations

   With the rapid growth of mobile robotics and embedded intelligence, there is an increasing demand for efficient on-device data processing on edge platforms. A promising research direction is the use of neuromorphic sensors inspired by human sensory systems, which generate sparse, event-based data encoding changes in the environment. In this work, we present the first end-to-end FPGA implementation of a keyword spotting system that integrates a Neuromorphic Auditory Sensor (NAS) and a graph neural network (GNN) on a single FPGA device, enabling real-time processing of raw audio data. The proposed architecture eliminates conventional signal preprocessing and operates directly on event-based audio streams. Leveraging a compute-near-memory network architecture, the system achieves efficient inference with low latency and low power consumption. Experimental results demonstrate an accuracy of 87.43% after quantization on the Google Speech Commands v2 dataset processed through the neuromorphic sensor, with end-to-end latency below 35 us and average power consumption of 1.12 W. The processed datasets, software models, and hardware modules are available at https://github.com/vision-agh/NAS-GNN-KWS.

   @inproceedings{badap-agh:168992,
     title = {End-to-end keyword spotting on FPGA using graph neural networks with a neuromorphic auditory sensor},
     author = {Matykiewicz, Wiktor and Wzorek, Piotr and Jeziorek, Kamil and Mu\~{n}oz, Tom\'{a}s and Rios-Navarro, Antonio and Jim\'{e}nez-Fern\'{a}ndez, Angel and Kryjak, Tomasz},
     year = {2026},
     series = {Applied Reconfigurable Computing : architectures, tools, and applications : 22nd international symposium, ARC~2026 : Cagliari, Sardinia, Italy, April 8–10,~2026 : proceedings},
     pages = {119–136},
     doi = {10.1007/978-3-032-29365-7_8},
     collection = {Applied Reconfigurable Computing : architectures, tools, and applications : 22nd international symposium, ARC~2026 : Cagliari, Sardinia, Italy, April 8–10,~2026 : proceedings},
     projects = {Preludium-Wzorek},
   }

2025

1. KU KDM Preludium
   

   High-performance computing for event cameras: DIF filtering and graph convolutional networks for object classification

   Marcin Kowalczyk, Kamil Jeziorek, Tomasz Kryjak, and Marek Gorgon

   In KU KDM 2025 : seventeenth ACC Cyfronet AGH HPC users’ conference : Zakopane, 2-4 April 2025 : proceedings, 2025

   Bib

   @inproceedings{badap-agh:159338,
     title = {High-performance computing for event cameras: DIF filtering and graph convolutional networks for object classification},
     author = {Kowalczyk, Marcin and Jeziorek, Kamil and Kryjak, Tomasz and Gorgon, Marek},
     year = {2025},
     projects = {Preludium-Kowalczyk},
     series = {KU KDM~2025 [Dokument elektroniczny] : seventeenth ACC Cyfronet AGH HPC users' conference : Zakopane, 2-4 April~2025 : proceedings},
     pages = {49-50},
     collection = {KU KDM~2025 [Dokument elektroniczny] : seventeenth ACC Cyfronet AGH HPC users' conference : Zakopane, 2-4 April~2025 : proceedings},
   }

2. ARC

   Hardware-accelerated event-graph neural networks for low-latency time-series classification on SoC FPGA

   Hiroshi Nakano, Krzysztof Blachut, Kamil Jeziorek, Piotr Wzorek, Manon Dampfhoffer, and 4 more authors

   In Applied Reconfigurable Computing : architectures, tools, and applications : 21st international symposium, ARC 2025 : Seville, Spain, April 9–11, 2025 : proceedings, 2025

   Abstract Collaboration DOI arXiv Bib

   • Digital Systems and Integrated Circuits Division
     CEA-List Grenoble, France
   • Faculty of Science and Technology, Department of System Design Engineering
     Keio University, Japan

   All our collaborations

   As the quantities of data recorded by embedded edge sensors grow, so too does the need for intelligent local processing. Such data often comes in the form of time-series signals, based on which real-time predictions can be made locally using an AI model. However, a hardware-software approach capable of making low-latency predictions with low power consumption is required. In this paper, we present a hardware implementation of an event-graph neural network for time-series classification. We leverage an artificial cochlea model to convert the input time-series signals into a sparse event-data format that allows the event-graph to drastically reduce the number of calculations relative to other AI methods. We implemented the design on a SoC FPGA and applied it to the real-time processing of the Spiking Heidelberg Digits (SHD) dataset to benchmark our approach against competitive solutions. Our method achieves a floating-point accuracy of 92.7% on the SHD dataset for the base model, which is only 2.4% and 2% less than the state-of-the-art models with over 10% and 67% fewer model parameters, respectively. It also outperforms FPGA-based spiking neural network implementations by 19.3% and 4.5%, achieving 92.3% accuracy for the quantised model while using fewer computational resources and reducing latency.

   @inproceedings{badap-agh:160129,
     title = {Hardware-accelerated event-graph neural networks for low-latency time-series classification on SoC FPGA},
     author = {Nakano, Hiroshi and Blachut, Krzysztof and Jeziorek, Kamil and Wzorek, Piotr and Dampfhoffer, Manon and Mesquida, Thomas and Nishi, Hiroaki and Kryjak, Tomasz and Dalgaty, Thomas},
     year = {2025},
     series = {Applied Reconfigurable Computing : architectures, tools, and applications : 21st international symposium, ARC~2025 : Seville, Spain, April 9–11,~2025 : proceedings},
     pages = {51–68},
     doi = {10.1007/978-3-031-87995-1_4},
     collection = {Applied Reconfigurable Computing : architectures, tools, and applications : 21st international symposium, ARC~2025 : Seville, Spain, April 9–11,~2025 : proceedings},
   }

3. SPIE RTIPDVI

   Self-supervised event representations: towards accurate, real-time perception on SoC FPGAs

   Kamil Jeziorek and Tomasz Kryjak

   In Real-time processing of image, depth, and video information 2025 : 7–10 April 2025, Prague, Czech Republic, 2025

   Abstract DOI arXiv Bib

   Event cameras offer significant advantages over traditional frame-based sensors. These include microsecond temporal resolution, robustness under varying lighting conditions and low power consumption. Nevertheless, the effective processing of their sparse, asynchronous event streams remains challenging. Existing approaches to this problem can be categorised into two distinct groups. The first group involves the direct processing of event data with neural models, such as Spiking Neural Networks or Graph Convolutional Neural Networks. However, this approach is often accompanied by a compromise in terms of qualitative performance. The second group involves the conversion of events into dense representations with handcrafted aggregation functions, which can boost accuracy at the cost of temporal fidelity. This paper introduces a novel Self-Supervised Event Representation (SSER) method leveraging Gated Recurrent Unit (GRU) networks to achieve precise per-pixel encoding of event timestamps and polarities without temporal discretisation. The recurrent layers are trained in a self-supervised manner to maximise the fidelity of event-time encoding. The inference is performed with event representations generated asynchronously, thus ensuring compatibility with high-throughput sensors. The experimental validation demonstrates that SSER outperforms aggregation-based baselines, achieving improvements of 2.4% mAP and 0.6% on the Gen1 and 1 Mpx object detection datasets. Furthermore, the paper presents the first hardware implementation of recurrent representation for event data on a System-on-Chip FPGA, achieving submicrosecond latency and power consumption between 1–2 W, suitable for real-time, power-efficient applications. Code is available at https://github.com/vision-agh/RecRepEvent.

   @inproceedings{badap-agh:160301,
     title = {Self-supervised event representations: towards accurate, real-time perception on SoC FPGAs},
     author = {Jeziorek, Kamil and Kryjak, Tomasz},
     year = {2025},
     series = {Real-time processing of image, depth, and video information~2025 : 7–10 April~2025, Prague, Czech Republic},
     pages = {1352602-1–1352602–14},
     doi = {10.1117/12.3056056},
     collection = {Real-time processing of image, depth, and video information~2025 : 7–10 April~2025, Prague, Czech Republic},
   }

4. CVPRW Preludium
   

   Learning from noise: enhancing DNNs for event-based vision through controlled noise injection

   Marcin Kowalczyk, Kamil Jeziorek, and Tomasz Kryjak

   In CVPRW 2025 : IEEE/CVF Computer Vision and Pattern Recognition Workshops : 11–15 June 2025, Nashville, [USA] : proceedings, 2025

   Abstract DOI arXiv Bib

   Event-based sensors offer significant advantages over traditional frame-based cameras, especially in scenarios involving rapid motion or challenging lighting conditions. However, event data frequently suffers from considerable noise, negatively impacting the performance and robustness of deep learning models. Traditionally, this problem has been addressed by applying filtering algorithms to the event stream, but this may also remove some of relevant data. In this paper, we propose a novel noise-injection training methodology designed to enhance the neural networks robustness against varying levels of event noise. Our approach introduces controlled noise directly into the training data, enabling models to learn noise-resilient representations. We have conducted extensive evaluations of the proposed method using multiple benchmark datasets (N-Caltech101, N-Cars, and Mini N-ImageNet) and various network architectures, including Convolutional Neural Networks, Vision Transformers, Spiking Neural Networks, and Graph Convolutional Networks. Experimental results show that our noise-injection training strategy achieves stable performance over a range of noise intensities, consistently outperforms event-filtering techniques, and achieves the highest average classification accuracy, making it a viable alternative to traditional event-data filtering methods in an object classification system. Code: https://github.com/vision-agh/DVS_Filtering

   @inproceedings{badap-agh:162623,
     title = {Learning from noise: enhancing DNNs for event-based vision through controlled noise injection},
     author = {Kowalczyk, Marcin and Jeziorek, Kamil and Kryjak, Tomasz},
     year = {2025},
     projects = {Preludium-Kowalczyk},
     series = {CVPRW~2025 [Dokument elektroniczny] : IEEE/CVF Computer Vision and Pattern Recognition Workshops : 11–15 June~2025, Nashville, [USA] : proceedings},
     pages = {5092–5102},
     doi = {10.1109/CVPRW67362.2025.00504},
     collection = {CVPRW~2025 [Dokument elektroniczny] : IEEE/CVF Computer Vision and Pattern Recognition Workshops : 11–15 June~2025, Nashville, [USA] : proceedings},
   }

5. CVPRW Preludium
   

   Live demonstration: real-time event-data processing with Graph Convolutional Neural Networks and SoC FPGA

   Piotr Wzorek, Krzysztof Blachut, Kamil Jeziorek, and Tomasz Kryjak

   In CVPRW 2025 : IEEE/CVF Computer Vision and Pattern Recognition Workshops : 11–15 June 2025, Nashville, [USA] : proceedings, 2025

   Abstract DOI Bib

   With growing demand for robust low-power and lowlatency computer vision, event cameras are becoming increasingly common in mobile robotics and autonomous vehicles due to their high temporal resolution, ability to operate in adverse lightning conditions and reduction of redundant information. However, fully exploiting the benefits of these sensors requires algorithms capable of processing sparse data efficiently and optimised for a specialised target platform. To address this demand, we present a hardware-aware model of Graph Convolutional Neural Networks (GCNNs) implemented on an embedded heterogeneous SoC FPGA. Leveraging the temporal sparsity of event data, our approach integrates partially asynchronous processing and 3D MaxPool layers to achieve real-time, high-accuracy object classification on embedded platforms. A key strength of our architecture is its scalability – customisable hardware modules enable flexible configurations to balance resource utilisation and latency. In this realtime demonstration we will showcase the capabilities of our GCNN model, address its limitations, and discuss future development directions. We will use the system operating on a heterogeneous SoC FPGA to classify event data captured directly by the integrated camera or read from an SD card with simulated time intervals between events. Our work has been shared in a open-source repository: https://github.com/vision-agh/gcnn-dvs-fpga

   @inproceedings{badap-agh:162625,
     title = {Live demonstration: real-time event-data processing with Graph Convolutional Neural Networks and SoC FPGA},
     author = {Wzorek, Piotr and Blachut, Krzysztof and Jeziorek, Kamil and Kryjak, Tomasz},
     year = {2025},
     series = {CVPRW~2025 [Dokument elektroniczny] : IEEE/CVF Computer Vision and Pattern Recognition Workshops : 11–15 June~2025, Nashville, [USA] : proceedings},
     pages = {5103–5104},
     doi = {10.1109/CVPRW67362.2025.00505},
     collection = {CVPRW~2025 [Dokument elektroniczny] : IEEE/CVF Computer Vision and Pattern Recognition Workshops : 11–15 June~2025, Nashville, [USA] : proceedings},
     projects = {Preludium-Wzorek},
   }

6. SPA Preludium
   

   Hardware-aware Graph Neural Networks prunning for embedded event-based vision

   Piotr Wzorek, Kamil Jeziorek, and Tomasz Kryjak

   In SPA 2025 : Signal Processing - algorithms, architectures, arrangements, and applications : Poznan, 17th–19th September 2025 : conference proceedings, 2025

   Abstract DOI arXiv Bib

   Event-based cameras are gaining popularity as the sensor of choice for mobile robotics, due to their high performance in dynamic environments. However, these applications require efficient real-time data processing with low latency and power consumption. One strategy to meet these stringent requirements is hardware acceleration of efficient algorithms that preserve the temporal sparsity of event data.In this work, we propose an optimization strategy for Graph Convolutional Neural Networks models aimed at adapting their architecture to the limited resources of embedded heterogeneous FPGA platforms. Our method incorporates hardware-aware pruning and quantization, taking into account the trade-off between on-chip memory savings and inference accuracy. Strategic exploration of the design space with Fine Grid Search and Greedy layer-wise Iterative Deepening Search methods enables flexible adaptation of the model architecture to the target platform. Our approach was evaluated across various network configurations and multiple datasets, resulting in BRAM memory reductions of 28.8% for CIFAR-10 (with a 1.65% decrease in accuracy), 31.4% for MNIST-DVS (accuracy drop of 3.55%), and 26.5% for N-Caltech101 (with a 5.18% accuracy reduction).

   @inproceedings{badap-agh:163126,
     title = {Hardware-aware Graph Neural Networks prunning for embedded event-based vision},
     author = {Wzorek, Piotr and Jeziorek, Kamil and Kryjak, Tomasz},
     year = {2025},
     series = {SPA~2025 [Dokument elektroniczny] : Signal Processing - algorithms, architectures, arrangements, and applications : Poznan, 17th–19th September~2025 : conference proceedings},
     pages = {182–187},
     doi = {10.23919/SPA65537.2025.11215154},
     collection = {SPA~2025 [Dokument elektroniczny] : Signal Processing - algorithms, architectures, arrangements, and applications : Poznan, 17th–19th September~2025 : conference proceedings},
     projects = {Preludium-Wzorek},
   }

7. SPA

   A model-driven TTFS translation for ReLU neural networks

   Michal Michniak, Kamil Jeziorek, and Tomasz Kryjak

   In SPA 2025 : Signal Processing - algorithms, architectures, arrangements, and applications : Poznan, 17th–19th September 2025 : conference proceedings, 2025

   Abstract DOI Bib

   Spiking neural networks (SNNs) offer a promising avenue for energy-efficient and biologically inspired computation, with the potential to run effectively on neuromorphic hardware. Recent advances have focused on translating conventional artificial neural networks (ANNs) into their spiking counterparts, particularly using time-to-first-spike (TTFS) coding to reduce the number of spikes and improve efficiency. However, most existing TTFS translation methods rely heavily on data-driven threshold estimation and multi-phase operations. In this work, we present a novel, fully model-driven TTFS translation approach for ReLU-PSP (Rectified Linear Postsynaptic Potential function) networks, eliminating the need for training data and employing a single, fixed activation threshold. We introduce a structural upper bound based on the ReLU1 function to control numerical stability and enable accurate inference, even with complex architectures containing residual connections. Our method is evaluated on multiple event-based datasets, demonstrating equivalent classification accuracy to conventional ANNs.

   @inproceedings{badap-agh:163127,
     title = {A model-driven TTFS translation for ReLU neural networks},
     author = {Michniak, Michal and Jeziorek, Kamil and Kryjak, Tomasz},
     year = {2025},
     series = {SPA~2025 [Dokument elektroniczny] : Signal Processing - algorithms, architectures, arrangements, and applications : Poznan, 17th–19th September~2025 : conference proceedings},
     pages = {188–193},
     doi = {10.23919/SPA65537.2025.11215160},
     collection = {SPA~2025 [Dokument elektroniczny] : Signal Processing - algorithms, architectures, arrangements, and applications : Poznan, 17th–19th September~2025 : conference proceedings},
   }

8. FPL Preludium
   Preludium
   

   Live demonstration: Continuous processing of event-data with Graph Convolutional Neural Networks implemented for SoC FPGA

   Piotr Wzorek, Kamil Jeziorek, Marcin Kowalczyk, Krzysztof Blachut, Tomasz Kryjak, and 1 more author

   In FPL 2025 : 35th International Conference on Field-Programmable Logic and Applications : 1–5 September 2025 Leiden, Netherlands : proceedings, 2025

   Abstract DOI Bib

   For perception systems in mobile robotics - particularly in demanding, highly dynamic environments - event cameras (DVS - Dynamic Vision Sensors) are being increasingly utilised as an alternative to conventional vision sensors. The real-time processing of the registered spatio-temporal sparse point cloud must satisfy stringent requirements in terms of latency, throughput, and energy efficiency. In this demonstration, we present our method for implementing Graph Convolutional Neural Networks on a heterogeneous SoC FPGA platform, aimed at meeting these constraints. We address the challenges associated with integrating event-based sensors with reconfigurable hardware, as well as the intricate relationship between latency, performance, and hardware resource utilisation.

   @inproceedings{badap-agh:166922,
     title = {Live demonstration: Continuous processing of event-data with Graph Convolutional Neural Networks implemented for SoC FPGA},
     author = {Wzorek, Piotr and Jeziorek, Kamil and Kowalczyk, Marcin and Blachut, Krzysztof and Kryjak, Tomasz and Gorgon, Marek},
     year = {2025},
     series = {FPL~2025 [Dokument elektroniczny] : 35th International Conference on Field-Programmable Logic and Applications : 1–5 September~2025 Leiden, Netherlands : proceedings},
     pages = {359},
     doi = {10.1109/FPL68686.2025.00065},
     collection = {FPL~2025 [Dokument elektroniczny] : 35th International Conference on Field-Programmable Logic and Applications : 1–5 September~2025 Leiden, Netherlands : proceedings},
     projects = {Preludium-Wzorek, Preludium-Kowalczyk},
   }

2024

1. FPT

   Increasing the scalability of graph convolution for FPGA-implemented event-based vision

   Piotr Wzorek, Kamil Jeziorek, Tomasz Kryjak, and Andrea Pinna

   In ICFPT 2024 : International Conference on Field Programmable Technology 2024 : Sydney, Australia, 10-12 December 2024 : proceedings, 2024

   Abstract Collaboration DOI arXiv Bib

   • SYEL (Systèmes Électroniques), LIP6 laboratory
     Sorbonne Université, France

   All our collaborations

   Event cameras are becoming increasingly popular as an alternative to traditional frame-based vision sensors, especially in mobile robotics. Taking full advantage of their high temporal resolution, high dynamic range, low power consumption and sparsity of event data, which only reflects changes in the observed scene, requires both an efficient algorithm and a specialised hardware platform. A recent trend involves using Graph Convolutional Neural Networks (GCNNs) implemented on a heterogeneous SoC FPGA. In this paper we focus on optimising hardware modules for graph convolution to allow flexible selection of the FPGA resource (BlockRAM, DSP and LUT) for their implementation. We propose a “two-step convolution” approach that utilises additional BRAM buffers in order to reduce up to 94% of LUT usage for multiplications. This method significantly improves the scalability of GCNNs, enabling the deployment of models with more layers, larger graphs sizes and their application for more dynamic scenarios.

   @inproceedings{badap-agh:162122,
     title = {Increasing the scalability of graph convolution for FPGA-implemented event-based vision},
     author = {Wzorek, Piotr and Jeziorek, Kamil and Kryjak, Tomasz and Pinna, Andrea},
     year = {2024},
     series = {ICFPT~2024 [Dokument elektroniczny] : International Conference on Field Programmable Technology~2024 : Sydney, Australia, 10-12 December~2024 : proceedings},
     pages = {[1-4]},
     doi = {10.1109/ICFPT64416.2024.11113443},
     collection = {ICFPT~2024 [Dokument elektroniczny] : International Conference on Field Programmable Technology~2024 : Sydney, Australia, 10-12 December~2024 : proceedings},
   }

2. DASIP

   Optimising graph representation for hardware implementation of graph convolutional networks for event-based vision

   Kamil Jeziorek, Piotr Wzorek, Krzysztof Blachut, Andrea Pinna, and Tomasz Kryjak

   In Design and Architectures for Signal and Image Processing : 17th international workshop, DASIP 2024 : Munich, Germany, January 17–19, 2024 : proceedings, 2024

   Abstract Collaboration DOI arXiv Bib

   • SYEL (Systèmes Électroniques), LIP6 laboratory
     Sorbonne Université, France

   All our collaborations

   Event-based vision is an emerging research field involving processing data generated by Dynamic Vision Sensors (neuromorphic cameras). One of the latest proposals in this area are Graph Convolutional Networks (GCNs), which allow to process events in its original sparse form while maintaining high detection and classification performance. In this paper, we present the hardware implementation of a graph generation process from an event camera data stream, taking into account both the advantages and limitations of FPGAs. We propose various ways to simplify the graph representation and use scaling and quantisation of values. We consider both undirected and directed graphs that enable the use of PointNet convolution. The results obtained show that by appropriately modifying the graph representation, it is possible to create a hardware module for graph generation. Moreover, the proposed modifications have no significant impact on object detection performance, only 0.08% mAP less for the base model and the N-Caltech data set.Finally, we describe the proposed hardware architecture of the graph generation module.

   @inproceedings{badap-agh:154027,
     title = {Optimising graph representation for hardware implementation of graph convolutional networks for event-based vision},
     author = {Jeziorek, Kamil and Wzorek, Piotr and Blachut, Krzysztof and Pinna, Andrea and Kryjak, Tomasz},
     year = {2024},
     series = {Design and Architectures for Signal and Image Processing : 17th international workshop, DASIP~2024 : Munich, Germany, January 17–19, 2024 : proceedings},
     pages = {110–122},
     doi = {10.1007/978-3-031-62874-0_9},
     collection = {Design and Architectures for Signal and Image Processing : 17th international workshop, DASIP~2024 : Munich, Germany, January 17–19, 2024 : proceedings},
   }

2023

1. SPA

   Memory-efficient graph convolutional networks for object classification and detection with event cameras

   Kamil Jeziorek, Andrea Pinna, and Tomasz Kryjak

   In SPA 2023 : Signal Processing Algorithms, Architectures, Arrangements, and Applications : Poznan, 20th - 22nd September 2023, 2023

   Abstract Collaboration DOI arXiv Bib

   • SYEL (Systèmes Électroniques), LIP6 laboratory
     Sorbonne Université, France

   All our collaborations

   Recent advances in event camera research emphasize processing data in its original sparse form, which allows the use of its unique features such as high temporal resolution, high dynamic range, low latency, and resistance to image blur. One promising approach for analyzing event data is through graph convolutional networks (GCNs). However, current research in this domain primarily focuses on optimizing computational costs, neglecting the associated memory costs. In this paper, we consider both factors together in order to achieve satisfying results and relatively low model complexity. For this purpose, we performed a comparative analysis of different graph convolution operations, considering factors such as execution time, the number of trainable model parameters, data format requirements, and training outcomes. Our results show a 450-fold reduction in the number of parameters for the feature extraction module and a 4.5-fold reduction in the size of the data representation while maintaining a classification accuracy of 52.3%, which is 6.3% higher compared to the operation used in state-of-the-art approaches. To further evaluate performance, we implemented the object detection architecture and evaluated its performance on the N-Caltech101 dataset. The results showed an accuracy of 53.7% mAP@0.5 and reached an execution rate of 82 graphs per second.

   @inproceedings{badap-agh:150128,
     title = {Memory-efficient graph convolutional networks for object classification and detection with event cameras},
     author = {Jeziorek, Kamil and Pinna, Andrea and Kryjak, Tomasz},
     year = {2023},
     series = {SPA~2023 : Signal Processing Algorithms, Architectures, Arrangements, and Applications : Poznan, 20th - 22nd September 2023},
     pages = {160–165},
     doi = {10.23919/SPA59660.2023.10274464},
     collection = {SPA~2023 : Signal Processing Algorithms, Architectures, Arrangements, and Applications : Poznan, 20th - 22nd September 2023},
   }

2022

1. ZSTN

   Traffic sign detection and recognition using event camera image reconstruction

   Kamil Jeziorek and Tomasz Kryjak

   Zeszyty Studenckiego Towarzystwa Naukowego, 2022

   Abstract arXiv Bib

   This paper presents a method for detection and recognition of traffic signs based on information extracted from an event camera. The solution used a FireNet deep convolutional neural network to reconstruct events into greyscale frames. Two YOLOv4 network models were trained, one based on greyscale images and the other on colour images. The best result was achieved for the model trained on the basis of greyscale images, achieving an efficiency of 87.03%.

   @article{badap-agh:143548,
     title = {Traffic sign detection and recognition using event camera image reconstruction},
     author = {Jeziorek, Kamil and Kryjak, Tomasz},
     year = {2022},
     journal = {Zeszyty Studenckiego Towarzystwa Naukowego},
     series = {Zeszyty Studenckiego Towarzystwa Naukowego},
     pages = {127},
     issn = {1732-0925},
     url = {https://skn.agh.edu.pl/strony/59-hutnicza-konferencja-studenckich-kol-naukowych-agh/},
     collection = {Zeszyty Studenckiego Towarzystwa Naukowego},
   }