NCN Completed

NCN Sonata

National Science Centre, Poland (Sonata 12)

The development of computing resources organization in latest generation of heterogeneous reconfigurable devices enabling real-time processing of UHD/4K video stream

Principal Investigator Tomasz Kryjak, PhD
Duration 2017 – 2021
Budget 255600 PLN
Project No. 2016/23/D/ST6/01389

4K Embedded Platforms

Abstract

The aim of the project is the development of computation organization in reconfigurable Multiprocessor System on Chip (MPSoC), which will enable the realization of real-time processing of UHD/4K video stream (3840 x 2160 pixels @ 24/30/50 frames per second).

Nowadays a very dynamic development of cameras can be observed. Devices which support 1920 x 1080 pixels resolutions (so-called Full HD) are widespread. What is more, recording with 3840 x 2160 pixels resolutions (i.e. UHD/4K in 16:9 format) is available in the last generation of popular sport cameras and camcorders. Also, today's smart phones are equipped with a 12 and more megapixels sensor, which allows to capture 4K videos. Additionally customer-grade displaying devices, like 4K monitors or TVs are already available on the market for some time.

While the acquisition and display of 4K video stream in now fully supported, performing image processing operations with the commonly used computing platforms (i.e. personal computers and notebooks) is very time consuming and requires a lot of disk memory. One can be convinced of this, when editing this type of video. One should realize that 1 uncompressed colour 4K frame equals about 23 MB of data and a 1 minute video (30 frames per second) more than 41 GB. However, in many cases the use of high resolution video stream could improve the performance of the considered vision systems. A good example is pedestrian detection used in recently very popular advanced driver assistance systems (ADAS). On a high resolution image we are able to detect persons, who are quite far away from the camera and this enables to detect a potentially hazardous situation much earlier.

At this point, we could ask the following question – is it possible to realise real-time processing, e.g. pedestrian detection for 4K video stream with 24/30/50 frames per second. It should be also noted, that in the considered example, as well as many others, such as drones (UAVs), electric vehicles or video surveillance, we cannot use powerful computing platforms like supercomputers or GPUs because they consume too much energy. A very promising solution seem the latest generation of reprogrammable devices (FPGA) additionally equipped with a powerful processor system and graphic processor (e.g. the Zynq UltraScale+ family by Xilinx). They allow to perform parallel data processing, which is essential for obtaining real-time performance. Additionally they can be used very close to the vision sensor (constituting the so-called smart camera). Therefore, the huge data stream does not need to be transmitted, as all required processing is performed locally. Moreover, a distributed computing system is also more energy efficient.

During the project inter alia the following elementary vision algorithms will be considered for real-time implementation: image filtration (averaging, Gaussian, median, but also more advanced methods like vector median filter, bilateral filter), edge detection (Sobel, Canny), single-pass and double-pass connected component labeling, as well as depth map estimation (stereo-vision, image rectification, structure from motion), moving objects segmentation (optical flow), foreground object segmentation, object detection (e.g. people, cars) and object tracking. With these components, it is possible to build advanced vision systems running up to 4K resolution e.g. abandoned luggage detection or re-identification of people in a multi-camera surveillance system.

Moreover, a methodology for transforming a high level language (C/C++) description of a vision algorithm into an efficient hardware-software implementation in a programmable MPSoC devices will be developed. In addition, the proposed approach will be compared with tools, which perform this computing resource assignment automatically. This experiments should provide an answer to the question, whether and for which algorithms it is possible to obtain a hardware-software computing system capable of processing 4K images in real time.

In this project video stream processing is considered, as it is a very nice example to show all the advantages of hardware-software solutions. However, the results, methodology and know-how could be also used in other applications that require fast data processing (e.g. biomedical), especially when embedded systems are considered.

Contractors: Dominika Przewłocka-Rus, Marcin Kowalczyk, Piotr Janus, Krzysztof Błachut, Hubert Szolc, Mateusz Wąsala, Piotr Ciarach

Publications

2023

DASIP Sonata
Real-time FPGA implementation of the semi-global matching stereo vision algorithm for a 4K/UHD video stream

Mariusz Grabowski and Tomasz Kryjak

In Design and Architecture for Signal and Image Processing : 16th international workshop, DASIP 2023 : Toulouse, France, January 16–18, 2023 : proceedings, 2023

Abstract DOI arXiv Bib

In this paper, we propose a real-time FPGA implementation of the Semi-Global Matching (SGM) stereo vision algorithm. The designed module supports a 4K/Ultra HD (3840 x 2160 pixels @ 30 frames per second) video stream in a 4 pixel per clock (ppc) format and a 64-pixel disparity range. The baseline SGM implementation had to be modified to process pixels in the 4ppc format and meet the timing constrains, however, our version provides results comparable to the original design. The solution has been positively evaluated on the Xilinx VC707 development board with a Virtex-7 FPGA device.
@inproceedings{badap-agh:146665, title = {Real-time FPGA implementation of the semi-global matching stereo vision algorithm for a 4K/UHD video stream}, projects = {Sonata}, author = {Grabowski, Mariusz and Kryjak, Tomasz}, year = {2023}, series = {Design and Architecture for Signal and Image Processing : 16th~international workshop, DASIP~2023 : Toulouse, France, January 16–18, 2023 : proceedings}, pages = {70–81}, doi = {10.1007/978-3-031-29970-4_6}, collection = {Design and Architecture for Signal and Image Processing : 16th~international workshop, DASIP~2023 : Toulouse, France, January 16–18, 2023 : proceedings}, }

2022

Electronics Sonata
Real-time CLAHE algorithm implementation in SoC FPGA device for 4K UHD video stream

Tomasz Kryjak, Krzysztof Blachut, Hubert Szolc, and Mateusz Wasala

Electronics, 2022

Abstract DOI Bib

One of the problems encountered in the field of computer vision and video data analysis is the extraction of information from low-contrast images. This problem can be addressed in several ways, including the use of histogram equalisation algorithms. In this work, a method designed for this purpose—the Contrast-Limited Adaptive Histogram Equalization (CLAHE) algorithm—is implemented in hardware. An FPGA platform is used for this purpose due to the ability to run parallel computations and very low power consumption. To enable the processing of a 4K resolution (UHD, 3840 × 2160 pixels) video stream at 60 fps (frames per second) by using the CLAHE method, it is necessary to use a vector data format and process multiple pixels simultaneously. The algorithm realised in this work can be a component of a larger vision system, such as in autonomous vehicles or drones, but it can also support the analysis of underwater, thermal, or medical images both by humans and in an automated system.
@article{badap-agh:141231, title = {Real-time CLAHE algorithm implementation in SoC FPGA device for 4K UHD video stream}, projects = {Sonata}, author = {Kryjak, Tomasz and Blachut, Krzysztof and Szolc, Hubert and Wasala, Mateusz}, year = {2022}, journal = {Electronics}, series = {Electronics}, volume = {11}, number = {14}, pages = {2248}, doi = {10.3390/electronics11142248}, issn = {2079-9292}, collection = {Electronics}, }
MECO Sonata
Real-time HOG+SVM based object detection using SoC FPGA for a UHD video stream

Mateusz Wasala and Tomasz Kryjak

In 11th Mediterranean Conference on Embedded Computing (MECO) : 7–10 June 2022, Budva, Montenegro, 2022

Abstract DOI arXiv Bib

Object detection is an essential component of many vision systems. For example, pedestrian detection is used in advanced driver assistance systems (ADAS) and advanced video surveillance systems (AVSS). Currently, most detectors (e.g. the YOLO - You Only Look Once - family) use deep convolutional neural networks. However, due to their high computational complexity, they are not able to process a very high-resolution video stream in real-time, especially within a limited energy budget. In this paper we present a hardware implementation of the well-known pedestrian detector with HOG (Histogram of Oriented Gradients) feature extraction and SVM (Support Vector Machine) classification. Our system running onAMD Xilinx Zynq UltraScale+ MPSoC(Multiprocessor System on Chip) device allows real-time processing of 4K resolution (UHD - Ultra High Definition,3840 × 2160 pixels) video for 60 frames per second. The system is capable of detecting a pedestrian in a single scale. The results obtained confirm the high suitability of reprogrammable devices in the real-time implementation of embedded vision systems.
@inproceedings{badap-agh:140732, title = {Real-time HOG+SVM based object detection using SoC FPGA for a UHD video stream}, projects = {Sonata}, author = {Wasala, Mateusz and Kryjak, Tomasz}, year = {2022}, series = {11th~Mediterranean Conference on Embedded Computing (MECO) [Dokument elektroniczny] : 7–10 June 2022, Budva, Montenegro}, pages = {1–6}, doi = {10.1109/MECO55406.2022.9797113}, collection = {11th~Mediterranean Conference on Embedded Computing (MECO) [Dokument elektroniczny] : 7–10 June 2022, Budva, Montenegro}, }
Sensors Sonata
Real-time efficient FPGA implementation of the multi-scale Lucas-Kanade and Horn-Schunck optical flow algorithms for a 4K video stream

Krzysztof Blachut and Tomasz Kryjak

Sensors, 2022

Abstract DOI Bib

The information about optical flow, i.e., the movement of pixels between two consecutive images from a video sequence, is used in many vision systems, both classical and those based on deep neural networks. In some robotic applications, e.g., in autonomous vehicles, it is necessary to calculate the flow in real time. This represents a challenging task, especially for high-resolution video streams. In this work, two gradient-based algorithms-Lucas-Kanade and Horn-Schunck-were implemented on a ZCU 104 platform with Xilinx Zynq UltraScale+ MPSoC FPGA. A vector data format was used to enable flow calculation for a 4K (Ultra HD, 3840 × 2160 pixels) video stream at 60 fps. In order to detect larger pixel displacements, a multi-scale approach was used in both algorithms. Depending on the scale, the calculations were performed for different data formats, allowing for more efficient processing by reducing resource utilisation. The presented solution allows real-time optical flow determination in multiple scales for a 4K resolution with estimated energy consumption below 6 W. The algorithms realised in this work can be a component of a larger vision system in advanced surveillance systems or autonomous vehicles.
@article{badap-agh:140919, title = {Real-time efficient FPGA implementation of the multi-scale Lucas-Kanade and Horn-Schunck optical flow algorithms for a~4K video stream}, projects = {Sonata}, author = {Blachut, Krzysztof and Kryjak, Tomasz}, year = {2022}, journal = {Sensors}, series = {Sensors}, volume = {22}, number = {13}, pages = {5017}, doi = {10.3390/s22135017}, issn = {1424-8220}, collection = {Sensors}, }

2021

FPL Sonata
Preludium

A comparison of real-time 4K/UltraHD connected component labelling architectures

Marcin Kowalczyk and Tomasz Kryjak

In FPL 2021 : 31st international conference on Field-Programmable Logic and Applications : Dresden, Germany 30 August – 3 September 2021 : proceedings, 2021

Abstract DOI Bib

This work presents a comparison of hardware architectures realising a connected component labelling (CCL) algorithm in reprogrammable logic. The architectures are capable of processing a video stream with 4K/UltraHD resolution at 60 frames per second in real-time. The modules were verified in simulation and in hardware on the Xilinx Zynq Ultrascale+ MPSoC chip on the ZCU104 evaluation board.

@inproceedings{badap-agh:137405,
  title = {A comparison of real-time 4K/UltraHD connected component labelling architectures},
  projects = {Sonata, Preludium-Kowalczyk},
  author = {Kowalczyk, Marcin and Kryjak, Tomasz},
  year = {2021},
  series = {FPL~2021 [Dokument elektroniczny] : 31st international conference on Field-Programmable Logic and Applications : Dresden, Germany 30 August – 3 September 2021 : proceedings},
  pages = {401–401},
  doi = {10.1109/FPL53798.2021.00086},
  collection = {FPL~2021 [Dokument elektroniczny] : 31st international conference on Field-Programmable Logic and Applications : Dresden, Germany 30 August – 3 September 2021 : proceedings},
}

FPL Sonata
Quantised Siamese tracker for 4K/UltraHD video stream – a demo

Dominika Przewlocka-Rus and Tomasz Kryjak

In FPL 2021 : 31st international conference on Field-Programmable Logic and Applications : Dresden, Germany 30 August – 3 September 2021 : proceedings, 2021

Abstract DOI Bib

This demo presents a hardware architecture for an object tracker based on a quantised Siamese neural network. The system is designed to work with a 4K/UltraHD input video stream and to meet the real time and energy efficiency constraints. The network is designed using Brevitas and implemented with FINN tool from Xilinx. The Siamese tracker runs on the ZCU 104 board, with the Zynq UltraScale+ MPSoC device from Xilinx.
@inproceedings{badap-agh:137408, title = {Quantised Siamese tracker for 4K/UltraHD video stream – a demo}, projects = {Sonata}, author = {Przewlocka-Rus, Dominika and Kryjak, Tomasz}, year = {2021}, series = {FPL~2021 [Dokument elektroniczny] : 31st international conference on Field-Programmable Logic and Applications : Dresden, Germany 30 August – 3 September 2021 : proceedings}, pages = {405–405}, doi = {10.1109/FPL53798.2021.00089}, collection = {FPL~2021 [Dokument elektroniczny] : 31st international conference on Field-Programmable Logic and Applications : Dresden, Germany 30 August – 3 September 2021 : proceedings}, }
JSPS Sonata
Preludium
Real-time FPGA implementation of parallel connected component labelling for a 4K video stream

Marcin Kowalczyk, Piotr Ciarach, Dominika Przewlocka-Rus, Hubert Szolc, and Tomasz Kryjak

Journal of Signal Processing Systems for Signal, Image, and Video Technology, 2021

Abstract DOI Bib

In this paper, a hardware implementation in reconfigurable logic of a single-pass connected component labelling (CCL) and connected component analysis (CCA) module is presented. The main novelty of the design is the support of a video stream in 2 and 4 pixel per clock format (2 and 4 ppc) and real-time processing of 4K/UHD video stream (3840 x 2160 pixels) at 60 frames per second. We discuss several approaches to the issue and present in detail the selected ones. The proposed module was verified in an exemplary application – skin colour areas segmentation – on the ZCU 102 and ZCU 104 evaluation boards equipped with Xilinx Zynq UltraScale+ MPSoC devices.
@article{badap-agh:134265, title = {Real-time FPGA implementation of parallel connected component labelling for a 4K video stream}, projects = {Sonata, Preludium-Kowalczyk}, author = {Kowalczyk, Marcin and Ciarach, Piotr and Przewlocka-Rus, Dominika and Szolc, Hubert and Kryjak, Tomasz}, year = {2021}, journal = {Journal of Signal Processing Systems for Signal, Image, and Video Technology}, series = {Journal of Signal Processing Systems for Signal, Image, and Video Technology}, volume = {93}, number = {5}, pages = {481}, doi = {10.1007/s11265-021-01636-4}, issn = {1939-8018}, collection = {Journal of Signal Processing Systems for Signal, Image, and Video Technology}, }

2019

DASIP Sonata

Hardware implementation of the SURF feature detector for 4K 4PPC video stream – a demo

Karol Radwan and Tomasz Kryjak

In dasip : the conference on Design and Architectures for Signal and Image Processing : TEXPO graduate student research competition : 16–18 October, 2019, Montréal, Canada, 2019

Abstract Bib

This paper describes a hardware implementation of the Speeded Up Robust Features (SURF) algorithm. The feature detector has been fully implemented in programmable logic resources (FPGA) of a Xilinx Zynq UltraScale+ MPSoC device on the ZCU 104 development board. The system has been adapted to work with a 4K (3840x2160 @ 60 fps) video stream in 4 ppc (pixel per clock) format.

@inproceedings{badap-agh:126255,
  title = {Hardware implementation of the SURF feature detector for 4K 4PPC video stream – a demo},
  projects = {Sonata},
  author = {Radwan, Karol and Kryjak, Tomasz},
  year = {2019},
  series = {dasip [Dokument elektroniczny] : the conference on Design and Architectures for Signal and Image Processing : TEXPO graduate student research competition : 16–18 October, 2019, Montr\'{e}al, Canada},
  pages = {1–1},
  url = {https://dasip-conference.org/wp-content/uploads/sites/2/2019/10/DASIP2019_paper_35.pdf},
  collection = {dasip [Dokument elektroniczny] : the conference on Design and Architectures for Signal and Image Processing : TEXPO graduate student research competition : 16–18 October, 2019, Montr\'{e}al, Canada},
}

ARC Sonata
Real-time FPGA implementation of connected component labelling for a 4K video stream

Piotr Ciarach, Marcin Kowalczyk, Dominika Przewlocka-Rus, and Tomasz Kryjak

In Applied Reconfigurable Computing : 15th international symposium, ARC 2019 : Darmstadt, Germany, April 9–11, 2019 : proceedings, 2019

Abstract DOI Bib

In this paper, a hardware implementation in reconfigurable logic of a single-pass connected component labelling (CCL) and connected component analysis (CCA) module is presented. The main novelty of the design is the support of a video stream in 2 and 4 pixel per clock format (2 and 4 ppc) and real-time processing of 4K/UHD video stream (3840 x 2160 pixels) at 60 frames per second. We discuss several approaches to the issue and present in detail the selected ones. The proposed module was verified in an exemplary application – skin colour areas segmentation – on the ZCU 102 and ZCU 104 evaluation boards equipped with Xilinx Zynq UltraScale+ MPSoC devices.
@inproceedings{badap-agh:121175, title = {Real-time FPGA implementation of connected component labelling for a 4K video stream}, projects = {Sonata}, author = {Ciarach, Piotr and Kowalczyk, Marcin and Przewlocka-Rus, Dominika and Kryjak, Tomasz}, year = {2019}, series = {Applied Reconfigurable Computing : 15th~international symposium, ARC~2019 : Darmstadt, Germany, April 9–11, 2019 : proceedings}, pages = {165–180}, doi = {10.1007/978-3-030-17227-5_13}, collection = {Applied Reconfigurable Computing : 15th~international symposium, ARC~2019 : Darmstadt, Germany, April 9–11, 2019 : proceedings}, }
DASIP Sonata
Real-time implementation of adaptive correlation filter tracking for 4K video stream in Zynq UltraScale+ MPSoC

Marcin Kowalczyk, Dominika Przewlocka-Rus, and Tomasz Kryjak

In dasip : conference on Design and Architectures for Signal and Image Processing : 16–18 October, 2019, Montréal, Canada, 2019

Abstract DOI Bib

In this paper a hardware-software implementation of adaptive correlation filter tracking for a 3840 × 2160 @ 60 fps video stream in a Zynq UltraScale+ MPSoC is discussed. Correlation filters gained popularity in recent years because of their efficiency and good results in the VOT (Visual Object Tracking) challenge. An implementation of the MOSSE (Minimum Output Sum of Squared Error) algorithm is presented. It utilizes 2-dimensional FFT for computing correlation and updates filter coefficients in every frame. The initial filter coefficients are computed on the ARM processor in the PS (Processing System), while all other operations are preformed in PL (Programmable Logic). The presented architecture was described with the use of Verilog hardware description language.
@inproceedings{badap-agh:126251, title = {Real-time implementation of adaptive correlation filter tracking for 4K video stream in Zynq UltraScale+ MPSoC}, projects = {Sonata}, author = {Kowalczyk, Marcin and Przewlocka-Rus, Dominika and Kryjak, Tomasz}, year = {2019}, series = {dasip [Dokument elektroniczny] : conference on Design and Architectures for Signal and Image Processing : 16–18 October, 2019, Montr\'{e}al, Canada}, pages = {53–58}, doi = {10.1109/DASIP48288.2019.9049203}, collection = {dasip [Dokument elektroniczny] : conference on Design and Architectures for Signal and Image Processing : 16–18 October, 2019, Montr\'{e}al, Canada}, }

DASIP Sonata

XNOR CNNs in FPGA: real-time detection and classification of traffic signs in 4K – a demo

Dominika Przewlocka-Rus, Marcin Kowalczyk, and Tomasz Kryjak

In dasip : the conference on Design and Architectures for Signal and Image Processing : TEXPO graduate student research competition : 16–18 October, 2019, Montréal, Canada, 2019

Bib

@inproceedings{badap-agh:126253,
  title = {XNOR CNNs in FPGA: real-time detection and classification of traffic signs in 4K – a demo},
  projects = {Sonata},
  author = {Przewlocka-Rus, Dominika and Kowalczyk, Marcin and Kryjak, Tomasz},
  year = {2019},
  series = {dasip [Dokument elektroniczny] : the conference on Design and Architectures for Signal and Image Processing : TEXPO graduate student research competition : 16–18 October, 2019, Montr\'{e}al, Canada},
  pages = {1–1},
  url = {https://dasip-conference.org/wp-content/uploads/sites/2/2019/10/DASIP2019_paper_34.pdf},
  collection = {dasip [Dokument elektroniczny] : the conference on Design and Architectures for Signal and Image Processing : TEXPO graduate student research competition : 16–18 October, 2019, Montr\'{e}al, Canada},
}

2018

DASIP Sonata
Real-time implementation of contextual image processing operations for 4K video stream in Zynq UltraScale+ MPSoC

Marcin Kowalczyk, Dominika Przewlocka-Rus, and Tomasz Kryjak

In dasip : the 2018 conference on Design & Architectures for Signal & Image Processing : Porto, Portugal, October 10–12, 2018, 2018

Abstract DOI Bib

In this paper hardware implementation of selected contextual based image pre-processing modules for a 3840×2160 @60 FPS video stream in a Zynq UltraScale+ MPSoC is discussed. The following operations are considered: simple averaging (box filter), Gaussian filter, edge detection using the Sobel and Canny methods, median filter and morphological erosion and dilation operations. The scheme for implementing contextual based operations for a video stream in the format of 2 and 4 pixels per clock and challenges related to the pipelined implementation of processing such data are described. Also the use of logic resources and energy efficiency of modules described in the Verilog hardware description language and using the High Level Synthesis tools (Vivado HLS, SDSoC and xfOpenCV library) are compared. All designed modules support real-time processing of a 4K@60 FPS video stream.
@inproceedings{badap-agh:119438, title = {Real-time implementation of contextual image processing operations for 4K video stream in Zynq UltraScale+ MPSoC}, projects = {Sonata}, author = {Kowalczyk, Marcin and Przewlocka-Rus, Dominika and Kryjak, Tomasz}, year = {2018}, series = {dasip [Dokument elektroniczny] : the 2018~conference on Design \& Architectures for Signal \& Image Processing : Porto, Portugal, October 10–12, 2018}, pages = {37–42}, doi = {10.1109/DASIP.2018.8597105}, collection = {dasip [Dokument elektroniczny] : the 2018~conference on Design \& Architectures for Signal \& Image Processing : Porto, Portugal, October 10–12, 2018}, }

DASIP Sonata

Real-time implementation of contextual image processing operations for 4K video stream in Zynq UltraScale+ MPSoC – a demo

Marcin Kowalczyk, Dominika Przewlocka-Rus, and Tomasz Kryjak

In dasip 2018 : conference on Design and Architectures for Signal and Image Processing : October 10–12 2018, Porto, Portugal, 2018

Bib

@inproceedings{badap-agh:118432,
  title = {Real-time implementation of contextual image processing operations for 4K video stream in Zynq UltraScale+ MPSoC – a~demo},
  projects = {Sonata},
  author = {Kowalczyk, Marcin and Przewlocka-Rus, Dominika and Kryjak, Tomasz},
  year = {2018},
  series = {dasip~2018 [Dokument elektroniczny] : conference on Design and Architectures for Signal and Image Processing : October 10–12 2018, Porto, Portugal},
  pages = {1–2},
  url = {https://web.fe.up.pt/~specs/events/dasip2018/files/paper_46.pdf},
  collection = {dasip~2018 [Dokument elektroniczny] : conference on Design and Architectures for Signal and Image Processing : October 10–12 2018, Porto, Portugal},
}