Buffer Overflow vulnerability in libde265 v1.0.12 allows a local attacker to cause a denial of service via the allocation size exceeding the maximum supported size of 0x10000000000.
## Libde265 v1.0.12 was discovered that requested allocation size exceeds maximum supported size of 0x10000000000

### Description

Libde265 v1.0.12 was discovered that requested allocation size 0xffffffffffff5b00 (0xffffffffffff6b00 after adjustments for alignment, red zones etc.) exceeds maximum supported size of 0x10000000000 (thread T0).

This vulnerability allows attackers to cause a Denial of Service (DoS) and cause the system storage space to be used up.

It is important to note that we recommend reproducing this vulnerability **in a docker environment**, as it is likely to affect your operating system and storage space!

The following two images show that my docker storage space was full at the time of fuzzing this vulnerability.

![image-20231104013659975](https://github.com/strukturag/libde265/assets/67280615/d451c191-7c39-4118-abd1-08286c4f2f58)

![image-20231104013554244](https://github.com/strukturag/libde265/assets/67280615/03ed2a2b-94d9-442c-b2e4-0d7050d9d2f4)

### Version

```shell
libde265/dec265  v1.0.12
```

### ASAN Log

./dec265/dec265 --noaccel --disable-deblocking --disable-sao -L poc2libde265

```shell
=================================================================
==69==ERROR: AddressSanitizer: requested allocation size 0xffffffffffff5b00 (0xffffffffffff6b00 after adjustments for alignment, red zones etc.) exceeds maximum supported size of 0x10000000000 (thread T0)
    #0 0x4c662d in operator new[](unsigned long) (/afltest/libde265/dec265/dec265+0x4c662d)
    #1 0x4c90d6 in convert_to_8bit(unsigned char const*, int, int, int, int) /afltest/libde265/dec265/dec265.cc:243:18
    #2 0x4c90d6 in display_sdl(de265_image const*) /afltest/libde265/dec265/dec265.cc:298:12
    #3 0x4cba68 in output_image(de265_image const*) /afltest/libde265/dec265/dec265.cc:353:12
    #4 0x4cd633 in main /afltest/libde265/dec265/dec265.cc:802:20
    #5 0x7ffff790d082 in __libc_start_main /build/glibc-BHL3KM/glibc-2.31/csu/../csu/libc-start.c:308:16

==69==HINT: if you don't care about these errors you may set allocator_may_return_null=1
SUMMARY: AddressSanitizer: allocation-size-too-big (/afltest/libde265/dec265/dec265+0x4c662d) in operator new[](unsigned long)
==69==ABORTING
```

### Reproduction

```shell
./autogen.sh
export CFLAGS="-g -lpthread -fsanitize=address"
export CXXFLAGS="-g -lpthread -fsanitize=address"
CC=clang CXX=clang++ ./configure --disable-shared
make -j 32

./dec265/dec265 --noaccel --disable-deblocking --disable-sao -L poc2libde265
```

### PoC

poc2libde265: https://github.com/Frank-Z7/z-vulnerabilitys/blob/main/poc2libde265

### Reference

https://github.com/strukturag/libde265

### Environment

```
ubuntu:20.04
gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)
clang version 10.0.0-4ubuntu1
afl-cc++4.09
```

### Credit

Zeng Yunxiang
Song Jiaxuan
Libde265 v1.0.12 was discovered that requested allocation size exceeds maximum supported size of 0x10000000000
This is a GitHub Issue
repo:libde265
owner:strukturag
Title : Libde265 v1.0.12 was discovered that requested allocation size exceeds maximum supported size of 0x10000000000
Issue date:

--- start body ---
## Libde265 v1.0.12 was discovered that requested allocation size exceeds maximum supported size of 0x10000000000

### Description

Libde265 v1.0.12 was discovered that requested allocation size 0xffffffffffff5b00 (0xffffffffffff6b00 after adjustments for alignment, red zones etc.) exceeds maximum supported size of 0x10000000000 (thread T0).

This vulnerability allows attackers to cause a Denial of Service (DoS) and cause the system storage space to be used up.

It is important to note that we recommend reproducing this vulnerability **in a docker environment**, as it is likely to affect your operating system and storage space!

The following two images show that my docker storage space was full at the time of fuzzing this vulnerability.

![image-20231104013659975](https://github.com/strukturag/libde265/assets/67280615/d451c191-7c39-4118-abd1-08286c4f2f58)

![image-20231104013554244](https://github.com/strukturag/libde265/assets/67280615/03ed2a2b-94d9-442c-b2e4-0d7050d9d2f4)

### Version

```shell
libde265/dec265  v1.0.12
```

### ASAN Log

./dec265/dec265 --noaccel --disable-deblocking --disable-sao -L poc2libde265

```shell
=================================================================
==69==ERROR: AddressSanitizer: requested allocation size 0xffffffffffff5b00 (0xffffffffffff6b00 after adjustments for alignment, red zones etc.) exceeds maximum supported size of 0x10000000000 (thread T0)
    #0 0x4c662d in operator new[](unsigned long) (/afltest/libde265/dec265/dec265+0x4c662d)
    #1 0x4c90d6 in convert_to_8bit(unsigned char const*, int, int, int, int) /afltest/libde265/dec265/dec265.cc:243:18
    #2 0x4c90d6 in display_sdl(de265_image const*) /afltest/libde265/dec265/dec265.cc:298:12
    #3 0x4cba68 in output_image(de265_image const*) /afltest/libde265/dec265/dec265.cc:353:12
    #4 0x4cd633 in main /afltest/libde265/dec265/dec265.cc:802:20
    #5 0x7ffff790d082 in __libc_start_main /build/glibc-BHL3KM/glibc-2.31/csu/../csu/libc-start.c:308:16

==69==HINT: if you don't care about these errors you may set allocator_may_return_null=1
SUMMARY: AddressSanitizer: allocation-size-too-big (/afltest/libde265/dec265/dec265+0x4c662d) in operator new[](unsigned long)
==69==ABORTING
```

### Reproduction

```shell
./autogen.sh
export CFLAGS="-g -lpthread -fsanitize=address"
export CXXFLAGS="-g -lpthread -fsanitize=address"
CC=clang CXX=clang++ ./configure --disable-shared
make -j 32

./dec265/dec265 --noaccel --disable-deblocking --disable-sao -L poc2libde265
```

### PoC

poc2libde265: https://github.com/Frank-Z7/z-vulnerabilitys/blob/main/poc2libde265

### Reference

https://github.com/strukturag/libde265

### Environment

```
ubuntu:20.04
gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)
clang version 10.0.0-4ubuntu1
afl-cc++4.09
```

### Credit

Zeng Yunxiang
Song Jiaxuan
--- end body ---

comments :
--- start comments ---

--- end comments ---