Peak file runaway growth with stereo and mono clips.

Discussion:

RICHARD WALKER

2012-05-30 21:03:58 UTC

In normal use the peak files generated for clips are not expected to
exceed about one eighth of the size of a 16bit 44.1 kHz clip.

Over a certain critical clip size, however, the peak file size just
grows and grows to many times the size of the source clip file.

Using a default build of non-daw from the "next" branch I have
determined the critical limits for stereo and mono files. These are a
touch more than 6.5 minutes for stereo and about 13 minutes for mono.

At Jonathan's suggestion I also experimented with changes to
Peaks::cache_levels in the file timeline/src/Engine/Peaks.C which has
a default value of 8. Repeating my tests with values of 7 to 3
resulted in an approximate doubling of the "critical" clip size with
each change.

The numbers seemed to be unremarkable until I converted them to
hexadecimal. For 16 bit 44.1 kHz stereo clips the results were:

Cache Level Stereo samples Peak file size Approx Len. (h:m:s)
8 0106FFFF 0081BFC0 0:06:30.84
7 0205FFFF 00E14FC8 0:12:49.79
6 0404FFFF 0180EFD0 0:25:29.17
5 0803FFFF 02809FD8 0:50:49.43
4 1002FFFF 04005FE0 1:41:31.43
3 2001FFFF 06002FE8 3:22:56.92

The results for mono clips were comparable in a crazy kind of way:

Cache Level Mono samples Peak file size Approx Len. (h:m:s)
8 020D80FF 0081B848 0:13:00.94
7 040BC0FF 00E14C40 0:25:39.21
6 0809E0FF 0180EE38 0:50:58.17
5 1007F0FF 02809F30 1:41:38.77
4 2005F8FF 04005FA8 3:23:02.82
3 4003FCFF 06002FE0 6:46:53.81

The mono and stereo test files for each level are very approximately
the same size as one stereo sample is the size of two mono samples (32
bits).

So, I think I will run with a value of 5 and see what that will mean
to the operation of non-daw. I don't really understand the purpose or
structure of the peaks file, beyond that it has something to do with
drawing the waveform on the time line, Should I be expecting the
screen draws to be unacceptably sluggish at high zoom levels? I don't
know yet, but I suppose I'll find out soon.

Richard

J. Liles

2012-05-30 21:23:59 UTC

Permalink

On Wed, May 30, 2012 at 2:03 PM, RICHARD WALKER <

Post by RICHARD WALKER
In normal use the peak files generated for clips are not expected to
exceed about one eighth of the size of a 16bit 44.1 kHz clip.
Over a certain critical clip size, however, the peak file size just
grows and grows to many times the size of the source clip file.
Using a default build of non-daw from the "next" branch I have
determined the critical limits for stereo and mono files. These are a
touch more than 6.5 minutes for stereo and about 13 minutes for mono.
At Jonathan's suggestion I also experimented with changes to
Peaks::cache_levels in the file timeline/src/Engine/Peaks.C which has
a default value of 8. Repeating my tests with values of 7 to 3
resulted in an approximate doubling of the "critical" clip size with
each change.
The numbers seemed to be unremarkable until I converted them to
Cache Level Stereo samples Peak file size Approx Len. (h:m:s)
8 0106FFFF 0081BFC0 0:06:30.84
7 0205FFFF 00E14FC8 0:12:49.79
6 0404FFFF 0180EFD0 0:25:29.17
5 0803FFFF 02809FD8 0:50:49.43
4 1002FFFF 04005FE0 1:41:31.43
3 2001FFFF 06002FE8 3:22:56.92
Cache Level Mono samples Peak file size Approx Len. (h:m:s)
8 020D80FF 0081B848 0:13:00.94
7 040BC0FF 00E14C40 0:25:39.21
6 0809E0FF 0180EE38 0:50:58.17
5 1007F0FF 02809F30 1:41:38.77
4 2005F8FF 04005FA8 3:23:02.82
3 4003FCFF 06002FE0 6:46:53.81
The mono and stereo test files for each level are very approximately
the same size as one stereo sample is the size of two mono samples (32
bits).
So, I think I will run with a value of 5 and see what that will mean
to the operation of non-daw. I don't really understand the purpose or
structure of the peaks file, beyond that it has something to do with
drawing the waveform on the time line, Should I be expecting the
screen draws to be unacceptably sluggish at high zoom levels? I don't
know yet, but I suppose I'll find out soon.
Richard

Thanks for doing the research. The mono vs. stereo results definitely
suggest it's an integer overflow problem to do with the size of the peak
files. However, I'm not sure if the problem is with the non-daw peak
handling or with ftell/fseek. All of the types used in the process seem to
be sufficiently large to handle much, much larger offset values before
approaching overflow. It's proven too troublesome to debug this in-place in
non-daw, so I think I'll have to write a little test program which just
builds peak files in order to get to the bottom of it.

FYI, the purpose of the peak files is manifold:

1) Improves IO performance by reducing the total number of bytes that must
be read to display a section of waveform at a given zoom level
2) Reduces CPU load by reducing the amount of downsampling required to draw
waveforms at wide zoom levels
2) Improves IO performance by increasing the read locality for peak data at
a given zoom level
3) Effectively precalculates normalization values for arbitrary slices of
the clip

The peak files are basically mipmaps (http://en.wikipedia.org/wiki/Mipmap),
but in sample-space instead of pixel-space.

Without peak files, and the zoom level fitted to your project, all the
bytes for all visible portions of all waveforms would have to be read from
disk and downsampled, which would be pretty slow unless you have an SSD and
CPU to spare.

RICHARD WALKER

2012-05-31 23:40:10 UTC

Permalink

Jonathan,
I have rebuilt non-daw with cache_level=5 as that gives me the best
compromise between maximum likely clip size and (I think) display
speed. I have played around a little with one of my 20 minute stereo
files and it seems to be quite usable, but after spending a week
building my experiment results I find messing with music to be
(temporarily I hope) much less exciting than bug-hunting.

So, plan B; I hope to create the simplest possible peak file structure
using cache_level of 1 and then see what anomalies (if any) show up
when I drop in a mono clip which is 1 sample too big.

There are a few problems with this which I can forsee. My best
guesstimate of the size of clip I will need is around 4GBytes. That is
BIG, but shouldn't take more than a couple of minutes to create with
Audacity. Trouble is I will likely have to make a dozen of them to
zero in on the critical number. Moving them around will be
time-consuming too.

The peaks file should be about 134MBytes, I reckon, so quite
manageable. I had been using noise files but as I intend to look for
anomalies in the data I am thinking of changing to a sine wave tone,
or perhaps a square wave

It would be helpful if I knew how to predict the size of the peaks
file for a given number of clip samples. Is it just a matter of
applying a scale factor and adding in a meta-data overhead?

The basic idea is to find the critical size for cache level 1 and take
a look at the peak file structure for a good clip. Square wave clip
data is likely to provide the most easily recognised data values in
the clip file and help to distinguish the data from any meta-data.

Then load up a clip which is 1 sample longer and look at the newly
created peak file data to see if it still looks right. My reasoning is
that the monster growth of the file is unlikely to be "proper" as it
has to be getting some weird information from somewhere about the clip
to go on up into the gigabyte regions.

I have tried looking at the code to see if I could discover the
information I need there, but I find it hard to see past the
strangeness of C++ and it is difficult to figure out what is being
done and when. Do you have a description of the peak file structure
somewhere?

Richard

Post by J. Liles
On Wed, May 30, 2012 at 2:03 PM, RICHARD WALKER <

Thanks for doing the research. The mono vs. stereo results definitely
suggest it's an integer overflow problem to do with the size of the peak
files. However, I'm not sure if the problem is with the non-daw peak
handling or with ftell/fseek. All of the types used in the process seem to
be sufficiently large to handle much, much larger offset values before
approaching overflow. It's proven too troublesome to debug this in-place in
non-daw, so I think I'll have to write a little test program which just
builds peak files in order to get to the bottom of it.
1) Improves IO performance by reducing the total number of bytes that must
be read to display a section of waveform at a given zoom level
2) Reduces CPU load by reducing the amount of downsampling required to draw
waveforms at wide zoom levels
2) Improves IO performance by increasing the read locality for peak data at
a given zoom level
3) Effectively precalculates normalization values for arbitrary slices of
the clip
The peak files are basically mipmaps (http://en.wikipedia.org/wiki/Mipmap),
but in sample-space instead of pixel-space.
Without peak files, and the zoom level fitted to your project, all the
bytes for all visible portions of all waveforms would have to be read from
disk and downsampled, which would be pretty slow unless you have an SSD and
CPU to spare.

J. Liles

2012-06-05 00:58:05 UTC

Permalink

On Thu, May 31, 2012 at 4:40 PM, RICHARD WALKER <

Post by RICHARD WALKER
Jonathan,
I have rebuilt non-daw with cache_level=5 as that gives me the best
compromise between maximum likely clip size and (I think) display
speed. I have played around a little with one of my 20 minute stereo
files and it seems to be quite usable, but after spending a week
building my experiment results I find messing with music to be
(temporarily I hope) much less exciting than bug-hunting.
So, plan B; I hope to create the simplest possible peak file structure
using cache_level of 1 and then see what anomalies (if any) show up
when I drop in a mono clip which is 1 sample too big.
There are a few problems with this which I can forsee. My best
guesstimate of the size of clip I will need is around 4GBytes. That is
BIG, but shouldn't take more than a couple of minutes to create with
Audacity. Trouble is I will likely have to make a dozen of them to
zero in on the critical number. Moving them around will be
time-consuming too.
The peaks file should be about 134MBytes, I reckon, so quite
manageable. I had been using noise files but as I intend to look for
anomalies in the data I am thinking of changing to a sine wave tone,
or perhaps a square wave
It would be helpful if I knew how to predict the size of the peaks
file for a given number of clip samples. Is it just a matter of
applying a scale factor and adding in a meta-data overhead?
The basic idea is to find the critical size for cache level 1 and take
a look at the peak file structure for a good clip. Square wave clip
data is likely to provide the most easily recognised data values in
the clip file and help to distinguish the data from any meta-data.
Then load up a clip which is 1 sample longer and look at the newly
created peak file data to see if it still looks right. My reasoning is
that the monster growth of the file is unlikely to be "proper" as it
has to be getting some weird information from somewhere about the clip
to go on up into the gigabyte regions.
I have tried looking at the code to see if I could discover the
information I need there, but I find it hard to see past the
strangeness of C++ and it is difficult to figure out what is being
done and when. Do you have a description of the peak file structure
somewhere?
Richard

Post by J. Liles
On Wed, May 30, 2012 at 2:03 PM, RICHARD WALKER <

I believe I have now fixed this bug on the next branch. Can you confirm?
You should be able to revert to using 8 cache levels.

RICHARD WALKER

2012-06-05 01:42:31 UTC

Permalink

Will check it 1st thing in morning (another public holiday for Her
Majesty's Jubilee - whoopee).
Glad you did - I hit a brick wall with my level 1 only 8G+ wav file. I
made it ok in Audacity but when I dropped it on the unsuspecting
non-daw it threw a wobbler and spat most of it out.

Wow; tweaks, features, clangs, AND you had time to fix the runaway!
Hats off to you sir!

R

Post by J. Liles
On Thu, May 31, 2012 at 4:40 PM, RICHARD WALKER <

Post by J. Liles
On Wed, May 30, 2012 at 2:03 PM, RICHARD WALKER <

I believe I have now fixed this bug on the next branch. Can you confirm?
You should be able to revert to using 8 cache levels.

RICHARD WALKER

2012-06-05 01:43:49 UTC

Permalink

Dammit, sorry if that was another top or bottom post - I keep
forgetting about this damn client's default defaults.

RICHARD WALKER

2012-06-05 02:42:31 UTC

Permalink

This is getting silly. Should have been in bed hours ago.

No runaways, of course. Congrats. And a nice side effect; the peaks
file is now only 1/8 the size it was for the same clip. That's a
result, I think.

I'll give it a brisker twirl tomorrow.

Richard

Post by RICHARD WALKER
Dammit, sorry if that was another top or bottom post - I keep
forgetting about this damn client's default defaults.

RICHARD WALKER

2012-06-05 10:25:43 UTC

Permalink

OK, I messed up. That's what comes of trying to rush things. The one
thing I forgot to do last night was install the freshly built non-daw!

I have tested the new build and it works for two test clips; one of
twice the original critical size, and one of 19min 46sec. Both now
enjoy stable, fixed peak file sizes. I won't comment on the length as
I have to take family into town for some early(?) shopping.

R

Post by RICHARD WALKER
This is getting silly. Should have been in bed hours ago.
No runaways, of course. Congrats. And a nice side effect; the peaks
file is now only 1/8 the size it was for the same clip. That's a
result, I think.
I'll give it a brisker twirl tomorrow.
Richard

Post by RICHARD WALKER
Dammit, sorry if that was another top or bottom post - I keep
forgetting about this damn client's default defaults.

RICHARD WALKER

2012-06-05 12:07:27 UTC

Permalink

Whew, that was lucky. Made it into town and back before the police
closed all the roads. Why did nobody tell me the Olympic Torch was
coming through Enniskillen today? When you live on one side of the
river and the shops are on the other and the only way across is the
island town and the police decide to close the roads... well, we made
it.

I have now tested with critical size (+1 stereo sample), double that
and 8 times that. The peak files are definitely much smaller than ever
before. The largest peak file I was able to safely generate was for
the critical size. Its peak file is now just a quarter of that size.

6.5 minutes used to give me a peak file of 8.5MBytes, now only 2.1MBytes
8.5MBytes peak file is now created for a clip size of around 26 minutes.

Done and dusted, thank you

Richard

Post by RICHARD WALKER
OK, I messed up. That's what comes of trying to rush things. The one
thing I forgot to do last night was install the freshly built non-daw!
I have tested the new build and it works for two test clips; one of
twice the original critical size, and one of 19min 46sec. Both now
enjoy stable, fixed peak file sizes. I won't comment on the length as
I have to take family into town for some early(?) shopping.
R

Post by RICHARD WALKER
Dammit, sorry if that was another top or bottom post - I keep
forgetting about this damn client's default defaults.