Hi Richard,

Would you share your script, after finishing? Would be very interesting!

Greetings, Stephan.

Am 06/06/2016 um 01:07 AM schrieb Richard F:

I meant to send this to the list

On 4/06/2016 08:45, Klaus Schmidinger wrote:

...

...

On 30 May 2016, at 22:45, Richard Flists@keynet-technology.com wrote:

Hi Klaus,

I'm 99% of the way to finishing my H264 ffmpeg transcoding script, I'm just getting 2 of these warnings when reindexing transcoded files

May 30 21:08:09 ha-server vdr: [31785] WARNING: required 5 video TS packets to determine frame type May 30 21:08:09 ha-server vdr: [31785] WARNING: required 5 video TS packets to determine frame type

Sometimes the number is 4. Fortunately not over 6 yet. Is this a functional problem ? (files play in everything I need them to). I reduced the I-frame gap which also improves seeking performance, but that's not the issue.

Looking at the code I'm not quite sure what it's looking for - can you help?

I may be able to adjust codec / muxer to avoid warnings

Hello Richard,

VDR scans the TS packets in order to determine where a new frame starts (unfortunately the authors of the DVB standard and codecs buried that information deep in the data, instead of flagging it at the top TS level). In doing so, it only checks as few TS packets as possible, and usually it finds that information in the first few packets of a frame. The warning message you get means that the information was found later than usual. This may be caused by audio TS packets that are interspersed with the video packets. Imagine a frame starting with one or two video TS packets (that don’t yet contain the frame start info) and then being followed by several audio TS packets. All these need to be buffered before the important video TS packets can be seen. For performance reasons this buffering is kept minimal, and the warning indicates the more of the buffer than usual is used. Hope I was able to clear things up a little.

Klaus

Klaus

Thanks. I did more research using DVB Inspector and now know where the issue comes from. FFmpeg gives libx264 a long command line with many parameters, and libx264 then shoves this whole string long-hand into the header of the video stream, like so

0001 0203 0405 0607 0809 0A0B 0C0D 0E0F 0123456789ABCDEF 0x000000 0000 01E0 0000 80C0 0A31 000F 4F41 1100 .........1..OA.. 0x000010 0F17 0100 0000 0109 F000 0000 0167 6400 .............gd. 0x000020 28AC D940 B012 6C10 4000 0003 0040 0000 (..@..l.@....@.. 0x000030 0C83 C60C 6580 0000 0001 68EB E172 C8B0 ....e.....h..r.. 0x000040 0000 0106 05FF FFA9 DC45 E9BD E6D9 48B7 .........E....H. 0x000050 962C D820 D923 EEEF 7832 3634 202D 2063 .,. .#..x264 - c 0x000060 6F72 6520 3134 3820 7232 3639 344D 2033 ore 148 r2694M 3 0x000070 6237 3036 3435 202D 2048 2E32 3634 2F4D b70645 - H.264/M 0x000080 5045 472D 3420 4156 4320 636F 6465 6320 PEG-4 AVC codec 0x000090 2D20 436F 7079 6C65 6674 2032 3030 332D - Copyleft 2003- 0x0000a0 3230 3136 202D 2068 7474 703A 2F2F 7777 2016 - http://ww

    0001 0203 0405 0607 0809 0A0B 0C0D 0E0F 0123456789ABCDEF

0x000000 4701 0011 772E 7669 6465 6F6C 616E 2E6F G...w.videolan.o 0x000010 7267 2F78 3236 342E 6874 6D6C 202D 206F rg/x264.html - o 0x000020 7074 696F 6E73 3A20 6361 6261 633D 3120 ptions: cabac=1 0x000030 7265 663D 3320 6465 626C 6F63 6B3D 313A ref=3 deblock=1: 0x000040 303A 3020 616E 616C 7973 653D 3078 333A 0:0 analyse=0x3: 0x000050 3078 3131 3320 6D65 3D68 6578 2073 7562 0x113 me=hex sub 0x000060 6D65 3D37 2070 7379 3D31 2070 7379 5F72 me=7 psy=1 psy_r 0x000070 643D 312E 3030 3A30 2E30 3020 6D69 7865 d=1.00:0.00 mixe 0x000080 645F 7265 663D 3120 6D65 5F72 616E 6765 d_ref=1 me_range 0x000090 3D31 3620 6368 726F 6D61 5F6D 653D 3120 =16 chroma_me=1 0x0000a0 7472 656C 6C69 733D 3120 3878 3864 6374 trellis=1 8x8dct 0x0000b0 3D31 2063 716D 3D30 2064 6561 =1 cqm=0 dea

   0001 0203 0405 0607 0809 0A0B 0C0D 0E0F 0123456789ABCDEF 0x000000

4701 0012 647A 6F6E 653D 3231 2C31 3120 G...dzone=21,11 0x000010 6661 7374 5F70 736B 6970 3D31 2063 6872 fast_pskip=1 chr 0x000020 6F6D 615F 7170 5F6F 6666 7365 743D 2D32 oma_qp_offset=-2 0x000030 2074 6872 6561 6473 3D33 206C 6F6F 6B61 threads=3 looka 0x000040 6865 6164 5F74 6872 6561 6473 3D31 2073 head_threads=1 s 0x000050 6C69 6365 645F 7468 7265 6164 733D 3020 liced_threads=0 0x000060 6E72 3D30 2064 6563 696D 6174 653D 3120 nr=0 decimate=1 0x000070 696E 7465 726C 6163 6564 3D30 2062 6C75 interlaced=0 blu 0x000080 7261 795F 636F 6D70 6174 3D30 2063 6F6E ray_compat=0 con 0x000090 7374 7261 696E 6564 5F69 6E74 7261 3D30 strained_intra=0 0x0000a0 2062 6672 616D 6573 3D33 2062 5F70 7972 bframes=3 b_pyr 0x0000b0 616D 6964 3D32 2062 5F61 6461 amid=2 b_ada

      0001 0203 0405 0607 0809 0A0B 0C0D 0E0F 0123456789ABCDEF

0x000000 4701 0013 7074 3D31 2062 5F62 6961 733D G...pt=1 b_bias= 0x000010 3020 6469 7265 6374 3D31 2077 6569 6768 0 direct=1 weigh 0x000020 7462 3D31 206F 7065 6E5F 676F 703D 3020 tb=1 open_gop=0 0x000030 7765 6967 6874 703D 3220 6B65 7969 6E74 weightp=2 keyint 0x000040 3D35 3020 6B65 7969 6E74 5F6D 696E 3D35 =50 keyint_min=5 0x000050 2073 6365 6E65 6375 743D 3430 2069 6E74 scenecut=40 int 0x000060 7261 5F72 6566 7265 7368 3D30 2072 635F ra_refresh=0 rc_ 0x000070 6C6F 6F6B 6168 6561 643D 3430 2072 633D lookahead=40 rc= 0x000080 6372 6620 6D62 7472 6565 3D31 2063 7266 crf mbtree=1 crf 0x000090 3D32 312E 3020 7163 6F6D 703D 302E 3630 =21.0 qcomp=0.60 0x0000a0 2071 706D 696E 3D30 2071 706D 6178 3D36 qpmin=0 qpmax=6 0x0000b0 3920 7170 7374 6570 3D34 2069 9 qpstep=4 i

    0001 0203 0405 0607 0809 0A0B 0C0D 0E0F 0123456789ABCDEF

0x000000 4701 0014 705F 7261 7469 6F3D 312E 3430 G...p_ratio=1.40 0x000010 2061 713D 313A 312E 3030 0080 0000 0165 aq=1:1.00.....e 0x000020 8884 0067 DBB3 8A4D 4B65 0FFF DF01 BC64 ...g...MKe.....d 0x000030 0BE5 E65E 5002 B515 BE36 21E0 666C FE04 ...^P....6!.fl.. 0x000040 A971 D20D 4000 0003 0063 B4AD 41E6 383E .q..@....c..A.8> 0x000050 E657 7109 309F 9B21 AD69 465B C382 3F5F .Wq.0..!.iF[..?_ 0x000060 A77F 8443 501A 6116 6784 0000 0300 41C0 ...CP.a.g.....A. 0x000070 6181 1905 201D 00B5 0690 3B02 7A1D 014C a... .....;.z..L 0x000080 146D CF41 7312 18BB 19BD A426 9662 9BC2 .m.As......&.b.. 0x000090 E3CD 8211 237D 7CD2 F176 D38C 0AB3 AB6D ....#}|..v.....m 0x0000a0 32FB 96DD 43D5 5C71 F1A5 2264 A9CA 7E2D 2...C.\q.."d..~- 0x0000b0 A72E FFAB 2125 E5CC 8132 93B2 ....!%...2..

So it takes 4 - 5 video TS packets before there's any useful video data !

Looking at x264 source this (surprisingly) doesn't appear configurable - for me it looks more like debug. One either has to comment out the offending line of code in x264 and run/maintain a custom libx264, ignore the warning, or change VDR like so

#define MAX_TS_PACKETS_FOR_VIDEO_FRAME_DETECTION 10

I did the latter and it's not so far having any bad side-effects

Richard

Curious why you're bothering to do this now when 4tb harddrives are always on sale in the $90 range. 6+ years ago it made more sense but now it seems like too little too late. Just wondering....

On Tue, Jun 14, 2016 at 6:21 AM, Richard F lists@keynet-technology.com wrote:

I will do.

Now I'm testing it on a range of recordings, I'm seeing a few issues with ffmpeg/libraries, which I'm addressing with workarounds or reports to the ffmpeg devs.

The aim is to convert all old VDR1.6, and current VDR2.x recordings, including radio, as well automatic transcoding after a recording is made, retaining 100% perceived quality / stream content which is possible with careful codec/parameter selection. Perhaps a plugin one day to save as x264/aac if the stream isn't already in that format, as the space saving can be as much as 50% (e.g. last night an SD recording reduced from 995M to 497M).

Subtitle streams seem to be the largest issue currently - probably not so well tested.

Richard

On 9/06/2016 19:52, Stephan Loescher wrote:

...

Hi Richard,

Would you share your script, after finishing? Would be very interesting!

Greetings, Stephan.

---

vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr

Yup, I see your reasoning behind it. Only one thing though... Transcoding is not going to give you improved quality regarding of what encoder and settings you use. Size can be reduced which is a plus but there's degradation with every trip a stream makes through an encoder. Encoding always has and always will be a one-way street in terms of quality.

On Wed, Jun 15, 2016 at 11:52 AM, Richard F lists@keynet-technology.com wrote:

For my specific situation logic goes like this, FWIW:

• The script started when I was working on the other side of the world a couple of years ago, and was downloading TV regularly. I started by using what others had done and improved/modified for my circumstances. Reducing the recorded size by ~60% into MKV's made this possible. It seemed a relatively short stretch to modify it to save local space, with improved quality as an option as well. It became a seamless reality, and then under real-world testing, a few non-fatal issues came up.

• 4TB disks are USD 150 - the ones I'd buy where I am at least.

• Having lost at least 2 new enterprise class disks in quick succession a few years ago with some valuable data and recordings not backed-up, I wouldn't consider not using a RAID or other reliable storage technique. Hence a RAID upgrade would be USD 450 equivalent.

• I'm at 93% on my 4TB RAID and by reducing this to maybe 60-70% I can see out the disks' lives, perhaps another 2 yrs by which time, based on current trends I expect rotating disks will no longer be required because TCO (considering the need to RAID them + power use over lifetime which is not insignificant) means SSD's will probably be cost effective. So the USD 450 on a new RAID right now would be largely wasted, I plan to wait and go direct to low power non-RAID SSD, which is the holy grail (financially and environmentally).

Make a bit more sense now?

Richard

----- On 14 Jun, 2016, at 15:20, VDR User user.vdr@gmail.com wrote:

...

Curious why you're bothering to do this now when 4tb harddrives are always on sale in the $90 range. 6+ years ago it made more sense but now it seems like too little too late. Just wondering....

On Tue, Jun 14, 2016 at 6:21 AM, Richard F lists@keynet-technology.com wrote:

...

I will do.

Now I'm testing it on a range of recordings, I'm seeing a few issues with ffmpeg/libraries, which I'm addressing with workarounds or reports to the ffmpeg devs.

---

vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr

Stephan - In case you didn't see the announcement on this list, the script and utils are now out there.

Enjoy! Richard

On 9/06/2016 19:52, Stephan Loescher wrote:

Hi Richard,

Would you share your script, after finishing? Would be very interesting!

Greetings, Stephan.

-------- Forwarded Message -------- Subject: vdr-convert released Date: Sun, 4 Sep 2016 22:33:20 +0100 From: Richard F lists@keynet-technology.com Organisation: Keynet Technology To: VDR Mailing List vdr@linuxtv.org

I'd like to announce the release of vdr-convert at https://projects.vdr-developer.org/projects/vdr-convert

One of two people asked me to release it when ready, and after converting / testing most of my 4TB archive with it, I think it's about ready for the VDR community to use.

What is it?

A shell script with associated tools to transcode the content of VDR1.x and VDR2.x recordings as accurately as possible, including all streams, audio, AD, subtitles, and metadata into more compressed format, while maintaining perceived quality. H264 and AAC are the chosen codecs for the main streams. SD recordings are reduced to anywhere from 35% - 90% of original size depending on content and compression settings. On average, you can expect to save 1/3 of the disk space used for SD recordings.

The USP is that this tool may be the only tool capable of converting/compressing old VDR1.x recordings with dvb subtitles into VDR2 compatible .ts files playable on VDR2 and 3rd party players such as Kodi, MPC-HC and VLC.

The script also has ongoing use to automatically compress recordings once complete, or for automatic export, optionally by FTP (I use it this way).

I created a comprehensive WIKI documenting the various usage scenarios and issues found during test. The files are in the Git repository. For VDR1.x conversion, a couple of tools need compiling from source.

Have fun!

Richard