VLC and Wowza sitting in a tree…

…. K-I-S-S-I-N-G.. 🙂

So as I’ve told you before, it’s been a busy few months and a lot has been going on. Currently working on a project for a client where we need to capture Multicasted IPTV Header, transcode it and then broadcast it to mobile terminals (iPhone’s, Android’s and older terminals as well – 3gpp).

It’s been an interesting time trying to figure out how to get VLC to jump through hoops and make it work together with Wowza, without getting a lot of qualityloss on the way.
During the testingphase of the project, about the same time as I was testing Wowza Media Server 3 Preview, we tried out the transcoding addon from Wowza. It might have gotten better in handling stuff since the preview I did but since we planned on being done BEFORE the release of Wowza Media Server 3 – we decided against using it.

Enter VLC Media Player

My readers from before will know that I’m sort of a *nix-nerd and as such I ‘refuse’ to use other operatingsystems if I can help it. So the operating systems on the below setup is CentOS 6 straight through. And after a few hours, more like a day, to get VLC to compile into the version I needed –  I actually gave up and went out hunting. And found, precompiled, versions of what I needed out there on the net. Installed and whammo.

Running VLC from commandline with a ‘cfg’-file is an interesting thing. Documentation, that I could find, are scarse. I’ll give you an example below of how I configured one of the channels, and told VLC to pick up the multicasted ts-stream and then send it on to Wowza.
[pre_code]
new channel1 broadcast enabled
setup channel1 option program=480
setup channel1 input “udp://@127.0.0.1:1112″
setup channel1 output #duplicate{dst=”transcode{venc=x264
{keyint=60,profile=baseline,level=3.0,nocabac},vfilter=
canvas{width=480,height=270,aspect=16:9},vcodec=x264,
vb=416,scale=1.0,acodec=mp4a,ab=64,channels=1,samplerate=
44100}”,select=”program=480″}:duplicate{dst=rtp{mux=ts,
dst=172.16.1.2,port=40010,ttl=3},select=”program=480″}
control channel1 play
[/pre_code]

So what am I doing? In short… setting up a new channel, telling the channel that multicast programID we’re looking for is 480 and that the multicast is at localhost UDP port 1112. The output should be double, in effect, I want to transcode it and then send the transcoded material on to another machine and a certain port on that machine. The I tell it to start doing it’s job. Then rinse and repeat, in this case, 7 times with double encodings on both.. so the above, 14 times per encoder.

If we put it mildly, the processing-power of these machines are nice. Dual processors with 12 cores each and 24 gigs of ram. Still.. the VLC-process gulps up about 600% of the CPU.. which looks weirder than it is.. the load? about 3 when humming along…

Wowza backend configuration

The “backend”-Wowza server is roughly half in size and processingpower than the encoders (and the frontend wowza). This is mainly due to the fact that it doesn’t get very much to do. It receives all the encoded material on it’s different ports. I’ve set it up as a liveorigin, this is mainly because I want the frontend to be scalable in an easy fashion. Adding more hardware and frontend streamingservers is quite easy if you use Wowza’s own loadbalancing solution for it. That’s the main reason why I set up origin-edge from scratch. With the start of the service I could probably have survived most of the traffic on this machine alone – but client wants scalable, client gets scalable. The backend-wowza is ONLY available on the internal network in this setup. The only port available to the outside is the streammanager, so we can reset streams if needed – without restart Wowza completely.

Wowza frontend configuration

A workhorse to say the least. Expandable hardware is nice. It’s basically a 2U version of the encoders and have 24gigs of ram and the same processor and core-count. I’ve even milked the support-personnel at Wowza for optimizing this setup so it gets as good as I can without overdoing it.

Example: I have 24gigs of ram. Wowza Support doesn’t recommend giving Wowza itself more than 8gig. Why? Because then Garbage Collection on the javaside starts taking too long and that hampers performance. But it’s basically doing a liveedge setup, times two since we’ve had to implement and old workaround (feature) in Wowza that makes it possible to change mp4 (android/iphone) to 3gpp (mp4latm) instead, for the ooooold terminals. How I wish that people would kick their old phones to the curb and get new ones. Would surely make my life a lot easier.

This machine is currently the only one available to the public.

Endpoint?

Well.. as usual, when it comes down to projects I’m involved in, the product isn’t available outside of Sweden and not outside of a certain cell network in Sweden. So I can’t show you anything worthwhile ‘live’ and direct. If you are interested in seeing the end product… what you can do is ask me and I can, perhaps, provide you with a temporary link that you can try out.

I can, however, show you a screencapture from my iPhone 4 (the old one, not the new and improved one)

And the only thing I’ve done is to hide the channel and the subtitling – I’m sure someone can decode what channel and program it is anyways but I’ve tried at least. The image actually looks better on my iPhone than it does in the “real world”.

If you have any questions in regards to this setup, don’t hesitate to ask – if the questions gets too intricate then we can discuss the consultingfees 🙂

Wowza Media Server 3.0 (preview)

So. I’ve spent a few hours installing a Wowza Media Server 3.0 Preview Release. Wowza themselves calls it “Any Screen Done Right™” which sound very promising for me with the type of work I’ve done for the past few years – Haven’t tested it much yet but on the “paper” it’s look pretty darn good. The current project, I’m involved in, is to port (livetranscode) TV-channels to mobile devices (phones and pda’s and such). This is where one of the new features of the coming Wowza Media Server 3 is coming in.

I’m talking about the Wowza Transcoder Addon.

According to the release notes this supports adaptive bitrate (ABR) for Flash, Silverlight, and Apple HLS. It should be able to ingest from live encoders, IP cams, TV headends, and more…

It can handle the following inputs – MPEG-2, MPEG-4 Part 2 and H.264 the output will then become H.264 over any protocol supported by Wowza Media Server. They will also be providing pre-built profiles for web, mobile and IPTV-streaming. The second part is what’s interesting me at this point.

Currently the system support GPU-acceleration hardware using Intel’s QuickSync or Nvidias CUDA, on windows only, but they’re aiming to offer at least Nvidia CUDA support in Linux as well.  My current configuration is being run on a Linux server – so at this point I will not have any GPU-acceleration and have to rely on the normal processors in the server to handle that. But then again the server is running with 12 cores and 24 gig ram… so we will be stress-testing the transcoder quite harshly – to see how many transcoders we can run on one hardware and still get good results.

As you can see I’m using “we will” – and that’s because we’re still waiting for the input (source’s) to tell us how they will be providing us with the material (hopefully something that doesn’t need to much configuring on our end).

Apart from the Wowza Transcoder Addon the other “MAIN” news are:

Wowza Network DVR (nDVR) AddOn

A thing that will be interesting to try, although I can’t see the use for it on my end (yet). A Single nDVR cache — with support for pausing, resuming and rewinding a live stream. It also integrates with the Transcoder for time-shifted streaming and comes with API for customization. For the final release they’re also planning multiple bitrate support and multiple server support. Could be interesting to try – but I’d have to wait a while, at least until my transcoding tests are done.

When you’re working with IP-holders they do have a tendency to obsess over security in general and DRM especially. I have a mantra, that I always tell clients when they bring up the question “what about people copying the content – they can’t do that can they?”, and that mantra is ‘if you can view/listen to the content on a computer you can record it’ – even if it comes down to placing a camera in front of the screen and pluggin a recording device into the soundcard – it’s always possible if you really, really want to do it. But that being said, you don’t have to make it easier for them. Thus we end up with the third and for some, most important, addon in the new Wowza Media Server 3.

Wowza DRM AddOn

This addon is offering integration 3rd party DRM Key Management Systems, including:

  • Verimatrix VCAS™ for playback on HLS devices with ViewRight™ clients
  • Microsoft® PlayReady® for Silverlight smooth streaming client playback (not current but for planned for final release)

According to Wowza they will be announcing other platforms in the future. Suffice to say the above mentioned are good enough but the more the merrier. As I’ve said, I’m no big fan of DRM since it only adds more load to servers but if clients demand it – the more support, the better. This addon supports on-the-fly encryption for live and VOD. Specifically for live its encryption per stream with the ability to rotate keys and for VoD it’s instead per asset or per session with the same abilities.

The only “issue” I’ve found, yet, is that neither GPU-support, nor Transcoder-addon will work on Mac OS X – and being a iPerson – I don’t really like that, hopefully it will come something to those platforms as well…. Another thing, not an issue, is that the Wowza Media Server 3 will have free upgrade for those on the Wowza Media Server 2 platform, which of course is good. However, the addons mentioned above – will be separately priced and the price will be made public at a later date.

Now, Wowza Media Server, in my opinion is one of the most friendly in terms of cost for a software of its caliber but these addons could turn that around. I hope it doesn’t but we will have to wait and see.

I tried to get the Wowza-people to give me a few hints on what’s coming post-3.0 but only got a “no, we don’t discuss that” – which I have to say isn’t that surprising. But I had to ask!

Using JotForm to Receive Submissions

Not so much a streamingpost, more on the subject of web development using services in the “cloud”.

I have for the past four or five years had the privilege to work with a swedish radio organisation heading up their website in general and the yearly awards-nominations specifically and it’s been a hassle with a lot of manual labour, but fun also (we get to go to the awards). But, suffice to say, I have also had opportunities to improve the process yearly with new functions available on the internet in many forms.

No pun intended but this year it was just that.  Forms.

Last year I moved the submissions from sound files/CD sent via email or regular snail mail to taking them in via Dropbox. After that I still had to do them manually to keep track of who had nominated who, and report back to the jurygeneral. Even though I didn’t need to actually encode the files anymore the workload was a bit too heavy for the “pay” (we’re sponsors of the event and my time is “free”).

So, into the picture comes a new jurygeneral (two actually) and I still wanted to use Dropbox but I wanted the submissions to be a bit more easily administered for both me and the jurygenerals. Sadly the website is hosted on cheap web hosting and that limits the functions I can use on the website (restrictions). So off I went hunting for something worthwhile using in the cloud. I found JotForm.

The user interface of JotForm makes it extremely easy to create and manage web forms with automatic upload into your Dropbox, if wanted. So, I took the normal word-document form for nominations and converted it to a JotForm which I then incorporated into the website (swedish only) with the last response being files. Why? Well, people are nominating radio shows to different categories, thus it would seem fitting to allow them to send in mp3-files that the jury can judge their different performances on.

For the most part this setup has worked, more or less, flawlessly. I’ve had some small “outages” in the service which made the users “yell” at me, of course. But I can still recommend the service of JotForm. And I would like to be able to translate all buttons and form parts to swedish – now I can’t find a way to make the “upload files” in swedish – that annoys me. I have but one true issue with JotForm. The swedish translation of the general text on the site (not the formbuilder) is hilarious. My suggestion, to JotForm, would be to not use google translate to translate general text to swedish because now – it’s a joke 🙂

That’s it for today. Next week I’ll be talking about Wowza 3.0 (prel).

Amazon Live Streaming and CloudFront

The weekend that passed marked the première for me to use Amazon Web Services as a streaming solution for a live broadcast. No, don’t get me wrong, I’ve done it using EC2 before – several times – but in this case I had to use CloudFormation and because of that it was a première.

The setup is pretty straightforward.

So here we go using Amazons pre-configured CloudFormation template. During about 10-15 minutes it automates a setup of CloudFront distribution as well as the EC2 instance needed to serve it. Since this current client was estimating about 1000+ concurrent viewers and they where running 1,2 Mbit/s video stream – and Amazon has a 1Gbit/s limit – I had to contact Amazon and ask them to raise the limit to 2Gbit/s, a request they happily obliged.

The basic setup of how it works is included here. Since the only way of doing live streaming via cloudfront, as far as I know, is to use HTTP-live-streaming instead of standard RTMP-streaming this is of course what we used. Amazon has an excellent article describing exactly how to get it set up so it’s not what I would consider a case of “rocket science”.

So, how did it go. Well – we found a few minor bugs – these are the issues we found during our two broadcasts this weekend:

  1. Minor buffering and drops – after checking this seemed to be an issue with caching on the cloudfront (we’ve reported it to Amazon). When a package (part of the live stream) was missing in the cache on cloudfront the time it took to retrieve it from the server was long.
  2. The cache, default, is way too long. Amazon has a 24 hour period for caching, after last access, on CloudFront. This gave us weird problems since live streaming is live. When we stopped encoding and then restarted CloudFront started serving us old content. We resolved it by killing our CloudFormation and started a new instance, thus getting a new CloudFront hostname and an empty cache. (we’ve reported this to Amazon as well).

One minor, economic, issue is that the FMS has either a 1000 concurrent viewers limit or, the next, 10.000 concurrent viewers. Since we where expecting 1000+ we had to go with the 10.000 version. But, what I haven’t gotten a response to as of yet, it would seem that the server only had ONE connection during the broadcasting (CloudFront presumably).

That said, it is a problem that I have to discuss for our next broadcast. As well as getting the small buffering and drops out of the system if possible.

Conclusion: Using Amazon for Live Streaming is quite good – It isn’t free but you can get “global coverage” with only one EC2 instance by using CloudFront for distribution. So, in short, I can’t complain and it was easily set up and the people at Amazon that I’ve been in contact with have been more than helpful. (Thanks Brad).

25 years of radio

A fuzzy fourteen year old with a passion for listening to radio steps into a community radio studio in Sollentuna 25 years ago. The journey from there to the present day has been long and turbulent. It has gone from simple analogue to a now almost all-digital production of ‘radio’.

I put ‘radio’ inside” to today’s radio can hardly be in any way resemble what I did in a community radio studio 25 years ago. Not that I would probably call it high-quality radio in any way but ratherattempt to do so, with very limited resources at that time. What made it possible, then and now is a passion for the medium radio.

From mono mixers, tape decks, turntables and a microphone to a fully digital environment that can run entirely on my laptop. Despite the production-related differences, it is well, in Sweden, most market has changed. If it’s for the better is something we can talk about for many long hours so I will drop it and stick to that which is closest to my heart and it’s technology in general and to make ‘quality radio’ with limited resources, in particular.

Not necessarily big, massive, commercial local radio, but rather small, tight, interested, focused event based one.

1998 I launched my “event radio ‘career as a colleague at my former employer went to my boss and asked to borrow a technician for a military exercise on the island of Gotland. A radio station was built at the former Military Command Gotland where I as a technician and Rolf Arsenius as host / reporter and technology borrowed from a variety of sites produced radio around the clock with features, interviews and reports – mostly to inform people of Gotland on what was going on. A long week and two people, we were a bit tired after the run – but it was a massive success – we had news broadcasts in seven languages ​​(it was after all an international exercise and we had some press officers from other countries come and run ‘radio news’ to their soldiers).

After the exercise, this has been repeated countless times in various forms and radio is necessarily a fast media to reach out to a targeted audience seeking information or entertainment in any form.

One of our expeditions led us to the city of Boden in the far north of Sweden and the locals told us that we should be doing radio professionally. They did not know that, when we did not wear uniforms, we did just that.

These broadcasts have always been analogue, although we used a computer program for playing music and schedule elements that have been pre-made. We used cell phones for live interviews and on one occasion, I had armored engineers rebuild a throat microphone so we could report on traffic from one of the Defence Forces helicopters – mmm .. they are not quite as nicely isolated and private options, so the sound went into our microphones pretty hard.

Today, I see a lot of opportunities with radio, but you’d have expand it into a form of web TV at the same time. Yeah, if you’d like me worked in radio long enough, you’ll also have to get into images in some form when working electronically. The possibilities are almost endless when you can with low productioncosts create something that makes it possible for millions of people (potential) to take part in.

I will use my blog to talk about this stuff, we’ll see if I can keep going or not, but this is at least an honest attempt. I have earlier in my life called me radiot (radio idiot) and that’s what I am basically. Now I guess I’ll start calling me mediot instead – because it is not only the spoken word that will be used and discussed here, but also still and moving images.