Avainsana: AS3

AS3: Perfect video sync with embedded frame numbers

Flash is still the best tool for online video and there are more and more gorgeous videos where external media like images from user’s Facebook profile are embedded to a video. World famous examples are Tackfilm, Take this lollipop and Lost in val Sinestra.

I participated in a similar campaign and our technical partner had a lot of problems with video sync. How can you tell with 100% reliability which video frame is currently shown? Zeh Fernando writes about possible solutions in this  article. The solutions (and their cons) are:

– embedding the video inside a SWF and getting the current frame of that movieclip (very inconvenient and cumbersome)
– use Netstream’s time property (very inaccurate)
– use NetStream’s decodedFrames + droppedFrames (works until Flash player hangs for one frame or the video is restarted or rewinded)
– check cue points (requires old video codec)

Eric Decker came up with a better solution. Ingenious!  Embed the the frame number to each frame with a barcode. Sounds very tedious, but quite easy task once the ExtendScript and ActionScript are ready. I didn’t find good examples, so I had to make one.

Frame numbers cannot be plain text, because decoding shapes would be too heavy for the CPU (nevermind the coder) and would not be 100% reliable. So the frame numbers are embedded as a binary number, ergo ones and zeros. My solution uses pixels of certain color, because it is very easy to get the color value of a pixel with ActionScript. If the color value exceeds a threshod value, the pixel equals ”1”.  First I tried a simple 1px high row of pixels below  the video, but compression causes pixels to blend with adjacent pixels so that is not even 70% reliable. So the blocks presenting binary value must be larger than 1*1px. In my tests even 3*3px blocks weren’t enough, if the compression is set to medium. 5×5 pixels with 7px gaps between blocks work with 100% reliability. I bet smaller gaps work too, but I didn’t test it.

Compression, and the blending caused by it, also fades the color, so the 100% color value (like 255,0,0 for pure red) cannot be used either. That’s why the AS3 script has a threshold parameter for the color value. If the parameter is 50, colors values decoded as ”1” would be 127 (255*50%) and above. You can easily test which threshold value is the best.

Here’s a screenshot of the video with some explanation:

Demo

The After Effects ExtendScript

I’ve never used After Effects before, but the script is so simple there probably isn’t a simpler way to add a barcode to a video. The script increases the height of the composition by certain amount (block height +bitOffsetFromBottom +bitOffsetFromTop) and adds a solid colored bar in that position. Then the number of frames is calculated from the frame rate and length of the video. That number is converted to binary (eg. 10010011) and one shape is added to the bar for each needed number (8 in this example).

The script adds an expression to each shape which controls the opacity of each shape as the playhead moves. Each shape is linked to a position in the binary number and if the number in that position is ”0”  the opacity is 0 and vice versa.

The AS3 script

After the AE script is run, it shows an alert box with used parameters and  those should be copied to Flash. This way the parameters between AE and AS3 match and you don’t have to set them by hand. Parameters are:

bitWidth: width of the shape
bitHeight: height of the shape
bitSpacing: : spacing between shapes
bgColor: background color
bitColor: shape color, this value is 0, 1 or 2.  0 indicates pure red, 1 pure green and 2 pure blue
bitOffsetFromBottom: shape distance from the bottom
bitOffsetFromTop: shape distance from the top of the background bar

AS3 has also the threshold parameter mentioned above.

It is also quite easy to test if the barcode works. The AS3 script warns if frames are skipped or the calculated frame number is smaller that previous, indicating that the barcode was calculated incorrectly. This is caused by too small shapes for the compression level and the solution is to increase the shape size and/or spacing.

One problem: the height of the video increases and we don’t want to show the binary shapes to the user. The AS3 script includes three solutions for this: mask the video, draw a shape over the barcode or capture each frame as a bitmap and cut off the barcode. Of course you can just make the height of the HTML element smaller and disable scaling.

Here is a demo. Note that I’ve added a text field to the video that displays the current frame as a binary. The AS3 script adds a textfield too that show the binary calculated from the shapes. This is just for debugging purposes and those can be disabled in AE and Flash.

Demo

 

Source files and downloads

Here’s the AE script:


{
// FrameNumberToBinary.jsx
	
    // copyright Niko Helle, nikohelle.net

    
    var bitWidth = 5;   // width of the shape representing binary
    var bitHeight = 8; // height of the shape representing binary
    var bitSpacing = 7; // spacing between shapes
    var bitBGHeight = 0; // do not set, calculated below
    var bitColor = [1,0,0]; /// color of the shape [R,G,B] Use only full color values: [1,0,0] or [0,1,0] or [0,0,1]. Other values will not work.
    var fillColor = [0,0,0]; /// color of the bar [R,G,B]
     var bitOffsetFromTop = 3; // shape distance from the top of the bar
    var bitOffsetFromBottom = 3;  // shape distance from the bottom of the bar
   
	
	function FrameNumbering(thisObj)
	{
        
        //calculate height of the bar
        bitBGHeight = bitOffsetFromTop+bitHeight+bitOffsetFromBottom; 
     
        //get the composition
        var firstComp = app.project.item(1);

        //increase the height by the bar height
        firstComp.height = firstComp.height+bitBGHeight; 

        // check if the there is enough space for the binary numbers. If shapes and spacing are wide, the numbers may not fit the video 
        if(!checkFrames(firstComp)){ 
            return;
        };

        //create the bar
       var bar = firstComp.layers.addSolid(fillColor,"bitBG", firstComp.width, bitBGHeight, 1) 
     
        // reset anchor for easier positioning
       resetAnchor(bar); 
       
       //position the bar to the bottom of the composition
       setPosition(bar,0,firstComp.height-bitBGHeight,false); 
   
        //create shapes
       calcBits(firstComp,bitColor); 
       
        // determine which color is used, and alert it for the AS3, 0=R, 1=G, 2=B. Only solid, full colors can be used!
        if(bitColor[0] == 1) bitColor = 0; 
        else if(bitColor[1] == 1) bitColor = 1;
        else bitColor = 2;       
       
       //alert used parameters so they can be copied to AS3
       
       var alertTxt = "Copy these values to the ActionScript file: \n";
       alertTxt +="_bitWidth="+bitWidth+";\n";
       alertTxt +="_bitHeight="+bitHeight+";\n";
       alertTxt +="_bitSpacing="+bitSpacing+";\n";
       alertTxt +="_bgHeight="+bitBGHeight+";\n";
       alertTxt +="_bitColor="+bitColor+";\n";
       alertTxt +="_bitOffsetFromBottom="+bitOffsetFromBottom+";\n";
       alertTxt +="_bitOffsetFromTop="+bitOffsetFromTop+";\n";
       alertTxt +="_frames="+(firstComp.frameRate*firstComp.duration)+";\n";
       
       alert(alertTxt); 
       
	}
	
    //sets position of an element
    function setPosition(element,x,y,z){ 

        var pos =element.transform.position.value;
        if(x !== false) pos[0] = x;
        if(y !== false) pos[1] = y;
        if(z !== false) pos[2] = z;
        element.transform.position.setValue(pos)

    }  

    // resets anchor to top left corner
    function resetAnchor(element){ 

        element.transform.anchorPoint.setValue([0,0,0])

    }  

    //calculate wether the binary fits the video
     function checkFrames(comp){ 
            
            var fps = comp.frameRate*comp.duration;
            var bitSpace = bitWidth*bitSpacing;
            var binaryLen = fps.toString(2).split("").length;
            bitSpace = bitSpace*binaryLen;
            if(bitSpace > comp.width) {
                alert("The video is not wide enough for the bit size and spacing. Required width is "+bitSpace+"px and video is "+comp.width+"px wide.");
                return false;
             }
            return true;

     }   
 
    //create shapes
    function calcBits(comp,bitColor){ 
        
        var fps = comp.frameRate*comp.duration;
         var bin = "1"+fps.toString(2);
         var bit;
         for(var i =0; i<bin.length;i++){
             createBit(comp,bitColor,i) 
         }   
     } 
 
    //create shape
    function createBit(comp,bitColor,pos){ 
         var gfx = comp.layers.addSolid(bitColor,"bit"+pos, bitWidth, bitHeight, 1)
          resetAnchor(gfx);
         setPosition(gfx,comp.width-(bitWidth+bitSpacing)*(pos)-bitWidth,comp.height-bitHeight-bitOffsetFromBottom,false);
         //add the expression
        var expression = 'bitpos = '+pos+';frame = timeToFrames();frame = parseInt(frame,10).toString(2).split("").reverse().join("");if(frame.length < bitpos) bit = "0";else bit = frame.substr(bitpos,1);if(bit == "1") value = 100;else value = 0;';
         gfx.opacity.expression = expression;
    }    

	//run the code
	FrameNumbering(this);
    

}

Here’s the  AS3:


/*
Capture current frame from video with a barcode
Copyright Niko Helle, nikohelle.net
*/

package 
{

	import com.nhe.video.VideoPlayer;
	import com.nhe.utils.Utils;
	import com.nhe.utils.ContentLoader;
	import flash.display.Sprite;
	import flash.display.BitmapData;
	import flash.display.Bitmap;
	import flash.display.DisplayObject;
	import flash.text.TextField;
    import flash.text.TextFormat;  
    import flash.text.TextFieldAutoSize;  
	import flash.events.Event;
	import flash.geom.*;

	public class Main extends Sprite
	{


		// ignore all variables
		// set them in Main()
		// or copy them from AE's alert box after the ExtendScript is run
		
		private var _videoPlayer:VideoPlayer;
		private var _cl:ContentLoader;
		private var _blocker:Bitmap;
		private var _view:Bitmap;
		private var _bmd:BitmapData;
		private var _byteSlice:Bitmap;
		private var _tf:TextField;
		private var _lastBin:int = -1;
		private var _lastValues:Array = [];
		
		
		private var _threshold:uint = 50;
		private var _bgHeight:uint = 0;
		
		
		private var _frames = 0;
		private var _bitColor:uint = 0;
		private var _bitWidth = 0;
		private var _bitHeight = 0;
		private var _bitSpacing = 0;
		private var _bitOffsetFromBottom = 0;
		private var _bitOffsetFromTop = 0;
		
		
		
		
		public function Main() {
			
			/* Copy below parameters from the AE alert window after you have run the script*/			
			
			_bitWidth=5;
			_bitHeight=8;
			_bitSpacing=7;
			_bgHeight=14;
			_bitColor=0;
			_bitOffsetFromBottom=3;
			_bitOffsetFromTop=3;
			_frames=3600;
			
			/* copy above */
			
			// set the threshold. If the value is 50, pixel with color value of 255*50% ( = 127) or higher is decoded as "1"
			_threshold = 50;
			
			
			// calculate how many bits are used
			_frames = _frames.toString(2).length;	
			
			//create a videoplayer
			_videoPlayer = new VideoPlayer();
			_videoPlayer.addEventListener(VideoPlayer.READY_TO_PLAY,video_READY_TO_PLAY);
			_videoPlayer.addEventListener(VideoPlayer.PLAY_PROGRESS,video_PLAY_PROGRESS);

			// load the video from given path
			_videoPlayer.load("FrameNumberedVideo.flv");
			
			
			//create a contentLoader and load an image with it if you want to test with a screen capture first
			_cl = new ContentLoader();
			_cl.addEventListener(Event.COMPLETE,cl_COMPLETE);
			//_cl.load("screenshot.png");

		}
		
		// video is loaded, add trackers etc
		public function video_READY_TO_PLAY(e:Event){
			
			_videoPlayer.playVideo(1);

			// 3 ways to remove bits from view:
			// 1: addView (copies pixels from the video and never shows the actual video)
			// 2: addBlocker adds a solid colored block over the added pixels
			// 3. addMask masks the video
			
			// add video child ONLY if addView is not used!!!
			addChild(_videoPlayer);
			
			
			//addView(_videoPlayer,_bgHeight);
			//addBlocker(_videoPlayer,_bgHeight,0xffffff);
			//addMask(_videoPlayer,_bgHeight);
			
			// show the calculated binaries and frames number over the video
			addBitTracker(_videoPlayer,_bgHeight);
			
			// show the slice cut from the video that is used for detecting binaries. Slice is placed under the video
			addByteSlice(_videoPlayer);

		}
		
		// image is loaded, calculate bits and frame
		public function cl_COMPLETE(e:Event){
			
			addChild(_cl.content);
			
			//get the slice where bits are drawn and read
			
			slice(_bmd,_cl.content,_bitHeight,_bitOffsetFromBottom);
			
			//read bits
			var pixelBits = readPixels(_bmd,_frames,_bitWidth,_bitSpacing,_bitColor,_threshold);

			//covert to integer
			var frame = parseInt(String(pixelBits), 2);
			
			//trace(bin+"/"+pixelBits );
			
			// see function video_READY_TO_PLAY for explanation about function class below
			//addMask(_cl.content,_bgHeight);
			//addBlocker(_cl.content,_bgHeight,0xffffff);
			addBitTracker(_cl.content,_bgHeight);
			addByteSlice(_cl.content);
			
			// if addByteSlice is run, draw the slice
			if(_byteSlice)_byteSlice.bitmapData = _bmd;
			
			//if addBitTracker is run, show the binary and frame
			if(_tf) _tf.text = "Frame:"+frame+", binary:"+pixelBits
		}


		
		public function video_PLAY_PROGRESS(e:Event){

			//save color values in case error occurs
			_lastValues = [];
			//get the slice where bits are drawn and read
			slice(_bmd,_videoPlayer.videoLoader,_bitHeight,_bitOffsetFromBottom);
			//read bits
			var pixelBits = readPixels(_bmd,_frames,_bitWidth,_bitSpacing,_bitColor,_threshold);
			//covert to integer
			var frame = parseInt(String(pixelBits), 2);
			//trace("frame:"+frame+",pixelBits:"+pixelBits);
			// if frame is smaller that before and error has occured
			
			if(frame < _lastBin) {
				
				trace("##### current frame went backwards from "+_lastBin+" to "+frame);
				
				//stop the video so you can check the pixels and error, comment this out later
				//frame = _lastBin++; //error correction
				_videoPlayer.pause();
				for(var i=0;i<_lastValues.length;i++){
					var px = _lastValues[i];
					trace("color:"+px);
				}
			}
			else if(_lastBin > -1 && Math.abs(_lastBin-frame) != 1){
				//trace("##### current frame skipped frames from "+_lastBin+" to "+frame);
			}
			
			_lastBin = frame;
			
			// if addByteSlice is run, draw the slice
			if(_byteSlice) _byteSlice.bitmapData = _bmd;
			// if addView() is run, draw the video frame to it
			if(_view) drawView(_videoPlayer,_bgHeight);
			//if addBitTracker is run, show the binary and frame
			if(_tf) _tf.text = "Frame:"+frame+", binary:"+pixelBits

			
		}
		
		//mask out the added slice
		public function addMask(obj:DisplayObject,bgh:uint){
			var mask:Sprite = new Sprite();
			mask.graphics.beginFill(0xFF0000);
			mask.graphics.drawRect(0, 0, obj.width, obj.height-bgh);
			mask.x = obj.x;
			mask.y = obj.y;
			
			addChild(mask);
			obj.mask = mask;
			
			
		}
		
		//draw over the added slice
		public function addBlocker(obj:DisplayObject,bgh:uint,color:uint){
			_blocker = new Bitmap();
			_blocker.bitmapData = new BitmapData(obj.width, bgh,false,color);
			
			_blocker.x = obj.x;
			_blocker.y = obj.y+obj.height-bgh;
			
			addChild(_blocker);
			
			
			
		}
		
		//dont show video, copy paste pixels without the added slice
		public function addView(obj:DisplayObject,bgh:uint){
			_view = new Bitmap();
			_view.bitmapData = new BitmapData(obj.width, obj.height-bgh,false,0x000000);

			addChild(_view);

		}
		
		//get new frame as video plays on
		public function drawView(obj:DisplayObject,bgh:uint){

			var mat:Matrix = new Matrix();
			var rect = new Rectangle(0,0,obj.width,obj.height-bgh);

			_view.bitmapData.draw(obj, mat,new ColorTransform(),null,rect,false);

		}
		
		//show the frame and binary over the video
		
		public function addBitTracker(obj:DisplayObject,bgh:uint){
		
			_tf = new TextField();
			var textFormat = new TextFormat();
			textFormat.size = 10;
			textFormat.color = 0xFFFFFF;
			textFormat.font = "Arial";
			textFormat.align = "right";
			
			_tf.autoSize = TextFieldAutoSize.NONE;
            _tf.selectable = false;
			_tf.defaultTextFormat = textFormat;
			_tf.width = 200;
			_tf.height = 40;
			_tf.x = obj.width-_tf.width ;
			_tf.y = obj.height-bgh-_tf.height;
			//_tf.border = true;
			addChild(_tf);
		
		
		}
		
		// show the slice  used for binary calculation
		public function addByteSlice(obj:DisplayObject){
			_byteSlice = new Bitmap();
			addChild(_byteSlice);
			_byteSlice.x = 0;
			_byteSlice.y = obj.height;
			
		}
		
		
		// get the slice  used for binary calculation
		public function slice(bmd:BitmapData,obj:DisplayObject,bh:uint,bob:uint){
			
			if(!bmd) bmd = _bmd = new BitmapData(obj.width,1,false,0x000000);
			
			
			var mat:Matrix = new Matrix();
			
			mat.translate(0,-obj.height+bob+Math.floor(bh/2));
			var rect = new Rectangle(0,0,obj.width,1);

			bmd.draw(obj, mat,new ColorTransform(),null,rect,false);
			
			
			
			
			
			
		}

		// calculation binaries from color values
		public function readPixels(bmd:BitmapData,bits:uint,bw:uint,bs:uint,color:uint,threshold:uint):String{
				var px;
				var bin = "";
				var xpos
				
				for(var i=0; i<bits;i++){
					xpos = bmd.width - Math.ceil(bw/2)-((bw+bs)*i)
					
					px = bmd.getPixel(xpos,0);
					
					
					if(color == 0) px = (px >> 16) & 0xFF // RED
					else if(color == 1) px = (px >> 8) & 0xFF //GREEN
					else px = px & 0xFF //BLUE

					_lastValues.push(px);
					if(px >= 255*threshold/100){
						bmd.setPixel(xpos,0,0xFFFF00);
						bin = "1"+bin;
						
					}
					else {
						bin = "0"+bin;
						bmd.setPixel(xpos,0,0x0000FF);
					}

				}
				
				return bin
		
		}
		
		
		
	}	
	
}

Download AE project and Flash files (videos not included)
Download full source: videos, AE project and Flash files

The video used in the examples is © Canon. You can get the video here.

The workflow to make this work is:

– make a new AE project
– import the video
– run the AE script
– copy parameters to Flash
– render the video
– insert the file path to Flash
– Publish Flash and watch the Flash decode the frame numbers in real time
– check if Flash traces any warnings
– if there are warnings, adjust shape size and height and run the AE script again

Have fun!

CSS sprites with AS3

CSS sprites is a very convenient way to use images. Graphics are placed into one file and CSS defines which part of the image is shown on mouse over and mouse click. One image could hold all graphics used on a website.

Flash has skins for components and the SimpleButton class is quite close to CSS Sprites. I wanted a simpler way to create buttons and use graphics. My IconButton class extends the SimpleButton and slices the given image and sets those slices as upState, overState and downState. HitState is always the upState. IconButton has one extra state: disabled.

MultiIconButton is an extended IconButton. Like its name implies, it has multiple IconButtons. Well, code-wise it has only one, but it has different phases so one image can have multiple different icons for various uses. The illustration below explains the principle. Jatka lukemista ”CSS sprites with AS3”

The simplest way to send an image from Flash to PHP without user interaction

With the FileReference class, it is quite easy to send an image to the server. But it requires user interaction. What if you need to send an image the user has drawn or captured form the webcam? Then send the data as a ByteArray and let the PHP save the raw data.

The bitmapdata of the image must be encoded to jpg or png and Adobe has classes for that: com.adobe.images.PNGEncoder and com.adobe.images.JPGEncoder. The contentType of the request must be ”application/octet-stream” and if you want to send parameters (file format, username, etc.), add them to the url as parameters: ”simpleSaveImage.php?fileformat=png&filename=cheetach.png”.

In the source files I’m using my ContentLoader to send the data. The sendBytes method  handles everything for you, but here’s also a summary of how to do it without the ContentLoader:
Jatka lukemista ”The simplest way to send an image from Flash to PHP without user interaction”

Versatile content / media loader: ContentLoader.as

ActionScript 3 introduced new ways to load content and new types of content – now Flash can handle raw bytedata. This also increased the complexity of the process. You need a Loader, URLLoader, URLRequest, event listeners…

ContentLoader is a class that simplifies the process into a one class that can fulfill most of the basic needs:

– send and load data

– load bytedata

– load variables

– hande errors

– load images, mp3s, xml, text (JSON too, of course, but parsing is not supported)

Example: Jatka lukemista ”Versatile content / media loader: ContentLoader.as”

Loading and handling external files in Flash: ExternalAssets.as

Here’s another helper class that loads one or multiple file(s) separately or all in a batch. There is one special feature: this class can check the filesize of each file (without fully loading them) and therefore it can show the loading progress of the whole batch not just single file. The loaded data is stored so the file or its size is never loaded twice.

This class uses my ContentLoader as the actual file loader. Read more about the ContentLoader »

Example: Jatka lukemista ”Loading and handling external files in Flash: ExternalAssets.as”

Loading an SWF and getting components from its library

Keeping all assets in one SWF causes problems while updating assests and compiling times are longer. Nowadays only crazy people add everything into a one SWF and the Flash platform has improved a lot to support alternative ways to embed, load and add assets.

The flash.utils.getDefinitionByName allows to load assets from the library of another SWF-file. I’ve simplified the process in my Utils.as and here is an example how to use it.

Main thing is to use the ApplicationDomain to keep the loaded SWF in its own domain. ApplicationDomain allows multiple definitions of the same class to exist and allow children to reuse parent definitions.
Jatka lukemista ”Loading an SWF and getting components from its library”

The ultimate VideoLoader with rtmp support

One of the best Flash production houses in Helsinki/Finland made a quite surprising comment while making a project with us: their own Flash video player can’t handle live streams. That functionality is very easily added and I’ve used my VideoLoader class (now version 4.X) in various video projects over the years.

Maybe someone else is missing a versatile video loader too, so here’s one. And I say Loader, because this class loads, plays, pauses, buffers, sends events, handles errors and does everything you need for handling video files, but it does not have any UI. But that’s what extends VideoLoader is for! Jatka lukemista ”The ultimate VideoLoader with rtmp support”

AS3 MagicWand

In one concept for a webshop I designed a tool where the user could fit clothes on her own image. One problem – images are not transparent, there was always a background, but usually a solid color. Well, the magic wand in Photoshop would be useful, but too many images to go through then and everytime new items are added. So I studied if Flash could be used and voilá: the MagicWand.as was created. Jatka lukemista ”AS3 MagicWand”

Ignoring black frames while starting a webcam

While playing with motion detection with a Webcam, I ran into a surprising problem. First n frames are 100% black. The number of frames vary, so this problem wasn’t solved with ignoring first 10 frames or anything similar. So I used getColorBoundsRect to detect any non-black pixels and when found, the WebCam class would start to send an event. The video will be visible before dispatching events, but it could be hidden until the event is dispatched. Jatka lukemista ”Ignoring black frames while starting a webcam”

My new best friend – describeType

Praise to the flash.utils.describeType – a method that became my number one friend when I was validating parameters set by user and solving how to validate proper data types without knowing what they were.

So, what does flash.utils.describeType do? In a nutshell it converts any class into a xml that describes