Reading this made me wonder. As I understand it most common compression schemes are either lossy or lossless and in video compression schemes run scene by scene. Thats what it indicates in that article. The term scene is a bit ambiguous to me.

My first thought is that they meant frame. Which got me thinking, I'm not an expert on video compression but has anyone actually investigated the potential of compressing multiple frames at a time based on frame content?

By this I mean that as I understand it, you look for "patterns" of data, whether colour based data, or perhaps ascii data, or even curves, and with those patterns noticed you simplify that into a stream of data that can be at best unambiguously decoded, at worst can be approximated. But what about processing several images in almost a 3D manner? Each frame layed over the other and perhaps merging several "scenes" with similar backgrounds or facial curves whilst and looking for a pattern in those to simplify it some more.

It seems pretty obvious to me which makes me think it's probably already done. If anyone already knows if its done please contact me, I'd love to learn a bit more. However if it's not already done. Why the hell not?

** Edit **
I spoke to a friend about this article, this is what I learnt.

* bryno knows a little on video compression :]
<bryno> And yea, there are some compressions that encode with patterns :P
<kay> over multiple frames? like how i was describing?
<bryno> Not just encoding frame by frame, but motion going on between frames
<bryno> Yea
<kay> do u know the names? id like to read about them
<bryno> I know mpeg does
<bryno> Like it's got 3 different frame types, and certain types depend on the other frames in the "group"
<kay> is mpeg openly documented?
<bryno> I'm not sure if the actual standard is available, but you can certainly find info
<kay> awesome
<bryno> Like iirc, mpeg uses the DCT to do lossy compression. Part of the video contains the matrix needed to do the transform, and the book just states how that is stored

** End Edit **

The article itself discusses the concept of getting HD (high definition) video content to be sent over the Internet at comfortable speeds to be used over commercial networks (for reasons why compression would be nessicary for this, see this wired.com article). see also Do Gamers really need HDTV?

I think we all saw this coming. It's really only ever been a matter of time, and to tell the truth, I'm glad it is! I'm a strong believer that one day it would be great to have the Internet replacing analog TV signals, and perhaps even radio by means of a national wireless net. Integration of information systems into our daily lives like traffic reports, GPS, warnings, user-customised news, customisable radio stations. For imagery purposes imagine all that in your car on your way to work. Or in your personal headset as you walk to school.

I'm sure google would kick ass in such a world where they can touch their customers directly, or selling off algorithms for guessing the tastes of customers. Anyone reminded of Minority report from what I'm saying? Seems reasonable to me, even if still far fetched.

I think the flexibility and potential by integrating all these types of services into a universal communication medium is incredible compared to current media. With standardising most of our communications to a single standard it allows easy extensibility and diversity amongst companies.

Hell, it's not hard to start you're own radio station online, or even a TV station! Just look at revision3 and how popular that has gotten. RSS allows people to receive content they want as soon as its available, pod casting is another step towards where I think technology is heading. Competitiveness with soon be all about quality and tailor-made content, for the same price as regular content we have at the moment.

And don't scream "but we already have this through cable and satellite for years!" sure, ran by large corporations who decide whats good and whats bad by market trends, rather than the user deciding.

Which brings another prospective, complete interactivity. With the recent boom in digital television over the past few years is a shining example of what I'm saying, they can cram more features into a single signal by way of protocols and standardisation of technology.

Okay well I think this post was more of a rant than an objective look at things, I did not research before posting for which I'm sorry, I have an active imagination. Isn't it inspiring though? To wonder where the future is heading? I guess those sort of articles often invoke this feeling in me. Does anyone get the same feeling as me? Excitement.

Introduction
The scenario is you are without Internet connectivity anywhere. You have found either an open wireless access pointed or perhaps you're staying in a hotel which permits rented Internet via services like Spectrum Interactive [1] (previously known as UKExplorer). You make the connection, whether its physically connecting the Ethernet cables, or instructing you're wireless adapter to lock onto the radio signal. You are prompted with some sort of authorization page when you open a browser. You don't have access to it, so what do you do?

Open a command prompt or terminal and ping an IP you know to be up, active and that you know to be not physically connected to the network you are jacking into. Perhaps even just ping google.com

[16:03] kay@client.example.com:~$ ping google.com
PING google.com (64.233.187.99) 56(84) bytes of data.
64 bytes from 64.233.187.99: icmp_seq=1 ttl=237 time=196 ms
64 bytes from 64.233.187.99: icmp_seq=2 ttl=237 time=164 ms
64 bytes from 64.233.187.99: icmp_seq=3 ttl=237 time=152 ms
64 bytes from 64.233.187.99: icmp_seq=4 ttl=237 time=153 ms

Right, so we're connected, and we can use ICMP ping requests. We are now in a position to "steal Internet". Now I do not promote any sort of actual illegal activity, but since these networks are "open" to us and acting as a service provider freely with no warnings of what we're doing could be considered unauthorized usage. Since ICMP works we can only assume that we're permitted to use it right? I don't see why they would allow this if they considered it unauthorized without notifying us beforehand.

For those that don't actually know what all this mention of ICMP is, I suggest you google it, or check out the wikipedia article on ICMP [2]

Building the tunnel
To set up the tunnel you need:
One up system with Internet connectivity that can receive ICMP traffic and make outbound TCP connections. This system we will call the proxy.example.com.
You (preferably) hold root or administrator level access on the proxy system.
You (preferably) hold root or administrator level access on your local system
A copy of PingTunnel [3] by By Daniel Stødle

The first step is simply to install PingTunnel. This is a extremely easy installation.

[16:15] kay@proxy.example.com:~$ wget http://www.cs.uit.no/~daniels/PingTunnel/PingTunnel-0.61.tar.gz
--16:15:49-- http://www.cs.uit.no/~daniels/PingTunnel/PingTunnel-0.61.tar.gz
=> `PingTunnel-0.61.tar.gz'
Resolving www.cs.uit.no... 129.242.16.40
Connecting to www.cs.uit.no|129.242.16.40|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 53,433 (52K) [application/x-gzip]

100%[====================================>] 53,433 14.12K/s ETA 00:00

16:15:56 (14.11 KB/s) - `PingTunnel-0.61.tar.gz' saved [53433/53433]

[16:15] kay@proxy.example.com:~$ tar xzf PingTunnel-0.61.tar.gz
[16:16] kay@proxy.example.com:~$ cd PingTunnel
[16:16] kay@proxy.example.com:~/PingTunnel$ make
gcc -Wall -g -MM *.c > .depend
gcc -Wall -g -c -o ptunnel.o ptunnel.c
gcc -Wall -g -c -o md5.o md5.c
gcc -o ptunnel ptunnel.o md5.o -lpthread -lpcap
[16:16] kay@proxy.example.com:~/PingTunnel$

Do this on both the proxy system and your local system.

On windows there are some ported copies of PingTunnel which are unsupported by the original creator, but I have used them and can they worked fine for me. You can find them here. One of the packages I believe should be ready for compiling with Microsoft Visual C++, the other is certainly ready for compiling with the windows gcc.

You will need the pcap library [4] to compile PingTunnel. For instructions on the libpcap installation see the appropriate manual. If you don't feel like going through all the trouble of compiling this, there should be an executable binary within one of the packages precompiled that you can just run.

Making a tunnel
On the proxy system, obtain administrator level privileges, and run PingTunnel as below

[16:16] root@proxy.example.com:~/PingTunnel$ ./ptunnel -f ping.log &
[16:16] root@proxy.example.com:~/PingTunnel$

This will run ptunnel in proxy mode and will log it's output to a file called ping.log rather than print it on stdout.

On the client system, obtain administrator level privileges again and run PingTunnel as below, lets assume we wanted to ssh to one of our servers, lets call it portal.example.com.

[16:16] root@client.example.com:~/PingTunnel$ ./ptunnel -p proxy.example.com -lp 8765 -da portal.example.com -dp 22 &
[inf]: Starting ptunnel v 0.60.
[inf]: (c) 2004-2005 Daniel Stoedle, daniels@cs.uit.no
[inf]: Relaying packets from incoming TCP streams.
[1] 10858
[16:17] root@client.example.com:~/PingTunnel$

Traveling through the tunnel
We now have a tunnel set up. So lets use it! To tell programs to use our tunnel we simply tell them to connect to port 8765 on the localhost. So for example

[16:16] kay@client.example.com:~$ ssh localhost -p 8765
The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is b7:2b:f3:3d:08:86:20:55:1d:82:08:2b:d0:33:fe:af.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Linux portal.example.com 2.4.27-0.3um #1 Thu Sep 2 11:39:16 GMT 2004 i686
root@portal.example.com:~#

Hurray, light at the end of the tunnel. We've used ICMP to make a proxied TCP connection to the outside world. We can use the same principle for any TCP based service.

Tunnel authentication
Potential problem. Our proxy end of the tunnel is available to be used by absolutely anyone. It has no authentication mechanism to permit only us to use it for tunneling. After all we don't want to get busted for sending spam because some jerk used our tunnel for his own illegal activity. So we need to secure it. The PingTunnel program provides a simple authentication option that allows us to set a password for our tunnel. Its not solid security but it's enough to keep our the majority of people. Especially since if we did a standard scan on our proxy we wouldn't see the ICMP daemon anyway.

To set a password, when you run the proxy end, use the -x switch like this.

[16:16] root@proxy.example.com:~/PingTunnel$ ./ptunnel -x ubersecret -f ping.log &
[16:16] root@proxy.example.com:~/PingTunnel$

Then when our client requests a tunnel we run it like so.

[16:16] root@client.example.com:~/PingTunnel$ ./ptunnel -x ubersecret -p proxy.example.com -lp 8765 -da portal.example.com -dp 22 &
[inf]: Starting ptunnel v 0.60.
[inf]: (c) 2004-2005 Daniel Stoedle, daniels@cs.uit.no
[inf]: Relaying packets from incoming TCP streams.
[1] 10858
[16:17] root@client.example.com:~/PingTunnel$

There you have it. Free Internet, no authentication needed with the service provider you are jacking into provided as they permit ICMP traffic.

Limitations.
There is some limitations however, the functionality of web browsers is limited due to having to make several connections to different places for a single page. The HTML will not indicate it should be passed through localhost. In HTTP 1.1 the Host header must be passed, most clients will pass the host of whatever domain they are currently connecting to, so you will have requests for host: localhost, which most probably wont exist on the web servers configuration and will just be rejected.

How it works
Enough of all this tutorial-like talk! Let's explain how it works. First of its important to know that ICMP is part of the IP suite of protocols. It's also important to be aware that like UDP, ICMP works on a datagram basis with no sort of transmission control or insurance of delivery. Finally its very important to note that TCP does have transmission control and mechanisms to ensure delivery or be alerted to a failure.

Lets take a look at the ICMP packet layout for echo requests and echo replies.

    0                   1                   2                   3

    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

   +-------------------------------+-------------------------------+

   |     Type      |     Code      |          Checksum             |

   +-------------------------------+-------------------------------+

   |           Identifier          |        Sequence Number        |

   +-------------------------------+-------------------------------+

   |                      Data ...

   +---------------------------------------------

The type field specifics what the packet is for, since ICMP is used for more than simple ping and ping replies there are many values, but for us we only need to be aware of two types. Type 8 tells us the packet is an echo request. Type 0 tells us its a echo reply.

The code field is for more information about the nature of the packet, but for both echo reply and echo request the code field will always be 0. If you want to know more about what the code field is used for see the ICMP RFC [5]

The checksum is a 16-bit one's complement of the one's complement sum of the ICMP message beginning with the ICMP type.

The identifier is used when the code field is 0, for echo requests & replies. Its used to help match requests & replies. This is often some information about the session, perhaps a PID or similar.

The sequence number, like the identifier is used as a to track which reply matches up with which request by way of a numerical value. This should be unique to all outstanding ICMP traffic. The RFC does not explicitly specify this. However if the ICMP implementation has a ambiguous identifier field and duplicate sequence numbers in outstanding, traffic could end up in confusion.

The data field of the request contains data to be copied into the replies data field. This could be used to add yet another way to identify request session, but in the case of the PingTunnel, this is one of the most important fields. This is where we put our ping tunnel protocol data into, which will include the TCP data.

For an overview of the sequence of packets during standard operations such as pinging hosts, or traceroutes this pdf document [6] gives an excellent brief.

So thats the background knowledge. The client will always be using type 8 request packets for all its traffic, the proxy will always use type 0 reply packets for all its traffic. The reason for this is simple routing. Say that the client was behind a router the router's operating system may itself reply to the proxies echo requests without forwarding them to a host within its network that is involved in the ICMP tunnel session. This also brings up an important point about the initial setup of the ICMP proxy host. It must be able to receive echo requests itself. This might make the host system a router or simply have all packets for the proxy IP routed directly to it.

The ping tunnel protocol is similar to TCP [7], stuffed ICMP data field with an extra field included. The extra field is called by the creator of PingTunnel, the "magic number" in simple terms its just an identifier to separate the echo traffic from regular echo traffic. Since the proxy host operating system should always reply with a regular echo reply to each request packet we send.

The internals of the ping tunnel protocol (no byte scaling).

  +----------------+------+--------+---------+-------------------+

  |  Magic Number  |  IP  |  Port  |  State  |   Acknowledgment  |

  +---------+------+------+--+-----+------+--+-------------------+

  | Length  |  Sequence No.  |  Reserved  |        Data ...

  +---------+----------------+------------+----------------------

The IP and port fields are purely used during the first echo request from the client to the proxy to setup where the proxy should forward received packets. These are both 32 bits wide.

The state field is crucial in the TCP-like control of the protocol. It shows what sort of semantics should be understood from the packet. Such as starting a session (state = 0), ending it (state = 3), authentication (state = 4), data forwarding (state = 1), and the TCP-like acknowledge data (state = 2). It also contains a flag indicating who send the packet, proxy or client. This is used with the magic number to differentiate tunnel ICMP traffic from the redundant echo replies that the proxy will make - since the proxy reply will contain identical data to the request, this will tell the client to discard packets claiming to be from the client, but are actually from the proxy. When the left most bit is set the proxy is sending the data, when the left most but one bit is set the client is sending the data. The state field is 32 bit wide. For example say state is 0x80000004 then we can see that the proxy is forward some received data back to the client.

The acknowledgment and sequence number fields are used to ensure delivery in the correct order and also to detect when a packet could get lost in the network cloud between the client and proxy. These are very much like the TCP fields of the same name, except the sequence number here is a 16 bit number thats allowed to wrap around just before overflow. The acknowledgment field is 32 bit wide, like it is in TCP. As the conversation progresses the sequence number is incremented both client and proxy recording the last received sequence number acknowledge by the remote peer. The acknowledgment timeout time used in ptunnel is 1500 milliseconds and after that time has elapsed without acknowledgment the packet is resent.

The length field is the number of bytes contained in the data field when the state field represents data forwarding (state = 1), and 0 otherwise. This is also a 32 bit wide field.

The reserved field is just for padding and is 16 bits wide.

The data field is exactly the same as the TCP data field and contains the actual data the be transported by TCP once the packet is received at the proxy.

For illustrated purposes, here is a diagram of the entire packet to be included within IP.

    0                   1                   2                   3

    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

   +-------------------------------+-------------------------------+

   |     Type      |     Code      |          Checksum             |

   +-------------------------------+-------------------------------+

   |           Identifier          |        Sequence Number        |

   +-------------------------------+-------------------------------+

   |                          Magic Number                         |

   +---------------------------------------------------------------+

   |                              IP                               |

   +---------------------------------------------------------------+

   |                             Port                              |

   +---------------------------------------------------------------+

   |                             State                             |

   +---------------------------------------------------------------+

   |                     Acknowledgment Number                     |

   +---------------------------------------------------------------+

   |                             Length                            |

   +-------------------------------+-------------------------------+

   |        Sequence Number        |            Reserved           |

   +-------------------------------+-------------------------------+

   |                            Data ...

   +----------------------------------------------------

As you can see the ptunnel protocol adds a lot of extra data to simple ICMP packets.

Ping tunnel also has some neat mechanisms for controlling multiple connections using the identifier field discussed earlier. It also emulated TCP transmission "windows" where the two hosts take turns in sending data. See the TCP RFC [7] for more information about TCP transmission windows. The authentication mechanism is simple and effective by sending a challenge for which the client should take the md5 hash of the password, appending to the challenge, and Finally taking another md5 hash of that. The proxy when match up this simple authentication and will respond accordingly.

The light at the end
When using PingTunnel you will probably have to expect a bit of latency due to all the extra data going around, not to mention that you're having to go through a proxy to get out onto the Internet! I do wonder if two sequence numbers are absolutely necessary, but hey, it's only 2 bytes wide anyway.

PingTunnel is an awesome utility and well work having around, it could do with some development. I'm tempted to request to join the project because I want to work on building a proxy-like interface to the program, such that you set a program, such as a web browser to use a SOCKS or HTTP proxy on localhost at the user-defined port number. This will overcome the program with web browsers and allow regular HTTP traffic to function perfectly.

Another idea and expansion I'd love to bring to the project would be a TCP wrapper shell, such that once within the wrapper all TCP traffic is automatically caught and redirected through the tunnel on the fly, this is a big project and will lead to the possibility of simply being able to start up the system in the TCP wrapper shell and have the full Internet experience without even having to move a finger or mess about changing tunnel destinations and so forth.

References (Viewed 10th to 12th may 2006)
[1] http://www.spectruminteractive.co.uk/
[2] http://en.wikipedia.org/wiki/Icmp
[3] http://www.cs.uit.no/~daniels/PingTunnel/
[4] http://sourceforge.net/projects/libpcap/
[5] http://www.ietf.org/rfc/rfc792.txt
[6] http://www.eventhelix.com/RealtimeMantra/Networking/Icmp.pdf
[7] http://www.ietf.org/rfc/rfc793.txt

This article as a pdf

I was reading this article at wired news and I almost find it outragous that he could face 5 years for an obviously unaggressive hack.

The original hack involved him penetrated The New York Times intranet and made use of a list of social security numbers to log into the networks user accounts, ending with him adding his own account after finding a high privledged account to add his from.

I actually would say that it was almost counter-productive to bring charged in the first place, considering his previously helpful history of exploiting a weakness, then helping fix it.

Check out this article from The Register about the hack.

Adrian Lamo has built an unusual reputation exposing security holes at large corporations, then voluntarily helping them fix the vulnerabilities he exploited -- sometimes visiting their offices or signing non-disclosure agreements in the process.

In December, Lamo was praised by communications giant WorldCom after he discovered, then helped close, security holes in their intranet that threatened to expose the private networks of Bank of America, CitiCorp, JP Morgan, and others.

In September, the hacker used a vulnerable Web-based production tool to tamper with a wire service story on Yahoo! News, deliberately choosing an old story to minimize the impact.

These other companies benefitted in the long run from their encourter with Lamo. The New York Times even seemed to have been more upset by it appearently. However, this is old news. The outragous part comes from current developments. The article begins

Hacker Adrian Lamo is in trouble again, this time for failing to give the federal government his DNA.

On Tuesday, federal probation officer Michael Sipe filed a notice of violation in a northern California court accusing Lamo of refusing to submit a blood sample, in violation of Sipe's instructions and a two-year-old federal law.

It's worth noting that DNA used to be only taken from violent and sexual criminals, but in 2004, 2 years after the original hack, the law was changed to force other types of criminals giving it. So this is my thought, was his original punishment supposed to force him to do things that weren't originally part of his sentence? I mean, I'm not law student or even that knowledge about the American legal system, being from the UK, but it just doesn't seem to completely make sense. I suppose there could be a clause in the ammendment to the law that would force all previously convicted criminals to adhere, unfortunately I don't have that information.

It's also silly because he is obviously not a type of criminal that the authorities would even have to match DNA too I'd imagine. His punishment also wasn't extreme, like his crime. He was serving house arrest with probation. It would be unreasonable to take his DNA unvoluntary, plus isn't it abit of a waste of resources? I think the odds of him ever commiting a violent or sexual crime are minimal from his record.

But hey, I'm from a different culture, I have different opinions and influences dictating how I view this situation, and I'd disagree with it strongly if this was a British law.