Data sent through sockets are sometimes inverted ?!!!

February 17th, 2012 - 03:38 am ET by mast4as | Report spam

Hi everyone

I have a simple client/server apps where the client send data (TCP/IP)
in the form of square of pixels colors (3 floats) to the server. I can
change the size of the square but by default, I started with something
like 32x32 pixels. So overall that's packets of 32*32*3*8 (8 bytes per
float) floats sent to the server. These packets are send to the server
for an entire image which can be say 1024x1024 in resolution. For my
testing I set the pixel color to always be 1 0 0.1.

On the client side:
* 1 write -> 4 integers to specify pos and size of the square in the
frame
* 1 write -> square dim^2 * 3 (RGB) * 8 bytes pixel color data

On the server side I do 2 reads:
* do a first read to find the dimension and position of the square in
the frame with 1 read
* read all the pixels for the current square (square size^2 * 3 (RGB)
* 8 bytes) with 1 read

It runs okay but occasionally the squares have wrong values. For a
block of 32x32 pixels say the first 20 pixels have RGB values that are
1 0 0.1 then the next 20 pixels have their RGB inverted say: 0 0.1 1
then the rest of the pixels will be 0.1 0 1 say. This problem only
happens when the square reach a certain size. When the square are 8x8
or 16x16.

So I was wondering if it is possible that when the packed of data sent
is too big, that the order of the bytes changes when it's read by the
server. Or is this not supposed to happen at all which would suggest a
bug in the code (however I really checked that I was writing and
reading the correct number of bytes so I find that strange).

If the write/read process doesn't guarantee that the bytes are not re-
ordered, is that just happening after the data written and read go
over a certain limit. Can I find this limit ? In other words is there
a limit (number of bytes) under which it is certain that bytes won't
be re-ordered. It seems that I can send the data in much smaller
packets (by sending say 8 pixels at a time from a square of 64x64) but
isn't that not very efficient ?

Thanks a lot for your help

-coralie

Follow the discussion Replies

27 replies Replies

Make a reply

Replies

#11 Ralph Spitzner

February 23rd, 2012 - 12:00 am ET | Report spam

Jorgen Grahn wrote:
[...]

Noone here has claimed that you don't need to check for partial writes,
but it could be very useful to know (e.g. for debugging purposes) how
Linux behaves in this case.

/Jorgen

Well write(2) and send(2) both return the number
of octets written, or -1.

What's the question here ?
Of course both might wait forever if the socket is
blocked for some reason.

Set O_NONBLOCK upon creating the socket and see what happens :-)
Another possibility is using a 'timed write', which could get you out
of waiting for(;;) or while(1) :-P

-rasp
PS: The OP should apply some sort of checksumming if he
really thinks he's losing some bytes

RTMPDump & ffmpeg are your friends..
-icke

Reply to this message

#12 Jorgen Grahn

February 23rd, 2012 - 02:34 am ET | Report spam

On Thu, 2012-02-23, Ralph Spitzner wrote:

Jorgen Grahn wrote:
[...]

Noone here has claimed that you don't need to check for partial writes,
but it could be very useful to know (e.g. for debugging purposes) how
Linux behaves in this case.

/Jorgen

Well write(2) and send(2) both return the number
of octets written, or -1.

What's the question here ?

The question is "under which situations does write() on a TCP socket
return a partial write?". ("Partial write" meaning that some, but not
all of the data you fed it got sent.)

PS: The OP should apply some sort of checksumming if he
really thinks he's losing some bytes

Why? TCP doesn't lose bytes[1]; only programming errors cause that.

/Jorgen

[1] Except the rare cases where packets get corrupted in transit
in such a way that the checksum is still valid.

// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

Reply to this message

#13 Chris Davies

February 23rd, 2012 - 06:12 am ET | Report spam

Jorgen Grahn <grahn+ wrote:

The question is "under which situations does write() on a TCP socket
return a partial write?". ("Partial write" meaning that some, but not
all of the data you fed it got sent.)

Empirically, I can confirm that partial writes can happen when the
network connection is broken.

I started with a socket connection to a remote host, and then had a loop
that wrote approximately 500 bytes per iteration. I used usleep() to slow
the loop a little. I had nc listening on the other end. I then manipulated
the firewall to simulate a network glitch - variously experimenting with
REJECT and DROP.

With the data session established but then blocked, I would stop the
netcat (nc) and then unblock the data path. The result was an immediate
failure of write.

For those who want to repeat my experiment, you can get my client from
http://www.roaima.co.uk/stuff/2012/02/23/client.c, and the commands I
ran were these:

nc -l -vvv \* 50194 # Listener

make client && ./client # Client

iptables -I OUTPUT -p tcp --dport 50194 -j REJECT # Client block
iptables -D OUTPUT -p tcp --dport 50194 -j REJECT # Client unblock

Chris

Reply to this message

#14 Ralph Spitzner

February 23rd, 2012 - 11:03 am ET | Report spam

Jorgen Grahn wrote:
[...]

The question is "under which situations does write() on a TCP socket
return a partial write?". ("Partial write" meaning that some, but not
all of the data you fed it got sent.)

Well if the connection somehow gets interrupted write will return
a number less that the number of bytes you fed to it.

[]

Why? TCP doesn't lose bytes[1]; only programming errors cause that.

/Jorgen

[1] Except the rare cases where packets get corrupted in transit
in such a way that the checksum is still valid.

Well yes, but the OP _believes_that TCP is doing some evil to his data,
apart from this not being true and not knowing his programming
techniques and/or skills, a checksum might help him :-P

-rasp

RTMPDump & ffmpeg are your friends..
-icke

Reply to this message

#15 Joe Pfeiffer

February 23rd, 2012 - 12:17 pm ET | Report spam

Ralph Spitzner writes:

Jorgen Grahn wrote:
[...]

The question is "under which situations does write() on a TCP socket
return a partial write?". ("Partial write" meaning that some, but not
all of the data you fed it got sent.)

Well if the connection somehow gets interrupted write will return
a number less that the number of bytes you fed to it.

[]

Why? TCP doesn't lose bytes[1]; only programming errors cause that.

/Jorgen

[1] Except the rare cases where packets get corrupted in transit
in such a way that the checksum is still valid.

Well yes, but the OP _believes_that TCP is doing some evil to his data,
apart from this not being true and not knowing his programming
techniques and/or skills, a checksum might help him :-P

At this point, the OP seems to have vanished back into the wilderness.
However, I think making sure he's really writing and reading all the
bytes he thinks he is will help him a lot more than adding another
checksum in his own code.

Note that my earlier observation that I don't remember ever seeing a
partial write() wasn't a result of an experiment seeing if I could
trigger it, it was just an observation of what I've encountered in
debugging my own code. It is interesting, though, to see people
experimenting with trying to find circumstances under which it can
happen.

Reply to this message