UDP socket && sendto && EPERM

[Dmitry V. Krivenok]

Hello!

I wrote simple SNMP-client to test polling performance.
As you know SNMP works over UDP and my question
is about UDP socket and sendto system call.

My program is designed to poll devices "asynchronously", i.e.
at first I create N non-blocking sockets, then I send all requests
using sendto() and then wait for responses using select (in this
version, but I'm going to use epool in Linux).

The program works fine when I poll few hundred of devices,
but I get strange error when I poll several thousand of devices:

sendto returns -1 with errno set to EPERM (Operation not permitted).

sendto (2) manual page doesn't say anything about EPERM
error.
I searched through the google and found, that sendto may
fail with errno == EPERM if local firewall disallows outgoing UDP
packets.

iptables v1.2.11 is installed on target host, however OUTPUT
chain is empty:

#iptables -L OUTPUT -v -n
Chain OUTPUT (policy ACCEPT 39G packets, 7676G bytes)
pkts bytes target prot opt in out source
destination
#

I traced my program using strace utility and all
system calls related to the socket 22 are shown below:


5169 socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 22
...
...
5169 fcntl64(22, F_GETFL) = 0x2 (flags O_RDWR)
5169 fcntl64(22, F_SETFL, O_RDWR|O_NONBLOCK) = 0
...
5169 sendto(22, "0&\2\1\1\4\6public
\240\31\2\1\23\2\1\0\2\1\0000\0160\f"..., 40, 0, {sa_family=AF_INET,
sin_port=htons(161), sin_addr=inet_addr("10.19.120.1
")}, 16) = 40
5169 sendto(22, "0&\2\1\1\4\6public\240\31\2\1w\2\1\0\2\1\0000\0160\f
\6"..., 40, 0, {sa_family=AF_INET, sin_port=htons(161),
sin_addr=inet_addr("10.219.130.
46")}, 16) = 40
5169 sendto(22, "0\'\2\1\1\4\6public
\240\32\2\2\0\333\2\1\0\2\1\0000\016"..., 41, 0, {sa_family=AF_INET,
sin_port=htons(161), sin_addr=inet_addr("10.219.146
.78")}, 16) = 41
...
...
...
5169 sendto(22, "0(\2\1\1\4\6public\240\33\2\3\1^
\333\2\1\0\2\1\0000\016"..., 42, 0, {sa_family=AF_INET, sin_port=htons
(161), sin_addr=inet_addr("10.6.185.1
91")}, 16) = 42
5169 sendto(22, "0(\2\1\1\4\6public\240\33\2\3\1_?
\2\1\0\2\1\0000\0160\f"..., 42, 0, {sa_family=AF_INET, sin_port=htons
(161), sin_addr=inet_addr("10.7.2.199
")}, 16) = 42
5169 sendto(22, "0(\2\1\1\4\6public\240\33\2\3\1_
\243\2\1\0\2\1\0000\016"..., 42, 0, {sa_family=AF_INET, sin_port=htons
(161), sin_addr=inet_addr("10.8.51.52
")}, 16) = 42
5169 sendto(22, "0(\2\1\1\4\6public\240\33\2\3\1`
\7\2\1\0\2\1\0000\0160"..., 42, 0, {sa_family=AF_INET, sin_port=htons
(161), sin_addr=inet_addr("10.9.73.167
")}, 16) = -1 EPERM (Operation not permitted)

What may be the cause of my problem?
Any ideas?

Thank you beforehand!

[David Schwartz]

Post by Dmitry V. Krivenok
What may be the cause of my problem?

Failure to implement transmit pacing. With UDP, you are responsible
for spacing out your sends at a rational rate. You should start out at
a very slow rate and raise the rate as you confirm that there is
minimal or no packet loss. You can simply ignore this error if you
want, assuming you do properly implement transmit pacing.

DS

[Dmitry V. Krivenok]

Post by David Schwartz

Post by Dmitry V. Krivenok
What may be the cause of my problem?

Failure to implement transmit pacing. With UDP, you are responsible
for spacing out your sends at a rational rate. You should start out at

I didn't know about this.
I meant that send operation should always succeed or fail with EAGAIN
(or EWOULDBLOCK) error if non-blocking socket is used and there isn't
a room in a send buffer.
That is from sender point of view all the datagrams are successfully
delivered.
However, if sender is faster than received, then some buffers at
receiver side
may overflow and some datagrams may be discarded.
In such a case receiver may send ICMP source quench packet.
Am I right?

I executed tcpdump, but didn't see any ICMP source quench packets.
I also tried to send datagrams via blocking and non-blocking sockets.
There is no difference. sendto returns EPERM after some time.

Could you refer me to documentation explaining this problem?
I've just re-read Stevens's books and Linux manual pages, but I'm
still confused.

Post by David Schwartz
a very slow rate and raise the rate as you confirm that there is
minimal or no packet loss. You can simply ignore this error if you
want, assuming you do properly implement transmit pacing.
DS

[David Schwartz]

This post might be inappropriate. Click to display it.

[Dmitry V. Krivenok]

Post by David Schwartz

Post by Dmitry V. Krivenok

Post by David Schwartz
Failure to implement transmit pacing. With UDP, you are responsible
for spacing out your sends at a rational rate. You should start out at

I didn't know about this.
I meant that send operation should always succeed or fail with EAGAIN
(or EWOULDBLOCK) error if non-blocking socket is used and there isn't
a room in a send buffer.

The problem is that the operation may fail somewhere else. For
example, if the server has a gigabit ethernet link to a router that
has a 10 megabit circuit, the failure may occur at the router. The
'sendto' call can't detect that.

Yes, I agree.
But in my case 'sendto' _detects_ the error (EPERM) somehow.
I can't understand how Linux kernel finds out the error.
Indeed, as I said there isn't any ICMP source quench packets, so if
error
occurs at the router, then kernel can't detect it.
Hence, I guess that error occurs inside the Linux kernel.

Post by David Schwartz

Post by Dmitry V. Krivenok
That is from sender point of view all the datagrams are successfully
delivered.

Then ignore the error. Then, from your point of view, all datagrams
are successfully delivered.

Post by Dmitry V. Krivenok
However, if sender is faster than received, then some buffers at
receiver side
may overflow and some datagrams may be discarded.
In such a case receiver may send ICMP source quench packet.
Am I right?

Sure.

Post by Dmitry V. Krivenok
I executed tcpdump, but didn't see any ICMP source quench packets.
I also tried to send datagrams via blocking and non-blocking sockets.
There is no difference. sendto returns EPERM after some time.

Right, that's because you're not implementing transmit pacing.

Post by Dmitry V. Krivenok
Could you refer me to documentation explaining this problem?
I've just re-read Stevens's books and Linux manual pages, but I'm
still confused.

Look, you have no idea what the available bandwidth is. And UDP
doesn't provide any flow control except by accident. So you have to
determine experimentally the bandwidth. If you just blast full speed
and hope for the best, you will overfill buffers, lose packets, cause
the network to become unusable for other people, and generally act
anti-socially. TCP has this built-in, but UDP doesn't. So you have to
do it.
Why do you think TCP has slow start, exponential backoff,
retransmission, and so on? UDP has none of these things, so if you
need them, you have to code them.
DS

[Casper H.S. Dik]

Post by Dmitry V. Krivenok
Yes, I agree.
But in my case 'sendto' _detects_ the error (EPERM) somehow.
I can't understand how Linux kernel finds out the error.

Show the EXACT code (cut & paste) and we may give you an
explanation.

Casper

[Dmitry V. Krivenok]

Post by Casper H.S. Dik

Post by Dmitry V. Krivenok
Yes, I agree.
But in my case 'sendto' _detects_ the error (EPERM) somehow.
I can't understand how Linux kernel finds out the error.

Show the EXACT code (cut & paste) and we may give you an
explanation.

Whole program is big and uses external libraries.
All sending logic is implemented in SendRequests function.
It's presented below:

void SendRequests(const ParamMap_t& params, const std::vector<int>&
sockets)
{
int j = 0;
int l = 0;
int k = 0;
for(ParamMap_t::const_iterator i=params.begin(); i!=params.end(); +
+i)
{
Param p = i->second;
if(p.responded) continue;

j++;
OID_t oid;
ParseOID(global_config.oid_str, oid);
SnmpMessage msg(StrToSnmpVersion(p.version), p.community, oid,
p.request_id, "GetRequest");
const std::string message = msg.Serialize();

struct sockaddr_in servaddr;
bzero(&servaddr,sizeof(servaddr));
servaddr.sin_family = AF_INET;
servaddr.sin_addr.s_addr=inet_addr(p.host.c_str());
servaddr.sin_port=htons(p.port);
int m = j % global_config.sockets_per_thread;
while(1)
{
k++;
int r = sendto(sockets[m], message.c_str(), message.size(), 0,
(struct sockaddr *)&servaddr,sizeof(servaddr));
if(r==-1)
{
if(errno == EAGAIN || errno == EWOULDBLOCK || errno == EPERM)
{
std::cerr << "warning sendto : " << std::strerror(errno) <<
std::endl;
continue;
}
else
{
std::cerr << "sendto : " << std::strerror(errno) << std::endl;
assert(r!=-1);
}
}
else l++;
break;
}
}
std::cout << pthread_self() << " Sent " << j << "/" << l << "/" << k
<< " requests" << std::endl;
}

Post by Casper H.S. Dik
Casper
--
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

[Roy Smith]

In article

Post by Dmitry V. Krivenok
Hello!
I wrote simple SNMP-client to test polling performance.
As you know SNMP works over UDP and my question
is about UDP socket and sendto system call.
My program is designed to poll devices "asynchronously", i.e.
at first I create N non-blocking sockets, then I send all requests
using sendto() and then wait for responses using select (in this
version, but I'm going to use epool in Linux).

Are you creating one socket for each device you are polling? That's a very
expensive way to do it. Better to create a single socket and multiplex all
your SNMP traffic over that. You can tell which client responded by
looking at the source IP address which recvfrom() makes available.

Post by Dmitry V. Krivenok
The program works fine when I poll few hundred of devices,

You might want to check how many descriptors a single process can have open
on your system. By the time you get to several thousand sockets, you may
have well run into that limit.

[Dmitry V. Krivenok]

Post by Roy Smith
In article

Post by Dmitry V. Krivenok
Hello!
I wrote simple SNMP-client to test polling performance.
As you know SNMP works over UDP and my question
is about UDP socket and sendto system call.
My program is designed to poll devices "asynchronously", i.e.
at first I create N non-blocking sockets, then I send all requests
using sendto() and then wait for responses using select (in this
version, but I'm going to use epool in Linux).

Are you creating one socket for each device you are polling?  That's a very

Of course not.
I create N threads, each thread creates M sockets to send K UDP
datagrams.
N, M and K are set via command line options.
I have a lot of other configuration parameters (timeouts, retries,
etc).

Post by Roy Smith
expensive way to do it.  Better to create a single socket and multiplex all

I'm using select as I/O multiplexor.

Post by Roy Smith
your SNMP traffic over that.  You can tell which client responded by
looking at the source IP address which recvfrom() makes available.

Yes, I know.

Post by Roy Smith

Post by Dmitry V. Krivenok
The program works fine when I poll few hundred of devices,

You might want to check how many descriptors a single process can have open
on your system.  By the time you get to several thousand sockets, you may
have well run into that limit.

Yes, I know.
ulimit -n reports 1024
So I keep N*M < 1024

The question is not about program architecture at all.
It's about nature of EPERM error returned by sendto.

[David Schwartz]

Post by Dmitry V. Krivenok
The question is not about program architecture at all.
It's about nature of EPERM error returned by sendto.

Do you implement transmit pacing? If not, that's your problem. Without
transmit pacing, if you send too many packets in too little time,
something will break. That's why transmit pacing is mandatory. If you
use TCP, the kernel does it for you. If not, it is your
responsibility.

DS

[Dmitry V. Krivenok]

Post by David Schwartz

Post by Dmitry V. Krivenok
The question is not about program architecture at all.
It's about nature of EPERM error returned by sendto.

Do you implement transmit pacing? If not, that's your problem. Without
transmit pacing, if you send too many packets in too little time,
something will break. That's why transmit pacing is mandatory. If you
use TCP, the kernel does it for you. If not, it is your
responsibility.

That is if I send a lot of packets in a very short time interval, then
kernel
can simply return EPERM error instead of blocking in sendto system
call?

Post by David Schwartz
DS

[David Schwartz]

Post by Dmitry V. Krivenok
That is if I send a lot of packets in a very short time interval, then
kernel
can simply return EPERM error instead of blocking in sendto system
call?

1) You cannot rely on the kernel blocking in the sendto system call.
While this might happen, you cannot rely on it. For example, if the
packets are overloading a link that is not directly connected to this
computer, there is no way for the kernel to know.

2) If you don't like the EPERM error, ignore it. The kernel is trying
to be nice and telling you that your packet wasn't sent because, by
sheer luck, it happens to know this. If this confuses you or causes
you problems, fine, pretend you don't know it. The kernel is not
required to tell you there's a problem, so pretend it didn't if
finding out bothers you.

Again, you *MUST* implement transmit pacing. Simply blasting an
arbitrarily large amount of packets over a link in an arbitrarily
small amount of time is anti-social and may disrupt whatever else is
going on with that link. The kernel is entirely right to refuse to
permit you to behave that way.

You must use some kind of feedback loop to determine how rapidly to
send packets and back off when packets get dropped. UDP does not
provide transmit pacing like TCP does -- it is the application's
responsibility.

DS

[Barry Margolin]

In article

Post by David Schwartz
2) If you don't like the EPERM error, ignore it. The kernel is trying
to be nice and telling you that your packet wasn't sent because, by
sheer luck, it happens to know this. If this confuses you or causes
you problems, fine, pretend you don't know it. The kernel is not
required to tell you there's a problem, so pretend it didn't if
finding out bothers you.

Do you know this for a fact, or are you just guessing? Why does it
return this error code in that case, surely there must be a more
appropriate code, since this has nothing to do with permissions.

[David Schwartz]

Post by Barry Margolin
Do you know this for a fact, or are you just guessing?

I know for a fact that his code is broken and it's giving him problems
other people don't have. The only way to prove causality is for him to
fix his code and demonstrate that he still has the problem. He has to
fix it anyway.

Post by Barry Margolin
 Why does it
return this error code in that case, surely there must be a more
appropriate code, since this has nothing to do with permissions.

It has everything to do with permissions. Only 'root' is permitted to
use network resources in anti-social ways. Ordinary users are subject
to rational limits.

DS

[Dmitry V. Krivenok]

Post by David Schwartz

Post by Barry Margolin
Do you know this for a fact, or are you just guessing?

I know for a fact that his code is broken and it's giving him problems
other people don't have. The only way to prove causality is for him to
fix his code and demonstrate that he still has the problem. He has to
fix it anyway.

Post by Barry Margolin
 Why does it
return this error code in that case, surely there must be a more
appropriate code, since this has nothing to do with permissions.

I agree that EPERM error code looks strange.
Even more, it isn't documented in man (2) sendto.
Man only says:
Additional errors may be generated and returned from
the underlying protocol modules; see their respective manual pages.

However, man (7) udp says nothing about EPERM error.

Post by David Schwartz
It has everything to do with permissions. Only 'root' is permitted to
use network resources in anti-social ways. Ordinary users are subject
to rational limits.

I run my program as 'root' and got the same error.

Post by David Schwartz
DS

I changed architecture of my program.
EPERM errors are handled properly and outgoing packets rate is
dynamically
configured.

But I'm still confused about EPERM error.
I'm going to examine Linux kernel source code to find out when
exactly
EPERM error is returned.

[Michael Shuldman]

Post by Dmitry V. Krivenok
But I'm still confused about EPERM error.
I'm going to examine Linux kernel source code to find out when
exactly
EPERM error is returned.

Hi, I'm only curious, but did you find out what the problem was?

I'm no linux expert, but I became curious enough and downloaded a
version of the linux kernel (2.6.30) to take a peek. I found one
entry that I wonder if might be related to the problem you saw.

Since you said you had a firewall configured, even though it was
configured to pass all packets out ...

In the file ip_output.c one of the comments says:

" Marc Boucher : When call_out_firewall returns FW_QUEUE,
silently drop skb instead of failing with -EPERM."

I did not immediately manage to find any function called that, so I
did not examine how that function works, but perhaps it's related
to the problem you saw?

I'm just guessing now, but presumably that function was called in
your case (since you had a firewall configured) and your sending
eventually filled up the FW_QUEUE (whatever that is).
Since there apparently was a bug in the kernel that made it return
EPERM when that happened, that was perhaps what caused you problems?
Presumably disabling the firewall completely could help verify this
theory.

Maybe you could post an update if you found out what the error was?

Regards,

[Dmitry V. Krivenok]

Post by Michael Shuldman

Post by Dmitry V. Krivenok
But I'm still confused about EPERM error.
I'm going to examine Linux kernel source code to find out when
exactly
EPERM error is returned.

Hi, I'm only curious, but did you find out what the problem was?

Hello!

Unfortunately, I didn't find out the cause of the problem.
I haven't enough time to dig into kernel source code, but the
problem is still in my todo list.

Post by Michael Shuldman
I'm no linux expert, but I became curious enough and downloaded a
version of the linux kernel (2.6.30) to take a peek. I found one
entry that I wonder if might be related to the problem you saw.
Since you said you had a firewall configured, even though it was
configured to pass all packets out ...
" Marc Boucher : When call_out_firewall returns FW_QUEUE,
silently drop skb instead of failing with -EPERM."
I did not immediately manage to find any function called that, so I
did not examine how that function works, but perhaps it's related
to the problem you saw?
I'm just guessing now, but presumably that function was called in
your case (since you had a firewall configured) and your sending
eventually filled up the FW_QUEUE (whatever that is).
Since there apparently was a bug in the kernel that made it return
EPERM when that happened, that was perhaps what caused you problems?

It could be the cause of the problem.

Thank you very much for your response!

Post by Michael Shuldman
Presumably disabling the firewall completely could help verify this
theory.
Maybe you could post an update if you found out what the error was?
Regards,

[Jorgen Grahn]

Post by David Schwartz

Post by Barry Margolin
Do you know this for a fact, or are you just guessing?

I know for a fact that his code is broken and it's giving him problems
other people don't have. The only way to prove causality is for him to
fix his code and demonstrate that he still has the problem. He has to
fix it anyway.

Post by Barry Margolin
 Why does it
return this error code in that case, surely there must be a more
appropriate code, since this has nothing to do with permissions.

It has everything to do with permissions. Only 'root' is permitted to
use network resources in anti-social ways. Ordinary users are subject
to rational limits.

You need to provide references for this I think ...

But I think you are right; Linux doesn't just do what you would expect
(drop the datagram and claim it was sent anyway). At least not always
-- there may be a difference between bound and unbound UDP sockets, and
if PMTU is enabled or not. Unfortunately I have never seen it
documented; it's just hearsay and experiments on my part.

/Jorgen

[Barry Margolin]

On Wed, 6 May 2009 07:38:00 -0700 (PDT), David Schwartz

Post by David Schwartz

Post by Barry Margolin
Do you know this for a fact, or are you just guessing?

I know for a fact that his code is broken and it's giving him problems
other people don't have. The only way to prove causality is for him to
fix his code and demonstrate that he still has the problem. He has to
fix it anyway.

Post by Barry Margolin
 Why does it
return this error code in that case, surely there must be a more
appropriate code, since this has nothing to do with permissions.

It has everything to do with permissions. Only 'root' is permitted to
use network resources in anti-social ways. Ordinary users are subject
to rational limits.

You need to provide references for this I think ...

I'd also like to see that. I've never heard of any Unix-type system
that checks for "anti-social" behavior. Either you have permission to
use a resource or you don't. While it's certainly possible to implement
rate limiting in the network driver, I've never heard of Linux actually
doing so, certainly not by default.

[r***@gmail.com]

This post might be inappropriate. Click to display it.

[Jorgen Grahn]

Post by r***@gmail.com

In article <>,

On Wed, 6 May 2009 07:38:00 -0700 (PDT), David Schwartz

Post by David Schwartz

Post by Barry Margolin
Do you know this for a fact, or are you just guessing?

I know for a fact that his code is broken and it's giving him problems
other people don't have. The only way to prove causality is for him to
fix his code and demonstrate that he still has the problem. He has to
fix it anyway.

Post by Barry Margolin
???Why does it
return this error code in that case, surely there must be a more
appropriate code, since this has nothing to do with permissions.

It has everything to do with permissions. Only 'root' is permitted to
use network resources in anti-social ways. Ordinary users are subject
to rational limits.

You need to provide references for this I think ...

I'd also like to see that. I've never heard of any Unix-type system
that checks for "anti-social" behavior. Either you have permission to
use a resource or you don't. While it's certainly possible to implement
rate limiting in the network driver, I've never heard of Linux actually
doing so, certainly not by default.

Sorry for resurrecting an old thread, but I see this error on
day-to-day basis and it has everything to do that I am sending UDP
packets without rate control. What is interesting is that - at least
on my testing servers - if we turn off iptables completely we don't
witness this problem. I wonder is iptables somehow causing EPERM.

You'd have to describe your own problem -- most people won't have
access to those postings from 2009, and the technical details aren't
present in the part you quote above.

Post by r***@gmail.com
I don't know are Google Groups the right venue for sending this
email. Hoping that somebody will see this :)

We see it. But this is not Google Groups -- this is USENET.
GG is just (barely) a gateway to it.

/Jorgen

[Rade Martinović]

Post by Jorgen Grahn

Post by r***@gmail.com
Sorry for resurrecting an old thread, but I see this error on
day-to-day basis and it has everything to do that I am sending UDP
packets without rate control. What is interesting is that - at least
on my testing servers - if we turn off iptables completely we don't
witness this problem. I wonder is iptables somehow causing EPERM.

You'd have to describe your own problem -- most people won't have
access to those postings from 2009, and the technical details aren't
present in the part you quote above.

Post by r***@gmail.com
I don't know are Google Groups the right venue for sending this
email. Hoping that somebody will see this :)

We see it. But this is not Google Groups -- this is USENET.
GG is just (barely) a gateway to it.
/Jorgen
--
\X/ snipabacken.se> O o .

Thank you for replying to me.

My team and I have tackling this issue in our short-term plans. I will then start a new thread, to keep discussion clear and with more technicalities.

But in a nutshell - we are sending immense amount of SNMP queries - which equate to UDP packets and keep hitting EPERM if the iptables is up. We use Java libraries for handling SNMP querying - snmp4j. I don't quite remember, but running our app as a restricted user or as a root does not make any difference.

I hope that using Google Groups as a gateway to USENET is OK. I am not really sure what are proper ways to use USENET.

Thanks,
Rade

[Jorgen Grahn]

Post by Rade Martinović

Post by Jorgen Grahn

Post by r***@gmail.com
Sorry for resurrecting an old thread, but I see this error on
day-to-day basis and it has everything to do that I am sending UDP
packets without rate control. What is interesting is that - at least
on my testing servers - if we turn off iptables completely we don't
witness this problem. I wonder is iptables somehow causing EPERM.

You'd have to describe your own problem -- most people won't have
access to those postings from 2009, and the technical details aren't
present in the part you quote above.

Post by r***@gmail.com
I don't know are Google Groups the right venue for sending this
email. Hoping that somebody will see this :)

We see it. But this is not Google Groups -- this is USENET.
GG is just (barely) a gateway to it.

Thank you for replying to me.
My team and I have tackling this issue in our short-term plans. I
will then start a new thread, to keep discussion clear and with more
technicalities.
But in a nutshell - we are sending immense amount of SNMP queries -
which equate to UDP packets and keep hitting EPERM if the iptables is
up. We use Java libraries for handling SNMP querying - snmp4j. I don't
quite remember, but running our app as a restricted user or as a root
does not make any difference.

Googling for iptables EPERM, I get hits like these:

http://comments.gmane.org/gmane.comp.security.firewalls.netfilter.devel/29993

describing the Linux behavior, and people complaining about it. That
was a few years ago, but the maintainers seem to think an output DROP
should be visible to the application as EPERM. Is that not what
you're seeing?

(At this point I hope someone more interested and knowledgable will
step in. I only know what I read in that URL, and don't remember what
the 2009 discussion was about.)

Post by Rade Martinović
I hope that using Google Groups as a gateway to USENET is OK. I am
not really sure what are proper ways to use USENET.

Google Groups has some issues, in particular some misfeatures which
makes the postings hard to handle for regular Usenet users. Lots has
been written about it; I'm sure you can google for a list of problems
and ways to deal with them.

/Jorgen