Issue5332

UW-HEP Help System

Issues
Show Unassigned
Show Critical
Show Urgent
Show Resolved
Show Open
Search

Login




Register
Lost your login?

Help
Roundup docs

Title www.hep.wisc.edu and the dasu account
Priority urgent Status chatting
Superseder Nosy List ajit, dan, dasu, help, rader, radtke, wcmaier, wsmith
Assigned To Topic Web
Group IT

Created 2008.06.30 19:50 by rader.
Last changed 2008.07.03 10:22 by wcmaier.

Files
File name Uploaded Type Edit Remove
smime.p7s dasu, 2008.07.01 00:29 application/pkcs7-signature
Messages
msg14451 (view) From: wcmaier To: ajit, dan, dasu, help, rader, radtke, wcmaier, wsmith Date: 2008.07.03 10:22 
On Tue, Jul 01, 2008 at 11:31:12AM -0500, Will Maier via UW-HEP Help System wrote:
> Dan's been surveying directories with condor-hosts ACLs. In addition
> to that, it would be nice to move away from using the wildcard '0'
> in the pts group. One option would be to automatically generate the
> membership for the condor-hosts group based on the output of
> condor_status. This is certainly doable from a Condor perspective.
> 
> Steve: would adding 2000 < N < 5000 pts entries be a Bad Thing?

Any thoughts on this?

-- 

o--------------------------{ Will Maier }--------------------------o
| jabber:...wcmaier@xmpp.lfod.us | email:..will.maier@hep.wisc.edu |
| office:...........608.263.9692 | cell:..............608.438.6162 |
*--------------------[ UW High Energy Physics ]--------------------*
msg14450 (view) From: wcmaier To: ajit, dan, dasu, help, rader, radtke, wcmaier, wsmith Date: 2008.07.03 10:20 
On Wed, Jul 02, 2008 at 08:34:26AM -0500, Will Maier via UW-HEP Help System wrote:
> Bah; I should've looked first. It doesn't seem like our httpd was
> compiled with libwrap support. Unless we want to fire up iptables, I
> guess we should leave it as-is for now.
> 
> If the logging becomes a problem again, the following in httpd.conf
> might help:
> 
>     SetEnvIf Request_URI "^/dasu/rootFiles$" dontlog
>     CustomLog logs/access_log common env=!dontlog

For now, I've simply added /var/log/httpd/*log to the nightly
logrotate (which had been ignoring the httpd logs for some reason). 

-- 

o--------------------------{ Will Maier }--------------------------o
| jabber:...wcmaier@xmpp.lfod.us | email:..will.maier@hep.wisc.edu |
| office:...........608.263.9692 | cell:..............608.438.6162 |
*--------------------[ UW High Energy Physics ]--------------------*
msg14435 (view) From: wcmaier To: ajit, dan, dasu, help, rader, radtke, wcmaier, wsmith Date: 2008.07.02 08:34 
On Wed, Jul 02, 2008 at 08:13:33AM -0500, rader@hep.wisc.edu wrote:
>  > ---- Original Message ----
>  > From: Will Maier via UW-HEP Help System <help@hep.wisc.edu>
>  > These two account for 99.5% of the recent bad traffic:
>  > 
>  >     216.104.34.62
>  >     206.225.81.178
>  > 
>  > Shall I add those to /etc/hosts.deny?
> 
> Yes, please.

Bah; I should've looked first. It doesn't seem like our httpd was
compiled with libwrap support. Unless we want to fire up iptables, I
guess we should leave it as-is for now.

If the logging becomes a problem again, the following in httpd.conf
might help:

    SetEnvIf Request_URI "^/dasu/rootFiles$" dontlog
    CustomLog logs/access_log common env=!dontlog

-- 

o--------------------------{ Will Maier }--------------------------o
| jabber:...wcmaier@xmpp.lfod.us | email:..will.maier@hep.wisc.edu |
| office:...........608.263.9692 | cell:..............608.438.6162 |
*--------------------[ UW High Energy Physics ]--------------------*
msg14434 (view) From: rader To: ajit, dan, dasu, help, rader, radtke, wcmaier, wsmith Date: 2008.07.02 08:21 
> ---- Original Message ----
 > From: Will Maier via UW-HEP Help System <help@hep.wisc.edu>
 > 
 > On Mon, Jun 30, 2008 at 07:50:19PM -0500, Steve Rader via UW-HEP Help System wrote:
 > > I guess we're okay now, but we should be aware that it seems
 > > there's someobody out there who understands AFS and is using loose
 > > ACLs to "own" us.
 > 
 > This is ongoing; we're still seeing numerous hosts GETing
 > http://www.hep.wisc.edu/dasu/rootFiles/, though they're getting 404s
 > now that Steve's removed it. The traffic caused Apache to nearly
 > fill the root disk with log events. I rotated and bzip2ed the log
 > files, but it might be prudent to block the most egregious IPs.
 > 
 > These two account for 99.5% of the recent bad traffic:
 > 
 >     216.104.34.62
 >     206.225.81.178
 > 
 > Shall I add those to /etc/hosts.deny?

Yes, please.

steve
--
msg14433 (view) From: wcmaier To: ajit, dan, dasu, help, rader, radtke, wcmaier, wsmith Date: 2008.07.02 07:38 
On Mon, Jun 30, 2008 at 07:50:19PM -0500, Steve Rader via UW-HEP Help System wrote:
> I guess we're okay now, but we should be aware that it seems
> there's someobody out there who understands AFS and is using loose
> ACLs to "own" us.

This is ongoing; we're still seeing numerous hosts GETing
http://www.hep.wisc.edu/dasu/rootFiles/, though they're getting 404s
now that Steve's removed it. The traffic caused Apache to nearly
fill the root disk with log events. I rotated and bzip2ed the log
files, but it might be prudent to block the most egregious IPs.

These two account for 99.5% of the recent bad traffic:

    216.104.34.62
    206.225.81.178

Shall I add those to /etc/hosts.deny?

-- 

o--------------------------{ Will Maier }--------------------------o
| jabber:...wcmaier@xmpp.lfod.us | email:..will.maier@hep.wisc.edu |
| office:...........608.263.9692 | cell:..............608.438.6162 |
*--------------------[ UW High Energy Physics ]--------------------*
msg14426 (view) From: wcmaier To: ajit, dan, dasu, help, rader, radtke, wcmaier, wsmith Date: 2008.07.01 11:31 
On Tue, Jul 01, 2008 at 09:49:32AM -0500, Dan Bradley via UW-HEP Help System wrote:
> Yes.  I think we should do a search for all cases where this ACL
> is being used.  I bet we can narrow down the range of allowed
> hosts in most or all cases.  We have been encouraging people to
> use condor file transfers to local scratch directories for a long
> time now, so I doubt there are a lot of legitimate needs for
> condor-hosts anymore.

Dan's been surveying directories with condor-hosts ACLs. In addition
to that, it would be nice to move away from using the wildcard '0'
in the pts group. One option would be to automatically generate the
membership for the condor-hosts group based on the output of
condor_status. This is certainly doable from a Condor perspective.

Steve: would adding 2000 < N < 5000 pts entries be a Bad Thing?

-- 

o--------------------------{ Will Maier }--------------------------o
| jabber:...wcmaier@xmpp.lfod.us | email:..will.maier@hep.wisc.edu |
| office:...........608.263.9692 | cell:..............608.438.6162 |
*--------------------[ UW High Energy Physics ]--------------------*
msg14425 (view) From: dan To: ajit, dan, dasu, help, rader, radtke, wcmaier, wsmith Date: 2008.07.01 09:49 
Steve Rader via UW-HEP Help System wrote:
>> The condor-hosts ACL does include the web server.  It is just this:
>>     
>  > 
>  >   128.104.28.0
>  >   128.104.29.0
>  >   128.105.0.0
>  >   198.51.254.0
>  >   198.51.255.0
>  >   128.104.3.0
>  >   128.104.202.0
>  >   128.104.32.0
>  >   144.92.101.0
>  >   144.92.180.0
>  >   144.92.181.0
>  >   144.92.182.0
>  >   144.92.183.0
>  >   128.104.55.0
>
> "just"??  That's potentially over 3500 systems!

Yes.  I think we should do a search for all cases where this ACL is 
being used.  I bet we can narrow down the range of allowed hosts in most 
or all cases.  We have been encouraging people to use condor file 
transfers to local scratch directories for a long time now, so I doubt 
there are a lot of legitimate needs for condor-hosts anymore.

>  
>
> Access to the web server is very tight.  The RW access only
> matters if you all have some cgi-kinda-thing that allows uploads?
>   

I'm not aware of any web-accessible scripts used by us that would allow 
uploads.  However, we should review what's there and remove anything 
that is not needed.

--Dan
msg14424 (view) From: rader To: ajit, dan, dasu, help, rader, radtke, wcmaier, wsmith Date: 2008.07.01 08:49 
> The condor-hosts ACL does include the web server.  It is just this:
 > 
 >   128.104.28.0
 >   128.104.29.0
 >   128.105.0.0
 >   198.51.254.0
 >   198.51.255.0
 >   128.104.3.0
 >   128.104.202.0
 >   128.104.32.0
 >   144.92.101.0
 >   144.92.180.0
 >   144.92.181.0
 >   144.92.182.0
 >   144.92.183.0
 >   128.104.55.0

"just"??  That's potentially over 3500 systems! 

Access to the web server is very tight.  The RW access only
matters if you all have some cgi-kinda-thing that allows uploads?

steve
--
msg14423 (view) From: rader To: ajit, dan, dasu, help, rader, radtke, wcmaier, wsmith Date: 2008.07.01 08:43 
> This is quite disturbing.  The loose ACLs (i.e., free write access to  
 > all nodes in condor-hosts) were set because we are using that method  
 > for shared file system across multiple DNs of GLOW.  I put those  
 > directories on web for convenience long ago but I don't really need  
 > them now.  Your removing the link is just fine. It appears to me that  
 > the "attack" is from one of the condor-hosts as I believe we don't let  
 > the web server post files to our directories, do we?

No, we don't directly allow uploading files.

We are not the only group to have loose AFS ACLs exploited, btw.

steve
--
msg14422 (view) From: wsmith To: ajit, dan, dasu, help, rader, radtke, wcmaier, wsmith Date: 2008.07.01 08:37 
Dear Dan:
Could we remove the web-server? What would be the consequence?
Regards,
Wesley

On Jul 1, 2008, at 3:34 PM, Dan Bradley via UW-HEP Help System wrote:

>
> The condor-hosts ACL does include the web server.  It is just this:
>
>  128.104.28.0
>  128.104.29.0
>  128.105.0.0
>  198.51.254.0
>  198.51.255.0
>  128.104.3.0
>  128.104.202.0
>  128.104.32.0
>  144.92.101.0
>  144.92.180.0
>  144.92.181.0
>  144.92.182.0
>  144.92.183.0
>  128.104.55.0
>
> --Dan
>
> Sridhara Dasu via UW-HEP Help System wrote:
>> Dear Steve,
>>
>> This is quite disturbing.  The loose ACLs (i.e., free write access to
>> all nodes in condor-hosts) were set because we are using that method
>> for shared file system across multiple DNs of GLOW.  I put those
>> directories on web for convenience long ago but I don't really need
>> them now.  Your removing the link is just fine. It appears to me that
>> the "attack" is from one of the condor-hosts as I believe we don't  
>> let
>> the web server post files to our directories, do we?
>>
>> Regards,
>> Sridhara
>>
>> ---------------------------------------------------------------------
>> Prof. Sridhara Rao Dasu                         Department of Physics
>> dasu@hep.wisc.edu                             University of Wisconsin
>> http://www.hep.wisc.edu/~dasu                    4289 Chamberlin Hall
>> 608-262-3678 ( Office )                        1150 University Avenue
>> 408-829-6625 (Wireless)                        Madison, WI 53706, USA
>>
>>
>> On Jun 30, 2008, at 5:50 PM, rader@hep.wisc.edu wrote:
>>
>>
>>> It happened again--the cause was: the index.html I created
>>> *disappeared*!
>>>
>>> It appears the person abusing our system has exploited the
>>> ACLs on /afs/hep.wisc.edu/cms/data/rootFiles.
>>>
>>> Since /afs/hep/home/dasu/www/rootFiles was a symlink to the
>>> directory above, so I just removed it (the symlink.)  The
>>> loadave and the server is okay now.
>>>
>>> On a lark, I checked for "viagra" in the logs and,
>>> alas, found that we served up various URLs of the form
>>> http://www.hep.wisc.edu//dasu/rootFiles/*/viagra about 1500
>>> times during the last 32 days.
>>>
>>> I guess we're okay now, but we should be aware that it seems
>>> there's someobody out there who understands AFS and is
>>> using loose ACLs to "own" us.
>>>
>>> steve
>>> --
>>>
>>
>> ----------
>> status: unread -> chatting
>>
>> ______________________________________
>> UW-HEP Help System <help@hep.wisc.edu>
>> <https://help.hep.wisc.edu/issue5332>
>> ______________________________________
>
> ______________________________________
> UW-HEP Help System <help@hep.wisc.edu>
> <https://help.hep.wisc.edu/issue5332>
> ______________________________________

=====================================================================
| Prof. Wesley H.Smith | Ph: 608-262-4690 or 2281, Fax:608-263-0800 |
| High Energy Physics  | Physics Dept., University of Wisconsin     |
| 4275 Chamberlin Hall | 1150 University Ave.,Madison WI 53706-1390 |
| wsmith@hep.wisc.edu  | http://hep.wisc.edu/wsmith/                |
=====================================================================
msg14421 (view) From: dan To: ajit, dan, dasu, help, rader, radtke, wcmaier, wsmith Date: 2008.07.01 08:34 
The condor-hosts ACL does include the web server.  It is just this:

  128.104.28.0
  128.104.29.0
  128.105.0.0
  198.51.254.0
  198.51.255.0
  128.104.3.0
  128.104.202.0
  128.104.32.0
  144.92.101.0
  144.92.180.0
  144.92.181.0
  144.92.182.0
  144.92.183.0
  128.104.55.0

--Dan

Sridhara Dasu via UW-HEP Help System wrote:
> Dear Steve,
>
> This is quite disturbing.  The loose ACLs (i.e., free write access to  
> all nodes in condor-hosts) were set because we are using that method  
> for shared file system across multiple DNs of GLOW.  I put those  
> directories on web for convenience long ago but I don't really need  
> them now.  Your removing the link is just fine. It appears to me that  
> the "attack" is from one of the condor-hosts as I believe we don't let  
> the web server post files to our directories, do we?
>
> Regards,
> Sridhara
>
> ---------------------------------------------------------------------
> Prof. Sridhara Rao Dasu                         Department of Physics
> dasu@hep.wisc.edu                             University of Wisconsin
> http://www.hep.wisc.edu/~dasu                    4289 Chamberlin Hall
> 608-262-3678 ( Office )                        1150 University Avenue
> 408-829-6625 (Wireless)                        Madison, WI 53706, USA
>
>
> On Jun 30, 2008, at 5:50 PM, rader@hep.wisc.edu wrote:
>
>   
>> It happened again--the cause was: the index.html I created
>> *disappeared*!
>>
>> It appears the person abusing our system has exploited the
>> ACLs on /afs/hep.wisc.edu/cms/data/rootFiles.
>>
>> Since /afs/hep/home/dasu/www/rootFiles was a symlink to the
>> directory above, so I just removed it (the symlink.)  The
>> loadave and the server is okay now.
>>
>> On a lark, I checked for "viagra" in the logs and,
>> alas, found that we served up various URLs of the form
>> http://www.hep.wisc.edu//dasu/rootFiles/*/viagra about 1500
>> times during the last 32 days.
>>
>> I guess we're okay now, but we should be aware that it seems
>> there's someobody out there who understands AFS and is
>> using loose ACLs to "own" us.
>>
>> steve
>> --
>>     
>
> ----------
> status: unread -> chatting
>
> ______________________________________
> UW-HEP Help System <help@hep.wisc.edu>
> <https://help.hep.wisc.edu/issue5332>
> ______________________________________
msg14420 (view) From: dasu To: ajit, dan, dasu, help, rader, radtke, wcmaier, wsmith Date: 2008.07.01 00:29 
Dear Steve,

This is quite disturbing.  The loose ACLs (i.e., free write access to  
all nodes in condor-hosts) were set because we are using that method  
for shared file system across multiple DNs of GLOW.  I put those  
directories on web for convenience long ago but I don't really need  
them now.  Your removing the link is just fine. It appears to me that  
the "attack" is from one of the condor-hosts as I believe we don't let  
the web server post files to our directories, do we?

Regards,
Sridhara

---------------------------------------------------------------------
Prof. Sridhara Rao Dasu                         Department of Physics
dasu@hep.wisc.edu                             University of Wisconsin
http://www.hep.wisc.edu/~dasu                    4289 Chamberlin Hall
608-262-3678 ( Office )                        1150 University Avenue
408-829-6625 (Wireless)                        Madison, WI 53706, USA


On Jun 30, 2008, at 5:50 PM, rader@hep.wisc.edu wrote:

>
> It happened again--the cause was: the index.html I created
> *disappeared*!
>
> It appears the person abusing our system has exploited the
> ACLs on /afs/hep.wisc.edu/cms/data/rootFiles.
>
> Since /afs/hep/home/dasu/www/rootFiles was a symlink to the
> directory above, so I just removed it (the symlink.)  The
> loadave and the server is okay now.
>
> On a lark, I checked for "viagra" in the logs and,
> alas, found that we served up various URLs of the form
> http://www.hep.wisc.edu//dasu/rootFiles/*/viagra about 1500
> times during the last 32 days.
>
> I guess we're okay now, but we should be aware that it seems
> there's someobody out there who understands AFS and is
> using loose ACLs to "own" us.
>
> steve
> --
msg14419 (view) From: rader To: ajit, dan, dasu, help, rader, radtke, wcmaier, wsmith Date: 2008.06.30 19:50 
It happened again--the cause was: the index.html I created 
*disappeared*!

It appears the person abusing our system has exploited the 
ACLs on /afs/hep.wisc.edu/cms/data/rootFiles.

Since /afs/hep/home/dasu/www/rootFiles was a symlink to the 
directory above, so I just removed it (the symlink.)  The
loadave and the server is okay now.

On a lark, I checked for "viagra" in the logs and,
alas, found that we served up various URLs of the form
http://www.hep.wisc.edu//dasu/rootFiles/*/viagra about 1500
times during the last 32 days.

I guess we're okay now, but we should be aware that it seems
there's someobody out there who understands AFS and is 
using loose ACLs to "own" us.

steve
--
History
Date User Action Args
2008-07-03 10:22:03wcmaiersetstatus: resolved -> chatting
messages: + msg14451
2008-07-03 10:20:10wcmaiersetstatus: chatting -> resolved
messages: + msg14450
2008-07-02 08:34:26wcmaiersetmessages: + msg14435
2008-07-02 08:21:02radersetmessages: + msg14434
2008-07-02 07:38:21wcmaiersetmessages: + msg14433
2008-07-01 11:31:12wcmaiersetmessages: + msg14426
2008-07-01 09:49:32dansetmessages: + msg14425
2008-07-01 08:49:38radersetmessages: + msg14424
2008-07-01 08:46:21wcmaiersetpriority: triage -> urgent
topic: + Web
assignedto: rader
2008-07-01 08:43:09radersetmessages: + msg14423
2008-07-01 08:37:07wsmithsetmessages: + msg14422
2008-07-01 08:34:24dansetmessages: + msg14421
2008-07-01 00:29:31dasusetfiles: + smime.p7s
status: unread -> chatting
messages: + msg14420
2008-06-30 19:50:19radercreate